Getting Shopify data into BigQuery

Shopify doesn't give you a database, so every route here goes through an API. That single fact decides the cost, the speed and the failure modes of all of them.

There is no database to connect

On PrestaShop, WooCommerce or Magento you can point something at MySQL. Shopify is API-only: you get the Admin API, and that's the whole surface. So the question isn't which database to read, it's how to move a store's history through an API without spending a week on pagination and rate limits.

The REST Admin API is paginated and rate-limited, and pulling several years of orders page by page is slow enough that people give up and pull only recent data. The right tool for volume is the GraphQL Bulk Operations API: you submit a query, Shopify runs it in the background and hands you a JSONL file to download. It's genuinely good, and it's also a job queue you now have to poll, retry and monitor.

Then there's shape. The API returns nested JSON — an order carries its line items, discounts, tax lines and fulfilments inside it. BigQuery can store that as nested fields, but every downstream query has to unnest it, and most people end up flattening line items into their own table anyway. That modelling is the actual work, exactly as it is on a MySQL store.

Three ways to do it

All three work. They differ in what they cost you — in money, in maintenance, and in load on the database your customers are using right now.

Managed ETL connector

A hosted service with a Shopify source that lands orders, products and customers in BigQuery on a schedule.

  • Shopify's API quirks, pagination and retries are someone else's problem
  • Running the same afternoon you sign up
  • Historical backfill included rather than hand-written
  • Priced per row or per active table, and order line items add up
  • You get the API's shape — modelling is still yours
  • Another processor holding a token to your store

Build it on the Bulk Operations API

A GraphQL bulk query per object, polled to completion, JSONL downloaded and loaded into BigQuery on your own cron.

  • Cheapest at scale, and it's the API designed for exactly this
  • Total control of which fields travel and how they're modelled
  • No third party holds your access token
  • You are now operating a job queue: submit, poll, retry, alert
  • Schema drift and API version deprecations land on you twice a year
  • Days of work before the first useful chart

A PostgreSQL copy from BeQuery

Not yet

The model we run for MySQL stores — an isolated, verified copy you query directly or load onward. Shopify needs a dedicated API connector, which is in development.

  • Would remove the extraction problem the same way it does for MySQL stores
  • A real PostgreSQL database rather than raw API JSON
  • Same SQL editor, insights, reports and BI connections as every other source
  • Not available for Shopify today — this is an early-access list, not a product you can buy
  • If you need this working this month, one of the other two columns is your answer

The route that works today, step by step

  1. 1

    Create a custom app and scope its token

    In Shopify admin, Settings → Apps and sales channels → Develop apps. Grant read access to orders, products and customers, and nothing else. The token this gives you is the only credential in the whole pipeline, so scope it tightly.

  2. 2

    Submit a bulk query per object

    Use the GraphQL bulkOperationRunQuery mutation for orders, products and customers separately. Shopify queues each one and runs it against the whole dataset — no pagination on your side.

  3. 3

    Poll, then download the JSONL

    Check the operation's status until it completes and Shopify gives you a signed URL to a JSONL file. Nested children arrive as their own lines with a parent id, which is what you'll join on.

  4. 4

    Load into BigQuery and flatten

    bq load handles JSONL directly. Land it raw, then build views that flatten line items into their own table — one row per line, joined to the order. Every useful revenue and product query hangs off that shape.

Do you actually need a warehouse?

Worth asking before you build anything. A warehouse is the right tool for some of these situations and overkill for the rest.

Yes — build it

  • You're joining Shopify data with ad spend, a CRM or offline sales
  • You have several stores or markets you want modelled together
  • Someone on the team already writes BigQuery SQL

Probably not — a clone is enough

  • One store, and the questions are revenue, products, customers and stock
  • Shopify's own reports plus a spreadsheet still answer everything you ask
  • Nobody wants to own a pipeline against an API that versions twice a year

Common questions

Can I use BeQuery with Shopify today?

No. BeQuery clones a store's MySQL database, and Shopify doesn't have one to connect — it's API-only. A dedicated Shopify connector is in development and there's an early-access list you can join. Until it ships, the honest answer for Shopify is a managed ETL connector or the Bulk Operations API, which is why both are described above in more detail than our own column.

Why not just use the REST API and paginate?

It works and it's the obvious first attempt, but it's slow under rate limits and it's the part that breaks quietly: a job that takes hours is a job that fails halfway. Bulk Operations exists precisely because exporting a full store through paginated REST calls is the wrong shape for the problem.

Do I need Shopify Plus for any of this?

No. The Admin API, custom apps and Bulk Operations are available on standard Shopify plans. What varies by plan is the depth of Shopify's own built-in analytics — which is often what sends people looking for a warehouse in the first place.

We're building this for Shopify

BeQuery works today on stores with a MySQL database — PrestaShop, WooCommerce and Magento. Shopify is API-only, so it needs a dedicated connector, and that's in development. Put your email down and we'll tell you the day it opens.