Loading Mock Machines data into your database

An experiment run with -format duckdb leaves a single, self-contained dataset.duckdb file in the experiment directory: one table per state machine, plus an event_log table, already typed and ready to query. This page is the “now what” — copy-pasteable recipes for getting those tables into BigQuery, Snowflake, Databricks, MotherDuck, DuckLake, MySQL, or Postgres (including Supabase and Neon).

What you have, and the two strategies

Running -run N -format duckdb (see the OSI doc’s export walkthrough for the full pipeline) leaves this in the experiment directory:

experiments/<id>/
  dataset.duckdb        # one table per machine, e.g. customer, order, line_item...
                         # plus event_log if event logging was on (the default)
  dataset.osi.yaml       # optional: the OSI semantic model describing it (-osi)

Every column is already resolved — BIGINT ids/counters, DOUBLE measures, VARCHAR categoricals and taxonomies, TIMESTAMP WITH TIME ZONE for any declared timestamp field — so none of the recipes below need type hints or a schema-mapping step. Which recipe you reach for depends on whether the destination speaks DuckDB natively:

Strategy	Destinations	Mechanism
Native attach	MotherDuck, DuckLake, Postgres, MySQL	DuckDB has a first-class extension that can ATTACH the target as a second database in the same session, so tables move over with plain SQL — no export step, no intermediate files.
Parquet-mediated bulk load	BigQuery, Snowflake, Databricks	DuckDB has no write extension for these. Export each table to Parquet (DuckDB already knows how — it’s the format ConvertToDuckDB itself imports from), then hand the files to the platform’s own bulk-load path.

One thing every recipe below shares

Every ATTACH-based recipe supports two ways of moving data: a single COPY FROM DATABASE src TO dst; statement that copies every table in one shot (DuckDB ≥ 0.10), or a per-table CREATE TABLE dst.<t> AS SELECT * FROM src.<t>; loop when you want to rename, filter, or exclude a table (e.g. skip event_log). Both are shown for Postgres; the others show whichever reads more naturally.

Listing tables generically

dataset.duckdb always has a different table list depending on the scenario, so every scripted example below starts from the same generic lookup instead of hardcoding table names. In SQL:

SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main';

Or in Python, iterating and pulling each table out as a DataFrame or Arrow table:

import duckdb

con = duckdb.connect("dataset.duckdb")
tables = [r[0] for r in con.sql(
    "SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main'"
).fetchall()]

for name in tables:
    df = con.sql(f'SELECT * FROM "{name}"').df()   # or .arrow() for zero-copy Arrow
    print(name, len(df), "rows")

The rest of this page reuses tables from this snippet and con from this connection.

MotherDuck

MotherDuck is hosted DuckDB, so this is the shortest recipe on the page — attach your MotherDuck database alongside the local file and copy straight across:

-- export MOTHERDUCK_TOKEN=<your token>   (or pass -motherduck_token=... to the CLI)
duckdb dataset.duckdb <<'SQL'
INSTALL motherduck;
LOAD motherduck;
ATTACH 'md:my_warehouse' AS md;

COPY FROM DATABASE dataset TO md;   -- every table, one shot
SQL

Tip

MotherDuck auto-installs and auto-loads the motherduck extension the first time you ATTACH 'md:...', so the explicit INSTALL/LOAD lines are optional — shown here for clarity and to match every other recipe on this page.

DuckLake

DuckLake is an open table format where the catalog (table/schema metadata) lives in a small database and the data lives as Parquet files wherever you point it — local disk or object storage. Attach a DuckLake catalog next to your export and copy the tables in:

duckdb dataset.duckdb <<'SQL'
INSTALL ducklake;
LOAD ducklake;
ATTACH 'ducklake:catalog.ducklake' AS lake (DATA_PATH 'lake_data/');

COPY FROM DATABASE dataset TO lake;
SQL

catalog.ducklake here is itself just a DuckDB file holding the catalog; for a shared/production lake, point the catalog at Postgres or MySQL instead and the data path at object storage:

ATTACH 'ducklake:postgres:dbname=ducklake_catalog host=catalog.internal user=lake'
  AS lake (DATA_PATH 's3://my-bucket/mock-machines/');

Postgres (Supabase, Neon)

DuckDB’s postgres extension attaches a live Postgres connection as a second database, so both the bulk copy and the per-table loop are plain SQL:

duckdb dataset.duckdb <<'SQL'
INSTALL postgres;
LOAD postgres;
ATTACH 'dbname=mydb host=127.0.0.1 user=postgres password=secret port=5432'
  AS pg (TYPE postgres);

-- Option A: every table in one shot
COPY FROM DATABASE dataset TO pg;

-- Option B: pick your tables, e.g. skip event_log
CREATE TABLE pg.customer   AS SELECT * FROM dataset.customer;
CREATE TABLE pg."order"    AS SELECT * FROM dataset."order";
CREATE TABLE pg.line_item  AS SELECT * FROM dataset.line_item;
SQL

The Python equivalent, looping the generic table list from earlier:

con.execute("INSTALL postgres; LOAD postgres;")
con.execute(
    "ATTACH 'dbname=mydb host=127.0.0.1 user=postgres password=secret port=5432' "
    "AS pg (TYPE postgres)"
)
for name in tables:
    if name == "event_log":
        continue
    con.execute(f'CREATE TABLE pg."{name}" AS SELECT * FROM dataset."{name}"')

Supabase and Neon are plain Postgres — only the connection string differs

Supabase: the pooled connection (port 6543) is transaction-mode PgBouncer, which doesn’t support prepared statements the way DuckDB’s bulk COPY/CREATE TABLE AS needs — use the direct connection (port 5432, host db.<project-ref>.supabase.co) for this kind of bulk load.

Neon: requires TLS — add sslmode=require to the connection string. Neon’s branch-specific hostnames work exactly like any other Postgres host string; there’s nothing else to change.

MySQL

DuckDB’s mysql extension mirrors the Postgres one exactly:

duckdb dataset.duckdb <<'SQL'
INSTALL mysql;
LOAD mysql;
ATTACH 'host=127.0.0.1 user=root password=secret database=mydb port=3306'
  AS mysql_db (TYPE mysql);

COPY FROM DATABASE dataset TO mysql_db;
SQL

Note

MySQL’s stricter reserved-word and identifier rules mean a table named order (as this project’s own OnlineShopping scenario exports) needs backtick-quoting on the MySQL side if you reference it outside DuckDB — DuckDB’s COPY FROM DATABASE handles the quoting for you.

BigQuery

DuckDB has no BigQuery write path, so export to Parquet first (DuckDB’s own COPY ... TO ... (FORMAT PARQUET), one file per table), then load with the bq CLI or the Python client:

# List tables, then export each to Parquet in ONE duckdb session (two
# concurrent `duckdb` processes on the same file conflict on its file lock,
# so gather the table list first, then run every COPY in a single invocation)
mkdir -p export
tables=$(duckdb dataset.duckdb -noheader -list -c \
  "SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main';")

sql=""
for t in $tables; do
  sql+="COPY (SELECT * FROM \"$t\") TO 'export/$t.parquet' (FORMAT PARQUET);"
done
duckdb dataset.duckdb -c "$sql"

# Load each Parquet file into BigQuery (dataset must already exist)
for f in export/*.parquet; do
  table=$(basename "$f" .parquet)
  bq load --source_format=PARQUET my_dataset."$table" "$f"
done

Or scripted end-to-end in Python, without an intermediate shell loop:

import duckdb
from google.cloud import bigquery

con = duckdb.connect("dataset.duckdb")
bq = bigquery.Client()
job_config = bigquery.LoadJobConfig(source_format=bigquery.SourceFormat.PARQUET)

tables = [r[0] for r in con.sql(
    "SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main'"
).fetchall()]

for name in tables:
    path = f"export/{name}.parquet"
    con.execute(f'COPY (SELECT * FROM "{name}") TO \'{path}\' (FORMAT PARQUET)')
    with open(path, "rb") as f:
        job = bq.load_table_from_file(f, f"my_dataset.{name}", job_config=job_config)
        job.result()   # block until the load finishes (or raises)

Note

bq load also accepts a gs:// URI directly — for large exports, upload the Parquet files to a bucket first (gsutil cp export/*.parquet gs://my-bucket/mock-machines/) and point bq load at the bucket path instead of a local file, which is the recommended path for anything beyond a quick load.

Snowflake

Same Parquet export, then either stage-and-COPY INTO in pure SQL, or the write_pandas helper in Python, which manages the stage for you. The Python route needs no manual stage/format setup, so it’s the simpler script to keep around:

import duckdb
import snowflake.connector
from snowflake.connector.pandas_tools import write_pandas

con = duckdb.connect("dataset.duckdb")
sf = snowflake.connector.connect(
    user="...", password="...", account="...",
    warehouse="...", database="MY_DB", schema="PUBLIC",
)

tables = [r[0] for r in con.sql(
    "SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main'"
).fetchall()]

for name in tables:
    df = con.sql(f'SELECT * FROM "{name}"').df()
    write_pandas(sf, df, table_name=name.upper(), auto_create_table=True)

The equivalent pure-SQL path, if you’d rather not add a Python dependency:

-- 1. Export to Parquet (as in the BigQuery section), then, in Snowflake:
CREATE OR REPLACE STAGE mm_stage;
PUT file://export/order.parquet @mm_stage AUTO_COMPRESS=FALSE;

CREATE OR REPLACE TABLE "order" (...);   -- column list matches dataset.duckdb's schema
COPY INTO "order"
  FROM @mm_stage/order.parquet
  FILE_FORMAT = (TYPE = PARQUET)
  MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

Databricks

Parquet export again, then land the files in a Unity Catalog Volume and COPY INTO a managed table — either from the CLI/SQL, or scripted with the databricks-sql-connector:

# Upload the Parquet files to a Unity Catalog Volume
databricks fs cp -r export dbfs:/Volumes/main/default/mm_export

-- Then, in a Databricks SQL session (or via the Python connector below)
COPY INTO main.default.order
FROM '/Volumes/main/default/mm_export/order.parquet'
FILEFORMAT = PARQUET;

Scripted end-to-end, looping every table after the upload:

from databricks import sql as dbsql

conn = dbsql.connect(
    server_hostname="...", http_path="...", access_token="...",
)
cur = conn.cursor()

for name in tables:   # from the generic listing snippet
    cur.execute(f"""
        COPY INTO main.default.{name}
        FROM '/Volumes/main/default/mm_export/{name}.parquet'
        FILEFORMAT = PARQUET
    """)

Choosing an approach

Target	Mechanism	Extra credentials/network beyond the target itself	All tables in one statement?
MotherDuck	Native attach	Just a MotherDuck token	Yes — COPY FROM DATABASE
DuckLake	Native attach	None (catalog + data path only)	Yes — COPY FROM DATABASE
Postgres / Supabase / Neon	Native attach	None	Yes — COPY FROM DATABASE
MySQL	Native attach	None	Yes — COPY FROM DATABASE
BigQuery	Parquet + bq load / Python client	GCP credentials; a bucket for anything beyond a quick load	No — one load job per table
Snowflake	Parquet + stage/COPY INTO, or write_pandas	Snowflake account/warehouse credentials	No — one load per table
Databricks	Parquet + Volume/COPY INTO	Workspace host + access token, Unity Catalog Volume	No — one COPY INTO per table

If the target has a native DuckDB extension, prefer it: no intermediate files, no per-table loop to maintain, and COPY FROM DATABASE picks up new tables the next time a scenario adds a machine without any script changes. Reach for the Parquet-mediated path only when the target leaves no other option.

References

• internal/engine/store/dataset/export.go — ConvertToDuckDB, the Parquet → DuckDB importer that produces dataset.duckdb in the first place.
• OSI Semantic Layer — the companion dataset.osi.yaml that describes the same tables’ business meaning, joins, and metrics.
• DuckDB Postgres extension, MySQL extension, MotherDuck extension, DuckLake — upstream docs for the native-attach extensions above.
• BigQuery Parquet loads, Snowflake COPY INTO, Databricks COPY INTO — upstream docs for the Parquet-mediated targets above.