Benchmarking

A layer of Benchmark* functions over the store hot path and the simulation loop, and an on-demand benchmarking suite that drives them into per-commit run folders: topline throughput, pprof profiles, scaling sweeps, and capacity answers (rows/second to disk, entities per GB, cache residency). None of it runs under go test or in CI — it is invoked by hand when you want numbers. The analyses behind the capacity questions live in ai/ram-cache-perf.md (cache & memory) and ai/flush-passive-to-disk.md (larger-than-memory generation).

Benchmark inventory

Seventeen benchmarks across five files, in two groups. All call b.ReportAllocs() so allocation regressions surface. Run them directly with go test ./internal/engine/... -bench=. -benchmem, or let the benchmarking suite drive the curated subset.

Group	File	Benchmarks	Measures
Store hot path (100 → 10k entities)	store/store_archetype_bench_test.go	BenchmarkForEach, BenchmarkBulkFieldSumComparative, BenchmarkCreateEntity, BenchmarkFindByField, BenchmarkAccess	Columnar iteration, bulk field aggregation, entity creation, worst-case lookup, and handle-to-accessor resolution on the archetype store.
Store hot path (100 → 10k entities)	runtime/store_bench_test.go	BenchmarkStep, BenchmarkWriteCSV, BenchmarkBulkFieldSum	Per-turn step cost, CSV export, and bulk field sum through the runtime.
Sim loop & scaling (FixtureBenchmark)	enginetest/bench_test.go	BenchmarkSimLoop, BenchmarkSimLoopOrder, BenchmarkSimLoopWithEventLog, BenchmarkExport, BenchmarkPaymentProcessorExport, BenchmarkDataGeneration, BenchmarkSimLoopPartitioned	End-to-end per-turn loop (with/without event log), order-fixed vs order-random locality, CSV/Parquet export encode, data generation to disk (export on, logging off), and partitioned execution.
Sim loop & scaling (FixtureBenchmark)	enginetest/bench_scaling_test.go, bench_scaling_flush_test.go	BenchmarkSimLoopTurns, BenchmarkSimLoopTurnsFlushing	Turn-horizon sweeps proving per-turn cost stays O(live) and (flushing on) resident memory stays bounded by the live set.

The per-turn phases of the sim loop (activate, step, cascade) carry pprof labels, plus a partition label, so a CPU profile can be sliced by phase. The benchmarking suite below wraps these benchmarks to produce topline, profile, scaling, throughput, cache, and historical reports.

The benchmarking suite

tools/bench/suite/ is a single package main command that runs the benchmarks and collects the results, profiles, and reports into one folder per commit. It sits outside the build, test, and CI path: a plain go test compiles the benchmark functions but never runs them, and the suite is never wired into the pre-commit hook or CI. You invoke it by hand when you want numbers:

# The default battery: topline → profile → scaling → throughput
go run ./tools/bench/suite

# A fast smoke run (small benchtime + a single throughput budget)
go run ./tools/bench/suite --quick

# Opt in to the cache stage (Order A/B delta + hardware counters when available)
go run ./tools/bench/suite --cache

# Opt in to the cross-version historical table
go run ./tools/bench/suite --historical

# The 50M-row flushing acceptance harness (needs a run directory)
go run ./tools/bench/suite --demo50m --runDir ./out

The default battery runs four stages in order — topline → profile → scaling → throughput — across the low-variance fixtures (FixtureBenchmark, FixtureBenchmark4) and the example scenarios (PaymentProcessor and friends, via the benchmarks). The remaining stages are opt-in.

Flag	What it does
--quick	Fast smoke run — a small benchtime and a single throughput budget instead of the full sweep.
--cache	Add the cache stage: the OrderFixed vs OrderRandom ns/entity delta (always), plus real L1/L2/LLC counters via perf / cachegrind when the host exposes them.
--historical	Add the historical stage: a cross-version throughput table (ns/entity/turn on an all-active workload) for the sim loop at past merge commits and HEAD, built in detached throwaway worktrees.
--demo50m	Run the 50M-row flushing acceptance harness (bounded-memory generation). Requires --runDir.
--benchtime <spec>	Override the benchtime for the profiled runs (default 10x).
--bench <regex>	Profile one custom benchmark matching the regex instead of the curated targets.
--deep	Profiled runs add -tags profile so the CPU profile carries the finer machine / step subphase labels (~6× slower).
--runDir <path>	Store root for the --demo50m harness output.

Per-commit run folders

Each invocation writes one timestamped, per-commit folder under tools/bench/runs/ (gitignored). The folder is named <YYYYMMDD>-<shortSHA> — with a -dirty suffix when the working tree is dirty — e.g. 20260608-3bd6dca53.

One folder per commit

Before each run, any prior folder for the same commit is removed, so a commit always has exactly one run folder — the latest. Re-running on the same commit overwrites in place; commit (or dirty up) to keep a separate folder.

The folder layout:

tools/bench/runs/<YYYYMMDD>-<shortSHA>/
├── meta.json            # commit, branch, dirty, when, host, os/arch, go version,
│                        #   numcpu, benchtime, flags, per-stage status + duration
├── index.html           # navigable report: env + stages tables, the parsed topline /
│                        #   scaling / throughput tables, and per profile a description,
│                        #   the embedded call graph, artifact links, and a pprof command
├── results/
│   ├── topline.txt          # raw `go test -bench`
│   ├── topline.json         # parsed per-run totals (ns / entities / rows / bytes /
│   │                        #   turns) + the normalised ns_per_entity_turn
│   ├── topline.table.txt
│   ├── scaling.txt          # (+ scaling.raw.txt)
│   ├── throughput.txt
│   ├── cache.txt            # when --cache ran
│   └── historical.txt       # when --historical ran
└── profiles/
    ├── engine.test          # one shared test binary (all targets profile the same package)
    └── <Target>/            # SimLoop, Export, DataGeneration,
        │                    #   PaymentProcessorExport, SimLoopPartitioned
        ├── bench.txt            # this profile's own benchmark result lines
        ├── cpu.prof
        ├── mem.prof
        ├── cpu.callgraph.svg    # human viewer (rendered via Graphviz `dot`)
        ├── cpu.callgraph.dot    # DOT source for analysis tools
        ├── cpu.top-cum.txt
        ├── cpu.labels.txt       # phase / partition label sets
        ├── cpu.phase-step.txt
        └── mem.alloc-space.txt

meta.json records the commit, branch, dirty flag, wall-clock, host, os/arch, Go version, CPU count, benchtime, the flags passed, and a per-stage status + duration. index.html is the human entry point: a self-contained, click-to-navigate page with the environment and stages tables, the parsed topline / scaling / throughput tables, and — per profile — a description of what was profiled, the embedded call graph, links to every artifact, and an interactive-flame command. The five targets share one compiled profiles/engine.test rather than a copy each.

Viewing the profiles

Each profiled Target — SimLoop, Export, DataGeneration, PaymentProcessorExport, SimLoopPartitioned — carries in index.html a short description of what is being profiled (the scenario, the seed/turn scale, and whether logging or export is on) and a summary data table of its own measured numbers (the bench.txt behind the graph), so a profile is never read out of context. The suite renders the call graph and the -top -cum table to disk so they can be read without launching anything: open profiles/<Target>/cpu.callgraph.svg (rendered via Graphviz dot) and cpu.top-cum.txt. The call graph is also written as cpu.callgraph.dot — the same graph as DOT source, so DOT-analysis tools can consume it and it can be re-rendered at any size. The coarse phase / partition labels are always present in the profile; cpu.labels.txt and cpu.phase-step.txt carry the label slices.

Flame graphs are interactive, not on disk

Go’s pprof cannot render a static flame graph headlessly, so index.html carries a ready-to-paste command per profile that opens the interactive Flame Graph view in a browser (the five targets share one profiles/engine.test):

go tool pprof -http=: profiles/engine.test profiles/SimLoop/cpu.prof

The finer machine / step subphase labels appear only when the run was profiled under --deep (which sets the profile build tag).

The historical stage

The historical stage answers “how has sim-loop throughput changed across versions?” by running one identical workload at every merged PR’s merge commit (PR#7 → present, in merge-date order) and at HEAD and printing a table whose headline column is ns/entity/turn — the cost of stepping one live entity for one turn. It is a throughput measurement, not a profiling one, and it isolates engine changes: no profile is collected, and every measured worktree has its runtime/pprof goroutine-label calls neutralised to no-op shims before building, so versions whose label placement differed compare on engine work alone (label sets are allocated and merged on every pprof.Do call whether or not a profile is being collected, so without the shim a row would measure that version’s instrumentation choices as well as its engine). Locating where cost sits inside the code is the profile stage’s job. The workload keeps every entity active for the whole run (nothing passivates), so the per-entity figure is undiluted by a passive pile — unlike the topline, whose fixtures retire entities by design. It is opt-in because it builds the engine once per measured version:

# Runs the default battery, then the historical stage (~60 versions, one
# engine build each — expect the better part of an hour)
go run ./tools/bench/suite --historical

# Measure a subset of versions (labels from histVersions)
HIST_ONLY="PR#10,PR#63,HEAD" go run ./tools/bench/suite --historical

# The table lands in the run folder and in the report
cat tools/bench/runs/<YYYYMMDD>-<shortSHA>/results/historical.txt

Today’s benchmark file cannot simply be checked out at an old commit — the engine API has drifted (the engine split moved pkg/fsm into internal/engine/, the pre-archetype API differs again, and signatures drifted within each era). The stage (tools/bench/suite/historical.go) absorbs that drift by probing each worktree’s source for its API shape and generating the benchmark from a per-era template in tools/bench/historical/. For each measured version it:

1. creates a detached throwaway worktree at the version’s merge commit (under a temp dir; nothing is committed, and every worktree is torn down afterwards);
2. probes the API era — package layout (pkg/fsm vs internal/engine), SeedAll presence, NewSimulation/NewExperiment arity, Run signature — and renders the era’s template with the right placeholder substitutions;
3. drops in the era’s frozen FixtureLegacy scenario YAML and the rendered benchmark as zz_hist_bench_test.go;
4. neutralises pprof: every non-test source file importing runtime/pprof has its label calls (Do / Labels / WithLabels) redirected to no-op shims, so the row measures engine work, not that version’s label placement;
5. runs go test -bench '^BenchmarkHistorical$' -benchtime 10x inside the worktree — the benchmark drives the run turn by turn and sums the live population after every turn into an entityturns metric (turn-by-turn driving is cumulative at every measured era, so it steps the same turns as one batch call);
6. parses ns/op and the reported entities / entityturns metrics into one table row: ns/entity = ns/op ÷ entities and ns/entity/turn = ns/op ÷ entityturns. A version that fails to build or run becomes an explicit failed row, with its output in results/historical-fail-<label>.log — gaps are reported, never silently skipped.

Asset	Role
suite/historical.go — histVersions	The measured versions: every merged PR from PR#7 to the present (merge-date order) plus HEAD, as label + merge commit only — everything else is probed. To measure another commit, add a row here. Each merge commit also carries a local bench/pr-N tag for navigation.
bench_pr7_test.go.tmpl	Pre-SeedAll pkg/fsm era: SeedInstances + InstanceCount(), with the probed NewSimulation arity substituted for __NEWSIM__. Pairs with the entity_order: fixture variant.
bench_modern_test.go.tmpl	Archetype-store pkg/fsm era: SeedAll + Store.Count, with __NEWSIM__ substituted.
bench_split_test.go.tmpl	Post engine-split era (PR#50 → present): dataset.SeedAll + Store.Count, with the probed constructor (NewSimulation/NewExperiment, with or without flushing) substituted for __NEWEXP__ and the probed run method (RunSimulation/Run, with or without a request arg) for __RUN__. A drift the probes don’t cover shows up as a failed HEAD row, never a silently wrong one.
FixtureLegacy (internal/engine/testdata/FixtureLegacy/)	The workload: a spawn-only scenario written to the lowest-common schema so it parses on every measured loader back to PR#7. 50 seeded parents each spawn an inert child with probability 0.5 per turn, 100 turns → ~2,500 entities, all permanently active. FixtureLegacy.pr7.yaml is the pre-archetype copy carrying the entity_order: key that era required.

Adding a version to the series

The measured set is the histVersions slice in tools/bench/suite/historical.go. To extend it, append one {"PR#N", "<shortSHA>"} row in merge-date order, before the trailing {"HEAD", "HEAD"} anchor. Find each commit with gh pr list --state merged --json number,mergeCommit,mergedAt — mergeCommit.oid is the commit that landed and is correct even for fast-forward merges (where there is no separate merge commit). Only merged PRs get a row; work that went straight to staging without a PR has none and needs none, since the HEAD anchor always captures the cumulative tip. Everything else about a version — package layout, API era, constructor and run-method shapes — is probed from the worktree, so a row is just label

• commit.

Excerpt of a real series (illustrative numbers — measure on your own hardware; the full table has one row per merged PR):

version             ns/op   entities   entity-turns    ns/entity   ns/entity/turn
-------  ---------------- ---------- -------------- ------------ ----------------
PR#7              1573371       2474         126625        636.0             12.4
PR#10             1415542       2526         129670        560.4             10.9
PR#18             5403108       2596         132036       2081.3             40.9
PR#25             3779996       2509         127649       1506.6             29.6
PR#42            15319333       2511         128455       6100.9            119.3
PR#44             5001038       2565         131735       1949.7             38.0
PR#45             3045275       2484         128747       1226.0             23.7
PR#57             2398279       2527         128089        949.1             18.7
PR#62             2929208       2540         131452       1153.2             22.3
PR#63             4321625       2469         127938       1750.4             33.8
PR#75             3507117       2541         130478       1380.2             26.9
HEAD              2769888       2503         126150       1106.6             22.0

This ns/entity/turn is the undiluted per-live-step cost

The historical column divides by measured entity-turns — the sum of the live population over every turn — and in this fixture every entity is live (stepped) every turn. The topline’s same-named column divides by the cumulative population × turns, and its fixtures passivate most of that population by design, so a change in per-live-step cost is diluted there. Compare historical rows against each other; treat the topline column as a cross-benchmark throughput index, not a per-step cost.

Performance / capacity

Beyond the in-process benchmarks, the suite stages and a pair of cloud scripts answer capacity questions: how fast the engine generates data to disk, how its time and memory scale with population, and where the hot loop sits in the cache hierarchy. The analyses behind them live in ai/ram-cache-perf.md (cache & memory) and ai/flush-passive-to-disk.md (larger-than-memory generation).

Numbers are hardware-specific — measure, don’t quote

Every recipe below is a repeatable command, not a figure. Throughput, cache residency, and bytes-per-entity all depend on the host, so run the command on your target hardware rather than trusting a number from someone else’s.

The stages

Stage / tool	What it reports
throughput stage	On-disk entity-generation throughput by wall-clock budget — grows the turn count until a run hits each budget, then reports rows-on-disk (read back from the Parquet footers after fsync), rows/sec, and ns/row, for one-partition (FixtureBenchmark) vs four-partition (FixtureBenchmark4). The --demo50m flag runs the bounded-memory acceptance harness for the flushing path.
profile stage	pprof CPU + heap capture per target, sliced by phase / partition by default and additionally by machine / step subphase under --deep (-tags profile). Call-graph SVGs and -top tables land on disk.
scaling stage	Turn-horizon sweep: ns/live/turn (flat ⇒ per-turn cost is O(live)) vs ns/any/turn (falls as the retired pile grows), plus residentMB under flushing (flat ⇒ memory bounded by the live set).
--cache stage	Cache behaviour the throughput metrics cannot see. Always prints the OrderFixed vs OrderRandom ns/entity delta (the worklist-shuffle locality cost, no PMU needed); when the host exposes a PMU it also reads real L1/L2/LLC counters via perf / cachegrind.
--historical stage	Cross-version throughput table (normalised to ns/entity/turn on an all-active workload, no profiling) for the sim loop at past merge commits and HEAD, built in detached throwaway worktrees. See the historical stage for how it absorbs API drift.
tools/bench/cloud/run-1b-{aws,gcp}.sh	A separate massive-scale tool. Provision a 200 GB cloud instance, generate ~1 billion entities to disk via the flushing binary, count the rows with DuckDB, and tear the instance down. One script for AWS EC2, one for GCE.

Answering capacity questions

How many rows can I generate and save to disk in 1 second?

Run the suite — its throughput stage grows the turn count until a run hits each wall-clock budget. The reported rows-on-disk is read from the Parquet footers after every file and the directory are fsynced — durably on disk and readable by a separate process, not an in-memory count.

go run ./tools/bench/suite            # the throughput stage runs in the default battery
go run ./tools/bench/suite --quick    # a single throughput budget for a fast read

The throughput report columns are budget · fixture · parts · seeds×turns · wall-clock · rows-on-disk · rows/sec · ns/row. Read rows-on-disk at the 1s budget; compare the one- and four-partition rows to see the parallel speedup.

How long does it take to checkpoint a 10M-entity experiment?

TBD — checkpoint timing is not yet measurable end-to-end

The on-disk checkpoint format exists (internal/engine/store/checkpoint/, CON-772): a flat store serialises to per-machine Arrow IPC files plus a manifest.json, and checkpoint.Write / checkpoint.Read round-trip every column type. What is missing for a 10M timing:

• No facade or CLI trigger — there is no Experiment method or flag to checkpoint a running experiment (blocked on CON-774).
• No timing benchmark — nothing times Write/Read yet.
• Segmented stores are rejected (ErrSegmentedUnsupported), so an experiment run with --flush — the path that reaches tens of millions of entities — cannot be checkpointed at all yet (blocked on CON-775).

Repeatable method, once a benchmark is added: a flat-store micro-benchmark that seeds N entities into an ArchetypeEntityStore, then times checkpoint.Write(st, cfg, dir) and checkpoint.Read(dir, cfg) around b.ResetTimer(), swept over N = 1e5 … 1e7. This is feasible today for a flat store that fits in RAM; end-to-end checkpoint/resume timing of a flushed 10M-entity run waits on CON-774 + CON-775.

How many entities can I fit in L2 cache?

The store is struct-of-arrays, so the unit that occupies cache is not an entity but a column cell: every hot-loop field is a fixed-width 8-byte value (int64 / float64; dimensions are 4-byte DimensionIdx), so 8 (or 16) rows fit in one 64-byte cache line, per column. The cache-resident set is therefore the live working set, not the population:

live_working_set_bytes ≈ live_entities × touched_columns × 8 bytes

One full 65536-row column segment is 65536 × 8 B = 512 KB, which exceeds L1 (32–48 KB) and most L2 (256 KB–2 MB) — so a column streams rather than sits resident. Because the loop sweeps columns sequentially, only ~10 cache lines (one per active column stream) are hot at once, so the live set stays L1/L2-resident regardless of how large the population grows. To size it for a given L2, divide L2 by touched_columns × 8. To measure the locality cost of the per-turn shuffle directly, run the cache stage — it prints the Order A/B delta and, where a PMU is present, real counters:

go run ./tools/bench/suite --cache   # OrderFixed vs OrderRandom ns/entity delta + L1/L2/LLC counters when available

TBD — real L1/L2/LLC miss counters need a PMU

The --cache stage reads hardware counters via perf (Linux bare metal), cachegrind (simulated), or xctrace (macOS Instruments). Containers and VMs — including this sandbox — almost never expose the PMU, so perf reports <not supported>; real counters require bare-metal Linux or native macOS. The Order A/B delta the stage always prints needs no PMU and runs anywhere.

How many entities can I run in 1 GB / 4 GB of memory?

This is the non-flushing, hold-everything-in-memory case: every activated entity stays resident, so capacity ≈ RAM ÷ bytes_per_entity. The byte cost per entity depends on its column count — measure it with the scaling stage (it prints resident heap per horizon) and divide:

go run ./tools/bench/suite   # read residentMB against the entity count per horizon in the scaling report

With a measured B bytes/entity, 1 GB holds 2^30 ÷ B entities and 4 GB holds 2^32 ÷ B — four times as many, since capacity is linear in RAM. The figure is config-specific: a wide entity (many counters/dimensions/measures) costs more bytes per row than a narrow one, so measure against the config you intend to run.

How long does it take to generate 1 billion rows?

A billion rows is a cloud-scale run — tens of GB of Parquet over minutes — so it is driven by a script that provisions a 200 GB instance, runs the static binary with flushing on (resident memory stays bounded by the live set, so the run never has to hold a billion entities at once), counts the rows on disk with DuckDB, and tears the instance down. One script per cloud:

Cloud	Script	Run it
AWS EC2 (c7i.2xlarge)	tools/bench/cloud/run-1b-aws.sh	bash tools/bench/cloud/run-1b-aws.sh # or: ... --region eu-west-1
Google Compute Engine (c3-standard-8)	tools/bench/cloud/run-1b-gcp.sh	bash tools/bench/cloud/run-1b-gcp.sh # or: ... --project my-proj --zone us-central1-a

Both scripts run against your existing cloud login — AWS uses your configured credentials and default region; GCP uses your active gcloud account and default project (gcloud config get-value project). Nothing else has to pre-exist: gcloud manages SSH keys for you, and the AWS script creates a throwaway key pair and a security group (SSH from your current IP only), deleting both on exit. Override the target with --project / --zone (GCP) or --region (AWS); to reuse your own AWS key/SG, set AWS_KEY_NAME + AWS_SSH_KEY and/or AWS_SG_ID.

Each script then builds the CGO_ENABLED=0 static binary locally (stripped with -ldflags="-s -w" and gzipped for upload — ~51 MB → ~35 MB → ~10 MB on the wire, decompressed on the instance), ships it with the FixtureBenchmark spawn-and-retire config, and runs ./mock_machines -flush -run <turns> -seedCount <n> -runDir ~/data on the instance. The defaults reach ~1.3B rows (rows ≈ 0.52 × seeds × turns for this config); pass TURNS and SEEDS as positional args to change the target, and set KEEP=1 to leave the instance running for inspection.

Checking your setup. Before launching anything, confirm your credentials resolve and which account/project the script will target. These calls are read-only and cost nothing — run them first so the script does not fail partway through (or after a billable instance is already up):

# AWS — caller identity and the default region the script will use
aws sts get-caller-identity
aws configure get region

# GCP — the authenticated account and the default project the script will use
gcloud auth list
gcloud config get-value project

Seeing the result. The script runs in your local terminal and streams each step (build → launch → copy → generate → count) as it goes. When the run finishes it times the generation and prints a results block — rows on disk, wall-clock, throughput — to your terminal and writes it to working/1b-rows/1b-result-aws.txt / working/1b-rows/1b-result-gcp.txt. Only then does it tear the instance down, so the answer is captured locally before anything is destroyed (the per-second cloud cost is why the instance does not linger). Example:

==> generating (~1B rows: 10000 turns × 250000 seeds, flushing on)
==> counting rows on disk
================ 1B-row generation result — AWS EC2 c7i.2xlarge ================
  rows on disk : 1312400000
  wall-clock   : 2280s (38m 0s)
  throughput   : 575614 rows/sec
  config       : FixtureBenchmark, flushing on, 250000 seeds × 10000 turns
================================================================================
==> result saved to working/1b-rows/1b-result-aws.txt (instance is torn down next)

Illustrative output — the actual rows, wall-clock, and throughput depend on the instance and disk.

Why the binary, not the suite throughput stage

The throughput stage resolves its fixtures via runtime.Caller against the source tree, so it needs the repo and the Go toolchain present. The mock_machines binary takes the scenario YAML as an argument and supports -flush directly, so a single static file plus one YAML is all the script ships to the instance.