Testing

Three layers of confidence: a broad base of fast unit tests, a build-tagged integration suite that runs whole experiments, and scaling benchmarks for the columnar store. On top sits a benchmarking surface — the on-demand benchmarking suite, which drives the benchmarks into per-commit run folders and answers capacity questions (rows/second to disk, entities per GB, cache residency). All of it is fed by minimal, single-purpose fixtures. Two generated reports — code coverage and a capability domains — summarize test health from complementary angles.

Counts are exact at time of writing (go test function counts across internal/engine).

Running the tests

# All unit tests (the ... wildcard covers every engine subpackage)
go test ./internal/engine/... -v

# A single test by name (runtime package)
go test ./internal/engine/runtime/ -run TestEvaluateConditions -v

# The build-tagged integration suite (every auto-discovered scenario, run end-to-end)
go test -tags=integration ./internal/engine/enginetest/ -run TestSimulationScenarios -v

# Benchmarks (store hot path + sim-loop / scaling)
go test ./internal/engine/... -bench=. -benchmem

# The on-demand benchmarking suite — topline, profiles, scaling, throughput (see Benchmarking)
go run ./tools/bench/suite

# An end-to-end experiment smoke test
go run main.go -run 50 scenarios/PaymentProcessor/PaymentProcessor.yaml

Baseline before changing the FSM package

Run go test ./internal/engine/ -v and go run main.go -run 50 scenarios/PaymentProcessor/PaymentProcessor.yaml first to establish a green baseline, then again after your change.

Two views of test health

Running the suite tells you it passes; two generated HTML reports tell you what it covers, from complementary angles. Both write to the project root.

Report	Generated by	What it shows
Code coverage	go tool cover	Which lines and functions the suite executed — a per-function breakdown and an interactive heatmap. Output: coverage.html.
Capability domains	tools/capabilities	Which capabilities are validated — a curated inventory mapping feature domains to their tests, fixtures, and scenarios, surfacing gaps that line coverage cannot. Output: capability-domains.html.

Code coverage

Go’s built-in coverage tooling shows which lines and functions the suite exercised. Generate the profile and the HTML heatmap:

go test ./internal/engine/ -coverprofile=cov.out
go tool cover -html=cov.out -o coverage.html

Open coverage.html: covered lines are green, uncovered red, with a file-by-file dropdown. For a quick read in the terminal:

Command	Output
go test ./internal/engine/ -cover	One-line percentage per package
go tool cover -func=cov.out	Per-function coverage table
go tool cover -func=cov.out | sort -t: -k3 -n	Sorted by percentage (gaps first)

Capability domains

Line coverage shows which code ran; the capability domains report shows which capabilities are validated. go run ./tools/capabilities (also wired into go generate ./...) renders it from a curated inventory — capabilities.yaml for capabilities and validations.yaml for the validation-rule catalog — cross-referenced against the tests, fixtures, and scenarios that exercise each domain:

go run ./tools/capabilities            # writes capability-domains.html in the project root
go run ./tools/capabilities --check    # CI / pre-commit drift gate (fails on orphan or stale refs)

The report has four sections:

Section	What it shows
Capability Domains	Each domain (Scheduling, Taxonomy, Partitioning, …) with test / fixture / scenario counts and a strength rating; expand a domain to see its capabilities and the exact test names.
Scenario × Domain Matrix	Which scenarios exercise which capability domains.
Scenarios	Each scenario config with its description and the domains it covers.
Known Gaps	Capabilities with no or insufficient coverage.

Each domain’s strength rating is derived from its inventory:

Rating	Criteria
Strong	≥ 3 tests, plus at least one fixture or scenario
Adequate	≥ 3 tests
Weak	1–2 tests
Untested	no tests

Maintaining the inventory

The inventory lives in tools/capabilities/capabilities.yaml. When you add tests, fixtures, or capabilities, update it and regenerate. It has three top-level keys:

Key	Purpose
domains	Capability domains — each with a name, slug, description, and capabilities; each capability lists its tests, fixtures, and scenarios.
scenarios	Scenario configs with the domain slugs they exercise.
gaps	Known coverage gaps with explanatory notes.

The binding from a capability (or a validation rule) to the test or fixture that validates it lives in code as a // @capability domain=… name=… annotation (# @capability … in a fixture’s YAML), so it travels with the code and survives renames. Validation rules work the same way: the rule catalog is validations.yaml, and each enforcement site carries a // @validation rule=… annotation. A catalog entry with no annotation renders as a gap; an annotation naming an unknown id is an orphan that --check rejects.

$EDITOR tools/capabilities/capabilities.yaml   # edit the inventory
go run ./tools/capabilities                    # regenerate capability-domains.html
go run ./tools/capabilities --check            # fail on drift (orphans / stale refs)

Unit tests by domain

Tests are colocated with their source across the engine’s subpackages (runtime/scheduler_test.go beside runtime/scheduler.go, store/store_archetype_test.go beside store/store_archetype.go, and so on) and lean on the stdlib testing package — table-driven where inputs vary, round-trip where there’s a serializer, statistical where there’s sampling. The main functional domains are below; the full authoritative inventory — including OSI, checkpoint/resume, collections, history, timestamps, the REST API, and MCP — lives in capabilities.yaml.

Domain	File(s)	What it covers
Finite State Machines	runtime/fsm_test.go, runtime/eval/selector_test.go	State transitions, SelectEvent probability sampling (stay, always-fire, age-clamp, empty-state), pre-selection condition masking + renormalization, age reset on transition.
Condition Evaluation	runtime/eval/selector_test.go, enginetest/evaluators_test.go	Type-aware comparisons (int is/atLeast/atMost, float with epsilon, is/is_not), AND logic, per-condition diagnostic results.
Probability & Distributions	model/distributions_test.go	Normalization (epsilon scaling, largest-gets-remainder, scale-down, immutability), YAML unmarshaling of mixed int/float values.
Taxonomy Sampling	enginetest/taxonomy_test.go, compile/load/value_distribution_test.go	All five sampling strategies (distribution, conditional, lookups, weights, options — 10k-sample frequency checks), dependency-order DAG, circular-dep detection, inherit from parent.
Entity Spawning & References	runtime/spawn_policy_test.go, runtime/rt_spawn_integrity_test.go, compile/validate/validate_spawn_test.go	Child entity creation (new_entity), referential integrity (child → parent), spawn-policy population sizing, spawn-drop logging, unused reference slot detection.
Message Passing	runtime/messages_accepted_test.go, runtime/cascade_fanin_test.go, runtime/scheduler_test.go	Immediate delivery (cascade queue), delayed delivery (time-wheel ordering/drain), cascade-loop detection, the unified handler-selection model.
Scheduling & Clock Modes	runtime/scheduler_test.go, compile/load/clock_override_test.go, model/dwell_test.go	Tick and event clock advancement, the wake queue, dwell sampling, regime validation (valid/invalid combos), --clock override, event-log timestamp enrichment.
Synchronous Updates	runtime/scheduler_test.go, runtime/partial_turn_test.go	Snapshot reads (int, float, string), deferred state/reference updates, delta accumulation (AddInt), commit-then-reveal end-to-end.
Partitioning & Parallelism	compile/plan/planner_test.go, runtime/rt_isolation_test.go, runtime/rt_dimension_partition_test.go	Hazard detection (spawn, message, reference edges), independent-machine identification, partition building, cross-partition routing, partition isolation, concurrent executor correctness.
Entity Storage	store/store_archetype_test.go, store/segment_test.go	Entity creation and ID generation, creation with references, pending lifecycle, iteration/lookup, state/field access, snapshot/commit, segmented (flushed) stores.
Configuration Loading	compile/load/loader_test.go, compile/load/convert_test.go, compile/load/schema_test.go	Type coercion (int/float/string), field flattening and mapping, condition/action linking, indirect path compilation, cycle detection (spawn/message).
Validation	compile/validate/validate_test.go, compile/validate/validate_reachability_test.go	The rule families — messaging, reachability, spawn, taxonomy, passivation, action-type, and payload-condition validators.
Event & Decision Logging	runtime/eventlog_test.go, runtime/decisionlog_test.go	Entry builders for every action type, JSONL round-trip, full-experiment integration, partition-aware logging, debug-gated decision traces.
Seeding & Import/Export	enginetest/seed_test.go, runtime/import_test.go, store/dataset/export_test.go	Seed reference parsing, seedCount fallback, CSV/Parquet round-trip (write → load with references and nextID), seed-schema validation.
Diagnostics & Dump	compile/diagnostics/dump_test.go, compile/diagnostics/report_data_test.go	The --dump JSON (pointer-to-name marshaling, matrices labelled) and the --analyze report (parallelism verdict, hazards, scalability prediction).
Checkpoint & Resume	store/checkpoint/roundtrip_test.go, runtime/checkpoint_test.go	Per-machine Arrow IPC + manifest round-trip across every column type, segmented-store handling, resume rehydration and history preservation.
OSI semantic layer	store/dataset/osi_test.go	Config → OSI generation rules, curated-overlay merge, and metric validation (bind/execute tiers + literal cross-check).
Collections & aggregation	runtime/collections_test.go, runtime/aggregate_test.go	Collection membership via back-references, and the aggregate fold (count/sum/max/min) with its type rules.
REST API	internal/api/*_test.go	Scenario deploy/list/get/delete, experiment lifecycle, async run + targeted requests, entity/event/dataset reads, export, checkpoints, CORS, and admin gating.

Fixtures & testdata

Scheduling and partitioning behaviours are tested against dedicated, minimal configs — each isolating one concern — under internal/engine/testdata/, following the Fixture<Scenario> naming convention. A fixture directory holds a mock-machine.yaml and any seed CSVs (named by machine type). The richer scenarios/* configs are reserved for end-to-end coverage.

Fixture	Concern it isolates
FixtureImmediateMessage	Same-turn message delivery via the cascade queue.
FixtureDelayedMessage	A delay defers delivery via the time wheel (proves the multi-tick lag).
FixtureCascadeLoop	A request/reply loop trips the cascade-loop guard.
FixtureEventClock	Multi-rate entities under the event clock (Fast.csv / Slow.csv).
FixtureIndependentPartitions	Disjoint machines partition into independent groups.
FixtureConnectedGroup	Machines linked by an edge stay in one partition (serialized).
FixtureMultiPartitionMessages	Messaging across multiple partitions (Ping/Pong/Click/Clack).
FixtureScanEdge	A set_reference population scan creates a reference edge.

Integration suite

The integration test file internal/engine/enginetest/simulations_integration_test.go is gated behind //go:build integration, so the default go test skips it. TestSimulationScenarios auto-discovers every scenario via mock_machine.DiscoverScenarios over scenarios/ — adding a new scenarios/<Name>/<Name>.yaml enrols it automatically, no edit to the test — and runs each whole experiment end-to-end: load → seed from files (integrationSimCount = 20) → run (integrationTurns = 10) → WriteCSV to a temp dir → assert the total entity count is greater than zero.

The discovered set is whatever lives under scenarios/ — currently twenty-two. A representative sample:

Scenario	Exercises
PaymentProcessor	Taxonomy-heavy seeding, references, multi-stage spawn chains.
ProgrammaticAds	Deep spawn fan-out (campaigns → ad slots → bid requests → impressions → clicks).
ServiceQueue	Queueing with spawned tellers/tickets/sessions.
OnlineShopping	Catalog, field families, message handlers, set_reference scans.
PrisonersDilemma	Self-referential players under the sync regime (commit-then-reveal).
RefusalDemo	Requested events masked by failing conditions (a refusal is a no-op, not a fallback).
CliffWalking, PageRank, SyncReveal	Reinforcement-learning grid walk (4×12 cliff grid), iterative web-graph ranking, and synchronous update revealed through StepTurn.
JaffleShop, LondonUnderground, PacMan, GameOfLife	Timestamp fields (a dbt-style orders dataset), transit-graph routing, a reactive-policy grid agent, and a generated cellular automaton.

Benchmarks and performance/capacity testing — the on-demand benchmarking suite, profiling, scaling sweeps, and cloud-scale generation — now live on the Benchmarking page.

Shared test helpers

Helper	Role
yamlPath(t)	Resolves scenarios/PaymentProcessor/PaymentProcessor.yaml relative to the test file via runtime.Caller. Variants exist for other fixtures (delayedEchoPath, syncRevealPath, dumpYamlPath, the bench paths).
loadTestConfig(t)	Loads and fatals on error — the common setup for engine tests.
slotParents(machine, parents)	Converts a readable map[string]EntityHandle of parents into the slot-indexed array CreateEntityWithRefs expects.
writeSeedCSV(t, …) / buildMinimalConfig()	Construct temporary seed files and minimal configs for seed/import tests.
splitJSONL(data)	Splits a JSONL log into lines for assertion.

Creating entities in tests

Build entities directly in the archetype store with store.CreateEntity(machine, machine.InitState) (or CreateEntityWithRefs via slotParents when an entity needs its reference slots wired).