Markov Decision Processes

Mock Machines’ core loop — states, probability matrices, condition-gated events, counters — is already most of a Markov Decision Process substrate. This article maps each MDP and POMDP concept (state, action, transition, reward, discount, observation) onto scenario YAML, names the patterns that make the mapping work, and is honest about what the engine cannot express in-config and how to work around it. Four scenarios serve as worked examples: CliffWalking, RecyclingRobot, TigerPOMDP, and Thermostat.

State

An MDP state in Mock Machines is the product of two layers:

• The FSM state (states:) — the discrete control state. Use it for the qualitative component of the MDP state (battery High/Low, Listening/Deciding) and for sequencing within a step (see the two-stage pattern below).
• Fields — counters (int), measures (float), taxonomy (categorical), dimensions (strings) — the factored, quantitative component (temperature, inventory, accumulated evidence).

A third encoding is references-as-position: CliffWalking’s Walker holds its grid position not as coordinates but as a current_cell reference into a graph of GridCell entities, and moves by re-pointing it (set_reference with an indirect value path like self.current_cell.north). Use this when the state space has structure worth materialising as entities.

Action and policy

The events of the current FSM state are the action set A(s). State-dependent action spaces (A(s) ⊆ A) are covered in depth in the companion article State-Dependent Action Spaces: pre-selection conditions mask ineligible events and the engine renormalizes the remaining probability mass.

How the action gets chosen is the policy, and there are four encodings:

Encoding	Policy class	Example
The state’s probabilities: rows	One fixed stochastic policy π(a|s), age-indexed	Every example scenario
policy_tables: + policy_field	Per-entity stochastic policies keyed by a taxonomy value — a small population of competing policies in one run	CliffWalking (Safe/Cautious/Risky), RecyclingRobot (Bold/Timid)
decision: first_valid_wins	Deterministic rule policy: condition order is priority order	Thermostat’s threshold rule, TigerPOMDP’s open-or-listen
--request <event> --target <Machine> (and the facade’s targeted-request run)	An external controller injecting one action per turn — the hook for driving an entity from an outside policy or learner	CLI stepping; REST POST .../run with a request

Policies live in the same config as the environment

Nothing in the engine separates “agent” from “environment” — both are events and probabilities. The separation is a modelling discipline: keep policy events conditioned only on what the agent may observe, and keep environment responses in their own states (next section) or message handlers.

The transition function P(s′|s,a)

A probability row chooses an event; an event’s transition action names one next state. So a single state cannot express “action a leads to s′ with probability 0.7 and s″ with 0.3”. The pattern that does is the two-stage action→outcome encoding:

Action state (the agent chooses)          Outcome state (the environment answers)
  High:                                     ForageHigh:
    policy_table: high_policy                 events:
    events:                                     - found_cans_stay_high  → High   (+5)
      - search  → ForageHigh                   - found_cans_battery_low → Low  (+5)
      - wait    (+2, stay)                    probabilities:
                                                - [0.0, 0.7, 0.3]   # P(s'|High, search)

The outcome state’s age-0 row is the row of the transition matrix for that (s,a) pair, and the reward rides on the outcome events. RecyclingRobot is the worked example. Three refinements:

• Conditional kernels: conditions on outcome events make P(s′|s,a) depend on fields — the engine masks failing events and renormalizes the rest (the same machinery as SDAS pre-selection).
• Time-inhomogeneous dynamics: probability rows are age-indexed (one row per turn spent in the state, last row clamps), so P can depend on sojourn time — a semi-Markov flavour. Age resets on every transition.
• Step-rate accounting: a two-stage step takes two engine turns. When comparing policies inside one scenario this cancels; when comparing across scenarios, normalise by turns.

Reward and discount

Rewards are update actions with operator: add on a counter or measure (CliffWalking’s total_reward). Keep distinct reward channels in distinct fields (Thermostat’s comfort_reward vs energy_cost) so objectives can be reweighed offline.

The honest limitation: no in-config discounting

Update values are constants, payload reads, or distribution draws — there is no field×field arithmetic, so γ^t·r cannot be accumulated inside the run. Two workarounds:

1. Offline discounting. Every entity carries the system TurnNumber column and the event log records every reward update with its turn, so the discounted return Σγ^t·r_t is one windowed query over the export or event log. This is the recommended route — it also lets one run be evaluated under many γ values.

2. Geometric-termination equivalence. Route probability mass (1−γ) per turn to a terminal event. The expected undiscounted return of the modified process equals the γ-discounted value of the original — expressible today with plain probability rows, at the cost of shorter episodes.

Observation: modelling POMDPs

Partial observability is one engine feature (bounded history, next section) plus two config disciplines:

• Hidden state = fields that policy events never condition on. Environment events (emissions, payoffs) may read them. In TigerPOMDP, tiger_position is sampled at spawn and read only by the emission events and the door-outcome split. For stronger separation, hold hidden state on a separate entity and reach it via references or messages — but the single-entity form reads better.
• Emission distributions O(o|s) = a stochastic state whose events are condition-gated on the hidden value. TigerPOMDP’s Listening row [0, .425, .075, .425, .075] crossed with tiger_position conditions renormalizes to 0.85/0.15 per hidden value; the selected event writes an observation code into an obs counter. The same shape generalises to any finite observation alphabet.

The policy then conditions on observations and their history, never on the hidden field — a finite-memory (window) policy, the standard practical proxy for the belief state.

Bounded history: the memory feature

A POMDP agent needs memory. Declaring history: K on a field keeps that entity’s last K end-of-turn values addressable, and lag: k on a condition reads the value as of the end of turn t−k:

fields:
  counters:
    - { name: obs, init: 0, history: 2 }
...
conditions:
  - { field: obs, lag: 1, operator: is, value: 2 }   # last turn's obs
  - { field: obs, lag: 2, operator: is, value: 2 }   # the turn before

Mechanics, in brief: the loader expands K hidden ring-slot fields (obs@h0…) plus one HistoryHead push counter per machine — ordinary archetype columns, so the columnar store, snapshots, checkpoints, and exports carry them with no special handling. At every turn open the engine copies each history field’s current value into slot head % K and increments the head; a lag-k read is slot (head−k) mod K. Rotation is index arithmetic over preallocated columns: nothing shifts, nothing allocates per turn. Before K pushes have happened a lag read returns the field’s init value (fail-closed-by-init). Depth is capped at 64; history on passive machines is rejected at load (it would never advance); history requires the tick clock — under the event clock an entity steps only when due, so the head would count steps rather than turns and lag: k would no longer mean “k turns ago”, so the combination is rejected at load (and a run-scoped --clock event override is ignored with a warning); lag applies to direct self fields only.

Note

Recording actions and FSM states into history. The feature records fields, not events or FSM states — but a mirror field subsumes both: have each action’s event also set a code into a counter (as TigerPOMDP’s emissions do for observations), declare history: K on it, and “the last K actions/states” become ordinary lag reads. This is why no separate prev_state built-in exists.

The no-engine-change alternative. Before this feature, a manual ring could be built from a field family (obs_0..obs_K) plus a pointer counter rotated by update actions with bracket expressions. It still works, but costs per-turn action boilerplate and offers no per-slot lag addressing — prefer history:/lag:.

Horizon and episodes

• Terminal states: a state with no outgoing transitions ends the process (CliffWalking’s FellOffCliff/ReachedTarget); add passive_on_entry: true to also retire the entity from the step loop (TigerPOMDP’s Opened).
• Finite horizon: --run <turns> bounds the run; age-indexed probability rows can also force progression after a fixed sojourn.
• Continuing tasks: simply provide no terminal state (RecyclingRobot); compare policies by reward rate.
• Episodic restart: transition back to the initial state to start a fresh episode inside one entity, or spawn a fresh entity per episode with new_entity.

Continuous spaces

Measures (float64 columns) give a continuous state space, and float conditions (atLeast/atMost) give threshold policies over it. Continuous dynamics come from distribution-valued updates — an update on a float field may draw its value per execution:

- { type: update, field: temperature, operator: add,
    value: { distribution: normal, mean: 1.5, std: 0.3 } }
# also: { distribution: uniform, min: A, max: B }
#       { distribution: exponential, rate: L }   # mean 1/L, always positive

Each drawn value is recorded in the event log as the update’s resolved value, so log replay still reproduces the run exactly (the engine’s no-seed reproduction convention). Per-entity continuous initial state comes from seed CSVs (Thermostat’s rooms seed at spread-out temperatures).

One boundary to be clear about: action selection remains a choice among finitely many events. Continuous action effects (a draw per execution) are first-class; a continuum of actions (e.g. “set the heater to any u ∈ [0,1]”) must be discretised into events.

The worked examples

Scenario	Class	Demonstrates
CliffWalking	Discrete MDP, deterministic moves	References-as-position, policy tables, pre-selection conditions + renormalization (the SDAS pattern), per-step and terminal rewards.
RecyclingRobot	Discrete MDP, stochastic transitions, continuing task	The two-stage action→outcome encoding of P(s′|s,a); competing policy populations (Bold/Timid).
TigerPOMDP	POMDP, episodic	Hidden-state discipline, emission distributions via condition renormalization, history:/lag: as the belief proxy, terminal passivation.
Thermostat	Continuous-state MDP	Float measures as state, threshold policies, normal-noise dynamics via distribution-valued updates, separable reward channels.

Each scenario’s own docs/index.html covers its theory and a suggested visualisation of its output data.

References

Reference	Relevance
Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., 2018 — §3 (Finite MDPs)	The MDP formalism; the recycling robot (Example 3.3) and cliff walking (Example 6.6) used by the example scenarios.
Kaelbling, Littman & Cassandra, “Planning and acting in partially observable stochastic domains”, Artificial Intelligence 101, 1998	The POMDP formalism and the tiger problem.
State-Dependent Action Spaces (this site)	The A(s) ⊆ A masking-and-renormalization machinery this article builds on for conditional kernels and emissions.