State-Dependent Action Spaces

When an entity’s available actions vary by state — a grid-world walker at an edge cell, a trader with insufficient funds, a vehicle at a dead-end — the simulation must decide which actions to offer the probability roll. This article covers the theory behind state-dependent action spaces, how existing RL frameworks solve the problem, and how Mock Machines handles it.

The problem

A standard MDP defines a fixed action space A available in every state. Many real models violate this: a grid-world cell on the north edge has no “move north” action; a poker player who has folded cannot raise; a machine that has no raw materials cannot produce. The action space is a function of state: A(s) ⊆ A.

In Mock Machines, this manifests whenever an event on a state is not always valid. The probability matrix assigns nonzero weight to every event column, but some events may be ineligible for a given entity at a given moment. The engine currently has no mechanism to remove ineligible events from the probability roll.

Concrete example

In the CliffWalking simulation, a Walker entity at the top-left corner of the grid has four movement events (move_north, move_south, move_east, move_west) each with nonzero probability. But the north and west neighbour references on that corner cell are unset — those moves are physically impossible. The probability roll can select an impossible move, and the engine silently no-ops the set_reference action while still executing subsequent actions (step counter, reward penalty, state transition).

SDAS-MDP: the theoretical framework

The formal framework for this class of problem is the State-Dependent Action Space MDP (SDAS-MDP), an extension of the standard Markov Decision Process where the set of available actions A(s) varies by state. Classical RL textbooks (Sutton & Barto) already allow for state-dependent action sets in the MDP definition, but practical deep RL literature focuses on a specific mechanism for enforcing them: invalid action masking.

Huang & Ontañón (2020) formalised the dominant approach: before sampling from the policy distribution, set the logits (or equivalently, the probabilities) of invalid actions to zero and renormalize. They proved that this masking preserves unbiased policy gradient estimates — the masked softmax yields correct gradients for policy gradient methods. This technique is used in production systems including AlphaStar (StarCraft II) and OpenAI Five (Dota 2).

In a simulation engine like Mock Machines, there is no learned policy and no gradient to preserve. But the core insight transfers directly: the probability matrix authored in YAML represents the modeller’s intended preferences between actions. When some actions are invalid, how the engine handles the invalid probability mass determines whether those preferences are preserved or distorted.

Three approaches to invalid actions

A. Pre-selection filtering with renormalization

Before rolling the dice, evaluate which events are valid. Zero out invalid events’ probability mass. Renormalize the remaining probabilities so they sum to 1. Then sample.

Effect on the probability distribution: given a Safe walker policy row [stay=0.0, north=0.40, south=0.05, east=0.45, west=0.10] at a cell where move_north is invalid:

Raw:          [0.00, 0.40, 0.05, 0.45, 0.10]   (sum = 1.0)
Filter north: [0.00,    0, 0.05, 0.45, 0.10]   (sum = 0.6)
Renormalize:  [0.00,    0, 0.083, 0.75, 0.167]  (sum = 1.0)

The relative preference between valid actions is preserved exactly. If the modeller intended east to be 9× more likely than south, that ratio holds at every cell regardless of which other directions are available.

Key property

Renormalization preserves the modeller’s intended ratios between valid actions. It answers: “given that you must choose among the actions that are available, how would you distribute your preference?”

B. Absorb invalid mass into stay

Keep original probabilities for valid events. Add invalid events’ probability mass to the stay column. No renormalization needed.

Raw:          [0.00, 0.40, 0.05, 0.45, 0.10]
Absorb north: [0.40,    0, 0.05, 0.45, 0.10]   (north's 0.40 → stay)

The walker now stays put 40% of the time at north-edge cells, even though the modeller set stay: 0.0. The absolute probability of each valid action is preserved, but the effective transition rate drops. In constrained states, the entity becomes more conservative than the authored weights intend.

This is semantically equivalent to the environment-rejection pattern: the agent proposes an action, the environment rejects it (wall bounce), and the entity stays put. Huang & Ontañón identify two problems with this approach in RL contexts:

1. Wasted samples. The agent explores actions that can never succeed at that state, burning simulation turns on no-ops.
2. Valid action suppression. In learning contexts, gradients that push down invalid actions at visited states propagate through shared parameters and suppress those same actions at unvisited states where they are valid.

Problem (2) doesn’t apply to Mock Machines (no learned policy), but problem (1) does — the simulation wastes turns on wall-bumps. More importantly, the semantic distortion of stay probability violates modeller intent.

C. Environment rejection (handler-based)

The agent proposes an action (e.g. emits a move_north message to the current cell). The environment entity (the cell) evaluates whether the move is valid and responds: move_confirmed or move_blocked. The agent’s handlers for each response execute the appropriate effects.

This maps naturally onto Mock Machines’ existing message/handler system. The CliffWalking simulation already uses a variant of this pattern: the CheckCell state evaluates the cell the walker moved to (cliff, goal, or normal). It is an environment-response pattern applied post-move.

However, for action eligibility, the handler approach has significant drawbacks:

• The agent still expends a turn on the invalid action attempt.
• The message cascade adds runtime cost (message dispatch, handler resolution, response handling).
• The YAML complexity increases substantially (attempt events, confirm/reject handlers on both entities, response handling on the agent).
• The invalid action has already been “chosen” — the approach mitigates consequences rather than preventing the choice.

When handlers are appropriate

Handlers are the right tool for modelling consequences — what happens after an action is taken (the walker steps onto a cliff cell and falls). They are the wrong tool for modelling eligibility — whether an action should be considered at all (the walker cannot move north because there is no cell to the north). In RL terms: the policy operates over the valid action space; the environment applies rewards and transitions after the action is taken.

How Mock Machines handles this: pre-selection with renormalization

Mock Machines takes Approach A. Event conditions — a list of Condition structs on each Event — are evaluated before the probability roll, not after it. The sequence in Step() is:

resolve probabilities → evaluate conditions → mask + renormalize → SelectEvent (roll) → execute actions

Events whose conditions fail have their probability column zeroed; the remaining columns are renormalized so they sum to 1, and only then is an event sampled. No new YAML fields or condition types are involved — the existing conditions on events are the pre-selection filter. (If every event is masked, the row collapses to a forced stay — a deliberate, visible outcome rather than a silent fall-through.)

Conditions are evaluated before SelectEvent: events whose conditions fail have their probability column zeroed and the remaining columns renormalized before sampling. Events without conditions always pass.

Renormalization algorithm

Given a probability row P = [p_stay, p_0, p_1, …, p_n] and a boolean pass/fail result for each event C = [c_0, c_1, …, c_n] (stay is always valid):

for i in 1..n:
    if not C[i-1]:      // event i-1 failed its conditions
        P[i] = 0

valid_sum = sum(P)
if valid_sum == 0:
    P[0] = 1.0          // all events failed → forced stay
else:
    for i in 0..n:
        P[i] /= valid_sum

The stay column is never zeroed (an entity can always stay). If all events fail their conditions and the original stay probability was zero, the entity is forced to stay — a deliberate, visible outcome rather than a silent side effect.

The is_not operator

The is_not operator is the negation of the existing is operator — a general-purpose condition operator useful across all models, not specific to grid worlds or action filtering.

conditions:
  - { field: "self.current_cell.north.kind", operator: is_not, value: blocked }

Combined with the sentinel cell modelling pattern described below, is_not enables conditions to express “the neighbour in this direction is a real cell, not a boundary” using existing field access and indirect path syntax.

Decision logging

The DecisionLogEntry records the per-event condition results and the renormalized probability row. When conditions filter events, the log captures which events were excluded and the effective probabilities used for selection.

Sentinel cells: modelling boundaries without unset references

Pre-selection conditions solve the engine problem (when to evaluate, how to redistribute probability). But grid-world edges also need a modelling solution: what do edge cells point their boundary-direction references to?

Unset references are not the answer. They are modelling errors — either fatal or non-fatal depending on context — and any mechanism that relies on detecting unset references at runtime encourages modellers to build errors into their model as a feature.

Why “sentinel”

In computer science, a sentinel value (or sentinel node) is a special value placed at the boundary of a data structure to eliminate edge-case logic. Classic examples: the null terminator in a C string, a sentinel node at the head/tail of a linked list, -1 as an end-of-data marker. Sentinels share three properties:

1. Same type as the data. A sentinel node in a linked list is a real node. A sentinel cell in a grid is a real GridCell.
2. Distinguishable by value, not by type. Code tests the sentinel’s value (e.g. kind == blocked), not its structural position or whether a pointer is null.
3. Eliminates special-case branches. The traversal logic is uniform; the sentinel absorbs the boundary condition.

All three properties apply here. A sentinel GridCell with kind: blocked is a real entity in the store, distinguishable by its kind field, and it eliminates the need for null-reference checks. The term is precise and well-understood. Alternatives like “wall cell” or “boundary cell” imply visual rendering or spatial meaning that doesn’t exist; “null cell” implies absence rather than presence.

The pattern

Add one value to the grid’s kind taxonomy:

taxonomy:
  kind:
    options: [normal, cliff, goal, blocked]

Add one sentinel row to the grid CSV. For a 4×12 grid (48 cells, indices 0–47), the sentinel is index 48:

# Row 48: the sentinel cell. References point to itself.
blocked,0,GridCell[48],GridCell[48],GridCell[48],GridCell[48]

Edge cells point their boundary-direction references to the sentinel instead of leaving them empty:

# Bottom-left corner (index 0): south and west point to sentinel
normal,-1,GridCell[12],GridCell[48],GridCell[1],GridCell[48]

Every reference in the grid is now set. The sentinel is in the Static state (no events, never stepped) and its own references point to itself. It exists purely as a typed boundary marker that conditions can read through indirect paths.

Topology comparison

Topology	Edge cell references	Sentinel needed?	Conditions fire?
Hard boundary	Point to sentinel (kind: blocked)	Yes (one per grid)	Yes — is_not blocked fails at edges
Torus (wrap-around)	Point to opposite-edge cells (kind: normal)	No	No — all neighbours pass, all directions valid

The YAML config is identical for both topologies. Only the CSV wiring changes.

YAML surface

The CliffWalking Walking state, rewritten with pre-selection conditions and a sentinel cell:

Walking:
  policy_table: movement_policy
  policy_field: policy
  events:
    - name: move_north
      conditions:
        - { field: "self.current_cell.north.kind", operator: is_not, value: blocked }
      actions:
        - type: set_reference
          reference: current_cell
          value: "self.current_cell.north"
        - { type: update, field: steps, operator: add, value: 1 }
        - { type: update, field: total_reward, operator: add, value: -1 }
        - { type: transition, state: CheckCell }

    - name: move_south
      conditions:
        - { field: "self.current_cell.south.kind", operator: is_not, value: blocked }
      actions:
        - type: set_reference
          reference: current_cell
          value: "self.current_cell.south"
        - { type: update, field: steps, operator: add, value: 1 }
        - { type: update, field: total_reward, operator: add, value: -1 }
        - { type: transition, state: CheckCell }

    - name: move_east
      conditions:
        - { field: "self.current_cell.east.kind", operator: is_not, value: blocked }
      actions:
        - type: set_reference
          reference: current_cell
          value: "self.current_cell.east"
        - { type: update, field: steps, operator: add, value: 1 }
        - { type: update, field: total_reward, operator: add, value: -1 }
        - { type: transition, state: CheckCell }

    - name: move_west
      conditions:
        - { field: "self.current_cell.west.kind", operator: is_not, value: blocked }
      actions:
        - type: set_reference
          reference: current_cell
          value: "self.current_cell.west"
        - { type: update, field: steps, operator: add, value: 1 }
        - { type: update, field: total_reward, operator: add, value: -1 }
        - { type: transition, state: CheckCell }

  probabilities:
    - [0.0, 0.25, 0.25, 0.25, 0.25]

The CheckCell state is unchanged — its conditions evaluate the cell the walker has already moved to (cliff, goal, or normal) and produce consequences. These conditions are also evaluated pre-selection, but since each age row assigns probability 1.0 to a single event, renormalization produces the same result as the current post-selection gate: the event either passes and fires, or all events are filtered and the entity stays (advancing age to the next row).

In summary

Two general-purpose mechanisms make this work, neither specific to grid worlds:

1. The is_not operator — the negation of is, a general condition operator useful across all models.
2. Pre-selection evaluation with renormalization — conditions are evaluated before SelectEvent; events whose conditions fail have their probability zeroed, and the valid events are renormalized before sampling.

No special YAML syntax is involved: the existing conditions field on events is the pre-selection filter. The sentinel cell is a modelling pattern, not an engine feature — it needs no code, only a convention in how grid CSVs are authored.

References

Source	Relevance
Huang, S. & Ontañón, S. (2022). “A Closer Look at Invalid Action Masking in Policy Gradient Algorithms.” AIIDE 2022 / arXiv:2006.14171.	Foundational paper proving that logit-level masking preserves unbiased policy gradients. Establishes renormalization as the theoretically correct approach over environment rejection.
Maiti, S. et al. (2025). “Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms.” arXiv:2603.09090.	Documents the “valid action suppression” failure mode in environment-rejection approaches — where penalising invalid actions at one state suppresses valid uses of those actions at other states.
Krasowski, H. et al. (2024). “Learning State-Specific Action Masks for Reinforcement Learning.” Algorithms, 17(2), 60.	Formalises the SDAS-MDP (State-Dependent Action Space MDP) framework and explores learning the mask function itself.
Gomes, D. & Ontañón, S. (2023). “Exploring the Use of Invalid Action Masking in RL: On-Policy and Off-Policy in RTS Games.” Applied Sciences, 13(14), 8283.	Empirical comparison of masking vs. rejection in RTS game environments. Confirms masking’s sample efficiency advantage.
Slijkhuis, M. et al. (2025). “Excluding the Irrelevant: Focusing RL through Continuous Action Masking.” arXiv:2406.03704.	Extends action masking to continuous action spaces. The discrete case analysis is relevant to probability-matrix masking.
Kanervisto, A. et al. (2022). “Inapplicable Actions Learning for Knowledge Transfer in Reinforcement Learning.” arXiv:2211.15589.	Formal SDAS-MDP definition and analysis of how action spaces transfer across related tasks.
Huang, C. (2020). “A Closer Look at Invalid Action Masking in Policy Gradient Algorithms” (blog post).	Accessible companion to the Huang & Ontañón paper with code examples and intuitive explanations.