+25 -22

CLAUDE.md

··· 50 50 51 51 ## Project Structure 52 52 53 53 - - `cm_inst.py` — Instruction set definitions (Port, MemOp, CfgOp, ALUOp hierarchy, ALUInst, SMInst, Addr) 54 54 - - `tokens.py` — Token type hierarchy (Token -> CMToken -> DyadToken/MonadToken; SMToken, CfgToken -> LoadInstToken/RouteSetToken, IOToken). Imports ISA enums from cm_inst. 55 55 - - `sm_mod.py` — Structure Memory cell model (Presence enum, SMCell dataclass, StructureMem resource) 53 53 + - `cm_inst.py` — Instruction set definitions (Port, MemOp, ALUOp hierarchy, ALUInst, SMInst, Addr) 54 54 + - `tokens.py` — Token type hierarchy (Token -> CMToken -> DyadToken/MonadToken/IRAMWriteToken; SMToken). Imports ISA enums from cm_inst. 55 55 + - `sm_mod.py` — Structure Memory cell model (Presence enum, SMCell dataclass with `is_wide` metadata flag) 56 56 - `dfasm.lark` — Lark grammar for dfasm graph assembly language 57 57 - `emu/` — Behavioural emulator package (SimPy-based discrete event simulation) 58 58 - `emu/types.py` — Config and internal types (PEConfig, SMConfig, MatchEntry, DeferredRead) ··· 81 81 - `dfgraph/frontend/` — TypeScript frontend: Cytoscape.js graph with ELK layout, SVG/PNG export 82 82 - `tests/` — pytest + hypothesis test suite 83 83 - `tests/conftest.py` — Hypothesis strategies for token/op generation 84 84 + - `tests/test_sm_tiers.py` — T0/T1 memory tier and EXEC bootstrap tests 85 85 + - `tests/test_exec_bootstrap.py` — EXEC opcode acceptance criteria tests 86 86 + - `tests/test_migration_cleanup.py` — Verifies removed types (SysToken, CfgOp, etc.) are absent from codebase 84 87 - `docs/` — Design documents, implementation plans, test plans 85 88 86 89 ## Tech Stack ··· 102 105 103 106 ## Architecture Contracts 104 107 105 105 - > **Note:** The design documents in `design-notes/` have been updated to 106 106 - > reflect the 1-bit SM/CM token split (eliminating SysToken/type-11), 107 107 - > IO as memory-mapped SM, bootstrap via SM00 EXEC, and variable 3/5 SM 108 108 - > opcode encoding. The emulator and assembler code below still uses the 109 109 - > old token hierarchy (SysToken, CfgToken, IOToken). An implementation 110 110 - > update is pending. See `design-notes/sm-and-token-format-discussion.md` 111 111 - > for the design rationale. 112 112 - 113 108 ### Token Hierarchy (tokens.py) 114 109 115 110 All tokens inherit from `Token(target: int)`. The hierarchy: ··· 117 112 - `CMToken(Token)` -- adds `offset`, `ctx`, `data` (frozen dataclass) 118 113 - `DyadToken(CMToken)` -- adds `port: Port`, `gen: int`, `wide: bool` 119 114 - `MonadToken(CMToken)` -- adds `inline: bool` 115 115 + - `IRAMWriteToken(CMToken)` -- adds `instructions: tuple[ALUInst | SMInst, ...]` 120 116 - `SMToken(Token)` -- `addr: int`, `op: MemOp`, `flags`, `data`, `ret: Optional[CMToken]` 121 121 - - `SysToken(Token)` -- `addr: Optional[int]` 122 122 - - `CfgToken(SysToken)` -- `op: CfgOp` (base class, no payload) 123 123 - - `LoadInstToken(CfgToken)` -- `instructions: tuple[ALUInst | SMInst, ...]` (contiguous block from `addr`) 124 124 - - `RouteSetToken(CfgToken)` -- `pe_routes: frozenset[int]`, `sm_routes: frozenset[int]` 125 125 - - `IOToken(SysToken)` -- `data: Optional[List[int]]` 126 117 127 118 ### Instruction Set (cm_inst.py) 128 119 129 120 - `ALUOp(IntEnum)` base with subclasses: `ArithOp`, `LogicOp`, `RoutingOp` 121 121 + - `MemOp(IntEnum)` -- read/write/atomic ops; supports EXEC, EXT, SET_PAGE, WRITE_IMM, RAW_READ for T0 tier control 130 122 - `ALUInst(op, dest_l: Optional[Addr], dest_r: Optional[Addr], const: Optional[int])` -- stored in PE IRAM 131 131 - - `SMInst(op: MemOp, sm_id: int, const, ret: Optional[Addr])` -- also in IRAM; causes PE to emit SMToken 123 123 + - `SMInst(op: MemOp, sm_id: int, const, ret: Optional[Addr], ret_dyadic: bool)` -- also in IRAM; causes PE to emit SMToken. `ret_dyadic` controls whether the return token is a DyadToken (True) or MonadToken (False) 132 124 - `Addr(a: int, port: Port, pe: Optional[int])` -- destination address 125 125 + - `is_monadic_alu(op: ALUOp) -> bool` -- canonical source of truth for monadic ALU op classification (used by `emu/pe.py` and `asm/opcodes.py`) 133 126 134 127 ### ALU (emu/alu.py) 135 128 ··· 160 153 **Output logging:** 161 154 - `PE.output_log: list` records every token emitted (for testing and tracing) 162 155 163 163 - **CfgToken handling:** 164 164 - - `LoadInstToken`: writes `token.instructions` into IRAM at `token.addr` base offset 165 165 - - `RouteSetToken`: restricts `route_table` to `token.pe_routes` and `sm_routes` to `token.sm_routes` 156 156 + **IRAMWriteToken handling:** 157 157 + - `IRAMWriteToken`: writes `token.instructions` into IRAM starting at `token.offset` 166 158 167 159 ### Structure Memory (emu/sm.py) 168 160 ··· 181 173 - Cell must be `FULL`; returns old value via return route 182 174 - CMP_SW: compares `token.flags` (expected) with current; swaps to `token.data` on match 183 175 176 176 + **Memory Tiers:** 177 177 + - **T1 (below tier_boundary):** Per-SM I-structure cells with presence tracking, deferred reads, atomic ops. Default tier_boundary: 256. 178 178 + - **T0 (at/above tier_boundary):** Shared raw storage across all SMs. No presence tracking. `list[Token]` shared by all SM instances. 179 179 + - T0 operations: READ (immediate return), WRITE (no presence check), EXEC (inject tokens from T0 into network) 180 180 + - I-structure ops on T0 addresses are errors (logged and dropped) 181 181 + 184 182 ### Network Topology (emu/network.py) 185 183 186 184 `build_topology(env, pe_configs, sm_configs, fifo_capacity) -> System` ··· 193 191 - If `PEConfig.allowed_pe_routes` or `allowed_sm_routes` is set, `build_topology` restricts routes at construction time 194 192 195 193 **System API:** 196 196 - - `System.inject(token: Token)` -- route token by type: SMToken → target SM, CMToken/CfgToken → target PE (direct append, bypasses FIFO) 194 194 + - `System.inject(token: Token)` -- route token by type: SMToken → target SM, CMToken → target PE (IRAMWriteToken routes to PE automatically as CMToken subclass) (direct append, bypasses FIFO) 197 195 - `System.send(token: Token)` -- same routing as inject() but yields `store.put()` (SimPy generator, respects FIFO backpressure) 198 196 - `System.load(tokens: list[Token])` -- spawns SimPy process that calls send() for each token in order 199 197 200 198 **PEConfig extensions (emu/types.py):** 201 199 - `allowed_pe_routes: Optional[set[int]]` -- if set, restrict PE route_table to these PE IDs 202 200 - `allowed_sm_routes: Optional[set[int]]` -- if set, restrict PE sm_routes to these SM IDs 201 201 + 202 202 + **SMConfig (emu/types.py):** 203 203 + - `sm_id: int`, `cell_count: int = 512`, `initial_cells: Optional[dict]`, `tier_boundary: int = 256` 204 204 + - `tier_boundary` controls the T0/T1 split: addresses below are T1 (I-structure), at/above are T0 (shared raw storage) 205 205 + - All SM instances share the same `t0_store: list[Token]` (wired by `build_topology`) 203 206 204 207 ### Module Dependency Graph 205 208 ··· 231 234 dfgraph/frontend/ 232 235 ``` 233 236 234 234 - <!-- freshness: 2026-02-24 --> 237 237 + <!-- freshness: 2026-02-26 -->

+5 -5

asm/CLAUDE.md

··· 1 1 # Assembler (asm/) 2 2 3 3 - Last verified: 2026-02-23 3 3 + Last verified: 2026-02-26 4 4 5 5 ## Purpose 6 6 ··· 18 18 2. **Resolve** (`resolve.py`): Validates all edge endpoints exist. Detects scope violations (cross-function label refs). Generates Levenshtein "did you mean" suggestions. 19 19 3. **Place** (`place.py`): Validates explicit PE placements. Auto-places unplaced nodes via greedy bin-packing with locality heuristic (prefer PE with most connected neighbours). 20 20 4. **Allocate** (`allocate.py`): Assigns IRAM offsets (dyadic first, then monadic). Assigns context slots (one per function scope per PE). Resolves symbolic destinations to `Addr(a, port, pe)`. 21 21 - 5. **Codegen** (`codegen.py`): Two modes: direct (PEConfig/SMConfig + seeds) and token stream (SM init -> ROUTE_SET -> LOAD_INST -> seeds). Computes route restrictions per PE. 21 21 + 5. **Codegen** (`codegen.py`): Two modes: direct (PEConfig/SMConfig + seeds) and token stream (SM init -> IRAM writes -> seeds). Computes route restrictions per PE. 22 22 23 23 ## Dependencies 24 24 25 25 - - **Uses**: `cm_inst` (Port, MemOp, CfgOp, ALUOp, ALUInst, SMInst, Addr), `tokens` (MonadToken, SMToken, CfgToken, LoadInstToken, RouteSetToken), `sm_mod` (Presence), `emu/types` (PEConfig, SMConfig), `lark` (parser) 25 25 + - **Uses**: `cm_inst` (Port, MemOp, ALUOp, ALUInst, SMInst, Addr), `tokens` (MonadToken, SMToken, IRAMWriteToken), `sm_mod` (Presence), `emu/types` (PEConfig, SMConfig), `lark` (parser) 26 26 - **Used by**: Test suite, user programs, `dfgraph/` (pipeline, graph_json use ir, lower, resolve, place, allocate, errors, opcodes) 27 27 - **Boundary**: `emu/` and root-level modules must NEVER import from `asm/` 28 28 ··· 38 38 - Names inside function regions are always qualified: `$funcname.&label` 39 39 - After placement, every IRNode has `pe is not None` 40 40 - After allocation, every IRNode has `iram_offset` and `ctx` set, and destinations are `ResolvedDest` with concrete `Addr` 41 41 - - Token stream order is always: SM init -> ROUTE_SET -> LOAD_INST -> seed tokens 41 41 + - Token stream order is always: SM init -> IRAM writes -> seed tokens 42 42 43 43 ## Key Files 44 44 ··· 52 52 - `MemOp.WRITE` arity depends on const: monadic when const is set (cell_addr from const), dyadic when const is None (cell_addr from left operand) 53 53 - `RoutingOp.FREE_CTX` (ALU context deallocation) and `MemOp.FREE` (SM free) are disambiguated by mnemonic: assembler uses `free_ctx` for ALU and `free` for SM 54 54 55 55 - <!-- freshness: 2026-02-24 --> 55 55 + <!-- freshness: 2026-02-26 -->

+2 -2

asm/__init__.py

··· 86 86 """Assemble dfasm source to hardware-faithful bootstrap token stream. 87 87 88 88 Chains the full pipeline: parse → lower → resolve → place → allocate → codegen (token mode). 89 89 - Returns an ordered sequence: SM init tokens → ROUTE_SET tokens → LOAD_INST tokens → seed tokens. 89 89 + Returns an ordered sequence: SM init tokens → IRAM write tokens → seed tokens. 90 90 This sequence is consumable by emulator System.inject() and System.load(). 91 91 92 92 Args: 93 93 source: dfasm source code as a string 94 94 95 95 Returns: 96 96 - List of tokens (SMToken, CfgToken, MonadToken) in bootstrap order 96 96 + List of tokens (SMToken, IRAMWriteToken, MonadToken) in bootstrap order 97 97 98 98 Raises: 99 99 ValueError: If any pipeline stage reports errors

+11 -22

asm/codegen.py

··· 3 3 Converts fully allocated IRGraphs to emulator-ready configuration objects and 4 4 token streams. Two output modes: 5 5 1. Direct mode: Produces PEConfig/SMConfig lists + seed tokens (for direct setup) 6 6 - 2. Token stream mode: Produces bootstrap sequence (SM init → ROUTE_SET → LOAD_INST → seeds) 6 6 + 2. Token stream mode: Produces bootstrap sequence (SM init → IRAM writes → seeds) 7 7 8 8 Reference: Phase 6 design doc, Tasks 1-2. 9 9 """ ··· 16 16 DEFAULT_IRAM_CAPACITY, DEFAULT_CTX_SLOTS 17 17 ) 18 18 from asm.opcodes import is_dyadic 19 19 - from cm_inst import ALUInst, CfgOp, MemOp, RoutingOp, SMInst 19 19 + from cm_inst import ALUInst, MemOp, RoutingOp, SMInst 20 20 from emu.types import PEConfig, SMConfig 21 21 - from tokens import LoadInstToken, MonadToken, RouteSetToken, SMToken 21 21 + from tokens import IRAMWriteToken, MonadToken, SMToken 22 22 from sm_mod import Presence 23 23 24 24 ··· 231 231 def generate_tokens(graph: IRGraph) -> list: 232 232 """Generate bootstrap token sequence from an allocated IRGraph. 233 233 234 234 - Produces tokens in order: SM init → ROUTE_SET → LOAD_INST → seeds 234 234 + Produces tokens in order: SM init → IRAM writes → seeds 235 235 236 236 Args: 237 237 graph: A fully allocated IRGraph (after allocate pass) 238 238 239 239 Returns: 240 240 - List of tokens (SMToken, CfgToken, MonadToken) in bootstrap order 240 240 + List of tokens (SMToken, IRAMWriteToken, MonadToken) in bootstrap order 241 241 """ 242 242 # Use direct mode to get configs and seeds 243 243 result = generate_direct(graph) ··· 258 258 ) 259 259 tokens.append(token) 260 260 261 261 - # 2. ROUTE_SET tokens 261 261 + # 2. IRAM write tokens 262 262 for pe_config in result.pe_configs: 263 263 - token = RouteSetToken( 264 264 - target=pe_config.pe_id, 265 265 - addr=None, 266 266 - op=CfgOp.ROUTE_SET, 267 267 - pe_routes=frozenset(pe_config.allowed_pe_routes or ()), 268 268 - sm_routes=frozenset(pe_config.allowed_sm_routes or ()), 269 269 - ) 270 270 - tokens.append(token) 271 271 - 272 272 - # 3. LOAD_INST tokens 273 273 - for pe_config in result.pe_configs: 274 274 - # Get instructions in offset order 275 263 offsets = sorted(pe_config.iram.keys()) 276 264 iram_instructions = [pe_config.iram[offset] for offset in offsets] 277 277 - token = LoadInstToken( 265 265 + token = IRAMWriteToken( 278 266 target=pe_config.pe_id, 279 279 - addr=0, 280 280 - op=CfgOp.LOAD_INST, 267 267 + offset=0, 268 268 + ctx=0, 269 269 + data=0, 281 270 instructions=tuple(iram_instructions), 282 271 ) 283 272 tokens.append(token) 284 273 285 285 - # 4. Seed tokens 274 274 + # 3. Seed tokens 286 275 tokens.extend(result.seed_tokens) 287 276 288 277 return tokens

+3 -3

asm/lower.py

··· 21 21 ) 22 22 from asm.errors import AssemblyError, ErrorCategory 23 23 from asm.opcodes import MNEMONIC_TO_OP 24 24 - from cm_inst import ALUOp, CfgOp, MemOp, Port 24 24 + from cm_inst import ALUOp, MemOp, Port 25 25 26 26 # Reserved names that cannot be used as node definitions 27 27 _RESERVED_NAMES = frozenset({"@system", "@io", "@debug"}) ··· 750 750 return (str(param_name), value) 751 751 752 752 @v_args(inline=True) 753 753 - def opcode(self, token: LarkToken) -> Optional[Union[ALUOp, MemOp, CfgOp]]: 754 754 - """Map opcode token to ALUOp, MemOp, or CfgOp enum, or None if invalid.""" 753 753 + def opcode(self, token: LarkToken) -> Optional[Union[ALUOp, MemOp]]: 754 754 + """Map opcode token to ALUOp or MemOp enum, or None if invalid.""" 755 755 mnemonic = str(token) 756 756 if mnemonic not in MNEMONIC_TO_OP: 757 757 # Add error but don't crash

+19 -17

asm/opcodes.py

··· 8 8 """ 9 9 10 10 from typing import Optional, Union 11 11 - from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp, is_monadic_alu 11 11 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp, is_monadic_alu 12 12 13 13 14 14 # Build mnemonic to opcode mapping 15 15 - MNEMONIC_TO_OP: dict[str, Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]] = { 15 15 + MNEMONIC_TO_OP: dict[str, Union[ArithOp, LogicOp, RoutingOp, MemOp]] = { 16 16 # Arithmetic operations 17 17 "add": ArithOp.ADD, 18 18 "sub": ArithOp.SUB, ··· 55 55 "rd_inc": MemOp.RD_INC, 56 56 "rd_dec": MemOp.RD_DEC, 57 57 "cmp_sw": MemOp.CMP_SW, 58 58 - # Configuration operations (system-level) 59 59 - "load_inst": CfgOp.LOAD_INST, 60 60 - "route_set": CfgOp.ROUTE_SET, 58 58 + "exec": MemOp.EXEC, 59 59 + "raw_read": MemOp.RAW_READ, 60 60 + "set_page": MemOp.SET_PAGE, 61 61 + "write_imm": MemOp.WRITE_IMM, 62 62 + "ext": MemOp.EXT, 61 63 } 62 64 63 65 ··· 83 85 """ 84 86 self._mapping = mapping 85 87 86 86 - def __getitem__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> str: 88 88 + def __getitem__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp]) -> str: 87 89 """Get the mnemonic for an opcode. 88 90 89 91 Args: ··· 100 102 raise KeyError(f"Opcode {op} ({type(op).__name__}) not found in mapping") 101 103 return self._mapping[key] 102 104 103 103 - def __contains__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> bool: 105 105 + def __contains__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp]) -> bool: 104 106 """Check if an opcode is in the mapping.""" 105 107 return (type(op), int(op)) in self._mapping 106 108 ··· 150 152 (MemOp, int(MemOp.CLEAR)), 151 153 (MemOp, int(MemOp.RD_INC)), 152 154 (MemOp, int(MemOp.RD_DEC)), 155 155 + (MemOp, int(MemOp.EXEC)), 156 156 + (MemOp, int(MemOp.RAW_READ)), 157 157 + (MemOp, int(MemOp.SET_PAGE)), 158 158 + (MemOp, int(MemOp.WRITE_IMM)), 159 159 + (MemOp, int(MemOp.EXT)), 153 160 ]) 154 161 155 162 ··· 169 176 """ 170 177 self._tuples = tuples 171 178 172 172 - def __contains__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> bool: 179 179 + def __contains__(self, op: Union[ArithOp, LogicOp, RoutingOp, MemOp]) -> bool: 173 180 """Check if an opcode is in the set, handling IntEnum collisions. 174 181 175 182 Args: ··· 198 205 MONADIC_OPS: TypeAwareMonadicOpsSet = TypeAwareMonadicOpsSet(_MONADIC_OPS_TUPLES) 199 206 200 207 201 201 - def is_monadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp], const: Optional[int] = None) -> bool: 208 208 + def is_monadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp], const: Optional[int] = None) -> bool: 202 209 """Check if an opcode is monadic (single input operand). 203 210 204 211 Args: 205 205 - op: The ALUOp, MemOp, or CfgOp enum value 212 212 + op: The ALUOp or MemOp enum value 206 213 const: Optional const value. Used to determine monadic form of WRITE. 207 214 If const is not None, WRITE is monadic (cell_addr from const). 208 215 If const is None, WRITE is dyadic (cell_addr from left operand). ··· 210 217 Returns: 211 218 True if the opcode is always monadic, or if it's WRITE with const set. 212 219 False for CMP_SW (always dyadic) and WRITE with const=None (dyadic). 213 213 - CfgOp operations are always monadic (system-level, no ALU involvement). 214 220 """ 215 215 - # CfgOp operations are always monadic (system-level configuration) 216 216 - if type(op) is CfgOp: 217 217 - return True 218 218 - 219 221 # Use canonical is_monadic_alu for ALU operations 220 222 if isinstance(op, (ArithOp, LogicOp, RoutingOp)): 221 223 return is_monadic_alu(op) ··· 232 234 return False 233 235 234 236 235 235 - def is_dyadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp], const: Optional[int] = None) -> bool: 237 237 + def is_dyadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp], const: Optional[int] = None) -> bool: 236 238 """Check if an opcode is dyadic (two input operands). 237 239 238 240 Args: 239 239 - op: The ALUOp, MemOp, or CfgOp enum value 241 241 + op: The ALUOp or MemOp enum value 240 242 const: Optional const value. Used for context-dependent operations like WRITE. 241 243 242 244 Returns:

+15 -15

cm_inst.py

··· 9 9 10 10 11 11 class MemOp(IntEnum): 12 12 - READ = 0b000 13 13 - WRITE = 0b001 14 14 - ALLOC = 0b011 15 15 - FREE = 0b100 16 16 - CLEAR = 0b101 17 17 - # reserved 18 18 - RD_INC = 0b1100 19 19 - RD_DEC = 0b1101 20 20 - CMP_SW = 0b1110 21 21 - # reserved 22 22 - 23 23 - 24 24 - class CfgOp(IntEnum): 25 25 - LOAD_INST = 0 26 26 - ROUTE_SET = 1 12 12 + # Tier 1 (3-bit opcode, 10-bit addr) 13 13 + READ = 0 14 14 + WRITE = 1 15 15 + EXEC = 2 16 16 + ALLOC = 3 17 17 + FREE = 4 18 18 + EXT = 5 19 19 + # Tier 2 (5-bit opcode, 8-bit payload) 20 20 + CLEAR = 6 21 21 + RD_INC = 7 22 22 + RD_DEC = 8 23 23 + CMP_SW = 9 24 24 + RAW_READ = 10 25 25 + SET_PAGE = 11 26 26 + WRITE_IMM = 12 27 27 28 28 29 29 class ALUOp(IntEnum):

+4 -4

dfgraph/CLAUDE.md

··· 1 1 # Dataflow Graph Renderer (dfgraph/) 2 2 3 3 - Last verified: 2026-02-24 3 3 + Last verified: 2026-02-26 4 4 5 5 ## Purpose 6 6 ··· 14 14 15 15 ## Dependencies 16 16 17 17 - - **Uses**: `cm_inst` (ArithOp, LogicOp, RoutingOp, MemOp, CfgOp, Addr), `asm/` (ir, lower, resolve, place, allocate, errors, opcodes), `lark` (parser), `fastapi`/`uvicorn` (server), `watchdog` (file watcher), `cytoscape`/`cytoscape-elk` (frontend) 17 17 + - **Uses**: `cm_inst` (ArithOp, LogicOp, RoutingOp, MemOp, Addr), `asm/` (ir, lower, resolve, place, allocate, errors, opcodes), `lark` (parser), `fastapi`/`uvicorn` (server), `watchdog` (file watcher), `cytoscape`/`cytoscape-elk` (frontend) 18 18 - **Used by**: Developer tooling only (not imported by any other package) 19 19 - **Boundary**: `cm_inst`, `asm/`, `emu/`, and root-level modules must NEVER import from `dfgraph/` 20 20 ··· 29 29 ## Invariants 30 30 31 31 - `dfgraph/` never imports from `emu/` -- it only needs the assembler IR, not the simulator 32 32 - - `categorise()` covers all ALUOp/MemOp/CfgOp values; raises `ValueError` on unknown types 32 32 + - `categorise()` covers all ALUOp/MemOp values; raises `ValueError` on unknown types 33 33 - `graph_to_json()` always returns a dict with `type: "graph_update"` -- even on parse failure (with empty node/edge lists) 34 34 - WebSocket protocol: server sends `GraphUpdate` JSON on connect and on every file change; client sends nothing meaningful (receive loop is just for keepalive) 35 35 ··· 49 49 - `DebouncedFileHandler` runs its callback on a `threading.Timer` thread, so `_on_file_change` uses `asyncio.run_coroutine_threadsafe` to bridge into the async event loop 50 50 - Frontend `dist/` directory must contain the compiled TypeScript bundle; the server mounts it as static files at `/dist` 51 51 52 52 - <!-- freshness: 2026-02-24 --> 52 52 + <!-- freshness: 2026-02-26 -->

+4 -6

dfgraph/categories.py

··· 1 1 """Opcode-to-category mapping for visual graph rendering. 2 2 3 3 - Maps each ALUOp/MemOp/CfgOp to a visual category and colour 3 3 + Maps each ALUOp/MemOp to a visual category and colour 4 4 for the dataflow graph renderer. 5 5 """ 6 6 ··· 9 9 from enum import Enum 10 10 from typing import Union 11 11 12 12 - from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp 12 12 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp 13 13 14 14 15 15 class OpcodeCategory(Enum): ··· 42 42 }) 43 43 44 44 45 45 - def categorise(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> OpcodeCategory: 45 45 + def categorise(op: Union[ArithOp, LogicOp, RoutingOp, MemOp]) -> OpcodeCategory: 46 46 """Categorise an opcode for visual rendering. 47 47 48 48 Maps each opcode to a visual category used by the graph renderer. 49 49 Handles special cases like LogicOp comparison ops and RoutingOp config ops. 50 50 51 51 Args: 52 52 - op: An opcode enum value (ArithOp, LogicOp, RoutingOp, MemOp, or CfgOp) 52 52 + op: An opcode enum value (ArithOp, LogicOp, RoutingOp, or MemOp) 53 53 54 54 Returns: 55 55 The OpcodeCategory for this opcode ··· 69 69 return OpcodeCategory.ROUTING 70 70 if isinstance(op, MemOp): 71 71 return OpcodeCategory.MEMORY 72 72 - if isinstance(op, CfgOp): 73 73 - return OpcodeCategory.CONFIG 74 72 raise ValueError(f"Unknown opcode type: {type(op).__name__}")

+209

docs/implementation-plans/2026-02-26-token-migration/phase_01.md

··· 1 1 + # Token Format Migration Implementation Plan 2 2 + 3 3 + **Goal:** Migrate OR1 dataflow CPU emulator and assembler from old 2-bit type field token encoding to 1-bit SM/CM split with prefix encoding. 4 4 + 5 5 + **Architecture:** Bottom-up migration: type definitions first, then emulator core, assembler/tools, tests, and documentation. Old system token category (SysToken, CfgToken, IOToken, LoadInstToken, RouteSetToken) eliminated. IRAMWriteToken added as CMToken subclass. SM gains T0/T1 memory tier split. 6 6 + 7 7 + **Tech Stack:** Python 3.12, SimPy 4.1, Lark, pytest + hypothesis 8 8 + 9 9 + **Scope:** 5 phases from original design (phases 1-5) 10 10 + 11 11 + **Codebase verified:** 2026-02-26 12 12 + 13 13 + --- 14 14 + 15 15 + ## Acceptance Criteria Coverage 16 16 + 17 17 + This phase implements: 18 18 + 19 19 + ### token-migration.AC1: Old token types removed 20 20 + - **token-migration.AC1.1 Success:** `tokens.py` has no SysToken, CfgToken, IOToken, LoadInstToken, or RouteSetToken classes 21 21 + - **token-migration.AC1.2 Success:** `cm_inst.py` has no CfgOp enum 22 22 + 23 23 + ### token-migration.AC2: IRAMWriteToken works 24 24 + - **token-migration.AC2.1 Success:** IRAMWriteToken routes to target PE via network (isinstance CMToken) 25 25 + 26 26 + ### token-migration.AC3: MemOp enum updated 27 27 + - **token-migration.AC3.1 Success:** MemOp contains EXEC, EXT, SET_PAGE, WRITE_IMM, RAW_READ, CLEAR with correct tier grouping 28 28 + 29 29 + ### token-migration.AC6: Presence metadata widened 30 30 + - **token-migration.AC6.1 Success:** SMCell has is_wide field (default False) 31 31 + 32 32 + **Note:** This is an infrastructure phase. "Done when" from the design plan is: type definitions compile; importing `tokens` and `cm_inst` does not error. Downstream breakage is expected and addressed in subsequent phases. 33 33 + 34 34 + --- 35 35 + 36 36 + <!-- START_SUBCOMPONENT_A (tasks 1-3) --> 37 37 + 38 38 + <!-- START_TASK_1 --> 39 39 + ### Task 1: Update MemOp enum and delete CfgOp in cm_inst.py 40 40 + 41 41 + **Verifies:** token-migration.AC1.2, token-migration.AC3.1 42 42 + 43 43 + **Files:** 44 44 + - Modify: `cm_inst.py:11-27` 45 45 + 46 46 + **Implementation:** 47 47 + 48 48 + Replace the current `MemOp` enum (lines 11-21) and delete the `CfgOp` enum (lines 24-26). 49 49 + 50 50 + The new MemOp uses sequential integer values. The tier 1 / tier 2 grouping is documented in comments but not enforced in the emulator: 51 51 + 52 52 + ```python 53 53 + class MemOp(IntEnum): 54 54 + # Tier 1 (3-bit opcode, 10-bit addr) 55 55 + READ = 0 56 56 + WRITE = 1 57 57 + EXEC = 2 58 58 + ALLOC = 3 59 59 + FREE = 4 60 60 + EXT = 5 61 61 + # Tier 2 (5-bit opcode, 8-bit payload) 62 62 + CLEAR = 6 63 63 + RD_INC = 7 64 64 + RD_DEC = 8 65 65 + CMP_SW = 9 66 66 + RAW_READ = 10 67 67 + SET_PAGE = 11 68 68 + WRITE_IMM = 12 69 69 + ``` 70 70 + 71 71 + Delete the entire `CfgOp` class (lines 24-26). Also remove the blank line between CfgOp and ALUOp. 72 72 + 73 73 + **Verification:** 74 74 + 75 75 + ```bash 76 76 + python -c "from cm_inst import MemOp; print(list(MemOp)); assert not hasattr(__import__('cm_inst'), 'CfgOp')" 77 77 + ``` 78 78 + 79 79 + Expected: Prints all 13 MemOp members. No `CfgOp` attribute. 80 80 + 81 81 + **Commit:** `jj commit -m "refactor: update MemOp enum with new opcodes and delete CfgOp"` 82 82 + 83 83 + <!-- END_TASK_1 --> 84 84 + 85 85 + <!-- START_TASK_2 --> 86 86 + ### Task 2: Update tokens.py — delete old types, add IRAMWriteToken 87 87 + 88 88 + **Verifies:** token-migration.AC1.1, token-migration.AC2.1 89 89 + 90 90 + **Files:** 91 91 + - Modify: `tokens.py:1-64` 92 92 + 93 93 + **Implementation:** 94 94 + 95 95 + The new `tokens.py` should contain only: Token, CMToken, DyadToken, MonadToken, IRAMWriteToken, SMToken. 96 96 + 97 97 + Changes: 98 98 + 1. Update import on line 4: remove `CfgOp` (it no longer exists in cm_inst). 99 99 + 2. Delete classes: SysToken (lines 40-42), IOToken (lines 45-47), CfgToken (lines 50-52), LoadInstToken (lines 55-57), RouteSetToken (lines 60-63). 100 100 + 3. Add IRAMWriteToken as a CMToken subclass after MonadToken. 101 101 + 102 102 + New IRAMWriteToken definition: 103 103 + 104 104 + ```python 105 105 + @dataclass(frozen=True) 106 106 + class IRAMWriteToken(CMToken): 107 107 + instructions: tuple[ALUInst | SMInst, ...] 108 108 + ``` 109 109 + 110 110 + This inherits `target`, `offset`, `ctx`, `data` from CMToken. Per the design: `offset` serves as `iram_addr`; `ctx` and `data` are set to 0 by callers (unused but satisfying the frozen dataclass contract). Network routing via `isinstance(token, CMToken)` sends it to PEs automatically. 111 111 + 112 112 + The complete file after changes: 113 113 + 114 114 + ```python 115 115 + from dataclasses import dataclass 116 116 + from typing import Optional 117 117 + 118 118 + from cm_inst import ALUInst, MemOp, Port, SMInst 119 119 + 120 120 + 121 121 + @dataclass(frozen=True) 122 122 + class Token(object): 123 123 + target: int 124 124 + 125 125 + 126 126 + @dataclass(frozen=True) 127 127 + class CMToken(Token): 128 128 + offset: int 129 129 + ctx: int 130 130 + data: int 131 131 + 132 132 + 133 133 + @dataclass(frozen=True) 134 134 + class DyadToken(CMToken): 135 135 + port: Port 136 136 + gen: int 137 137 + wide: bool 138 138 + 139 139 + 140 140 + @dataclass(frozen=True) 141 141 + class MonadToken(CMToken): 142 142 + inline: bool 143 143 + 144 144 + 145 145 + @dataclass(frozen=True) 146 146 + class IRAMWriteToken(CMToken): 147 147 + instructions: tuple[ALUInst | SMInst, ...] 148 148 + 149 149 + 150 150 + @dataclass(frozen=True) 151 151 + class SMToken(Token): 152 152 + addr: int 153 153 + op: MemOp 154 154 + flags: Optional[int] 155 155 + data: Optional[int] 156 156 + ret: Optional[CMToken] 157 157 + ``` 158 158 + 159 159 + **Verification:** 160 160 + 161 161 + ```bash 162 162 + python -c "from tokens import IRAMWriteToken, CMToken; assert issubclass(IRAMWriteToken, CMToken); print('IRAMWriteToken is CMToken subclass')" 163 163 + python -c "import tokens; assert not hasattr(tokens, 'SysToken'); assert not hasattr(tokens, 'CfgToken'); assert not hasattr(tokens, 'IOToken'); assert not hasattr(tokens, 'LoadInstToken'); assert not hasattr(tokens, 'RouteSetToken'); print('Old types removed')" 164 164 + ``` 165 165 + 166 166 + Expected: Both commands print success messages. 167 167 + 168 168 + **Commit:** `jj commit -m "refactor: delete old token types, add IRAMWriteToken"` 169 169 + 170 170 + <!-- END_TASK_2 --> 171 171 + 172 172 + <!-- START_TASK_3 --> 173 173 + ### Task 3: Add is_wide field to SMCell in sm_mod.py 174 174 + 175 175 + **Verifies:** token-migration.AC6.1 176 176 + 177 177 + **Files:** 178 178 + - Modify: `sm_mod.py:13-17` 179 179 + 180 180 + **Implementation:** 181 181 + 182 182 + Add `is_wide: bool = False` to the SMCell dataclass. This field is part of the widened 4-bit presence metadata (presence:2 + is_wide:1 + spare:1). The spare bit is not modelled. 183 183 + 184 184 + Updated SMCell: 185 185 + 186 186 + ```python 187 187 + @dataclass 188 188 + class SMCell(object): 189 189 + pres: Presence 190 190 + data_l: Optional[int] # data or length 191 191 + data_r: Optional[List[int]] # optional data 192 192 + is_wide: bool = False 193 193 + ``` 194 194 + 195 195 + Note: SMCell is intentionally *not* frozen (mutable) — this is existing behaviour and correct for a cell whose presence state changes during simulation. 196 196 + 197 197 + **Verification:** 198 198 + 199 199 + ```bash 200 200 + python -c "from sm_mod import SMCell, Presence; c = SMCell(Presence.EMPTY, None, None); assert c.is_wide is False; print('is_wide defaults to False')" 201 201 + ``` 202 202 + 203 203 + Expected: Prints success message. 204 204 + 205 205 + **Commit:** `jj commit -m "refactor: add is_wide field to SMCell for widened presence metadata"` 206 206 + 207 207 + <!-- END_TASK_3 --> 208 208 + 209 209 + <!-- END_SUBCOMPONENT_A -->

+515

docs/implementation-plans/2026-02-26-token-migration/phase_02.md

··· 1 1 + # Token Format Migration Implementation Plan 2 2 + 3 3 + **Goal:** Migrate OR1 dataflow CPU emulator and assembler from old 2-bit type field token encoding to 1-bit SM/CM split with prefix encoding. 4 4 + 5 5 + **Architecture:** Bottom-up migration: type definitions first, then emulator core, assembler/tools, tests, and documentation. Old system token category eliminated. IRAMWriteToken added as CMToken subclass. SM gains T0/T1 memory tier split. 6 6 + 7 7 + **Tech Stack:** Python 3.12, SimPy 4.1, Lark, pytest + hypothesis 8 8 + 9 9 + **Scope:** 5 phases from original design (phases 1-5) 10 10 + 11 11 + **Codebase verified:** 2026-02-26 12 12 + 13 13 + --- 14 14 + 15 15 + ## Acceptance Criteria Coverage 16 16 + 17 17 + This phase implements and tests: 18 18 + 19 19 + ### token-migration.AC2: IRAMWriteToken works 20 20 + - **token-migration.AC2.2 Success:** PE receives IRAMWriteToken and writes instructions to IRAM at the specified offset 21 21 + - **token-migration.AC2.3 Success:** PE executes instructions loaded via IRAMWriteToken correctly 22 22 + - **token-migration.AC2.4 Failure:** IRAMWriteToken with invalid target PE raises or is dropped 23 23 + 24 24 + ### token-migration.AC4: SM T0/T1 tier split 25 25 + - **token-migration.AC4.1 Success:** SM operations on addresses below tier_boundary use I-structure semantics (presence tracking, deferred reads) 26 26 + - **token-migration.AC4.2 Success:** SM WRITE to T0 address stores data without presence checking 27 27 + - **token-migration.AC4.3 Success:** SM READ on T0 address returns immediately (no deferral) 28 28 + - **token-migration.AC4.4 Success:** T0 storage is shared — all SMs reference the same T0 store 29 29 + - **token-migration.AC4.5 Failure:** I-structure ops (CLEAR, ALLOC, FREE, atomics) on T0 address produce error 30 30 + - **token-migration.AC4.6 Edge:** Tier boundary is configurable via SMConfig; default is 256 31 31 + 32 32 + ### token-migration.AC5: EXEC opcode 33 33 + - **token-migration.AC5.1 Success:** EXEC reads Token objects from T0 starting at given address and injects them into the network 34 34 + - **token-migration.AC5.2 Success:** Injected tokens are processed normally by target PEs/SMs 35 35 + - **token-migration.AC5.3 Success:** EXEC can load a program (IRAM writes + seed tokens) from T0 that executes correctly 36 36 + - **token-migration.AC5.4 Edge:** EXEC on empty T0 region is a no-op 37 37 + 38 38 + --- 39 39 + 40 40 + <!-- START_SUBCOMPONENT_A (tasks 1-2) --> 41 41 + 42 42 + <!-- START_TASK_1 --> 43 43 + ### Task 1: Add tier_boundary to SMConfig in emu/types.py 44 44 + 45 45 + **Verifies:** token-migration.AC4.6 46 46 + 47 47 + **Files:** 48 48 + - Modify: `emu/types.py:33-37` 49 49 + 50 50 + **Implementation:** 51 51 + 52 52 + Add `tier_boundary: int = 256` to the SMConfig frozen dataclass. This field configures where the T0/T1 boundary falls in the SM address space. 53 53 + 54 54 + ```python 55 55 + @dataclass(frozen=True) 56 56 + class SMConfig: 57 57 + sm_id: int 58 58 + cell_count: int = 512 59 59 + initial_cells: Optional[dict[int, tuple[Presence, Optional[int]]]] = None 60 60 + tier_boundary: int = 256 61 61 + ``` 62 62 + 63 63 + **Verification:** 64 64 + 65 65 + ```bash 66 66 + python -c "from emu.types import SMConfig; c = SMConfig(sm_id=0); assert c.tier_boundary == 256; print('tier_boundary defaults to 256')" 67 67 + ``` 68 68 + 69 69 + **Commit:** `jj commit -m "feat: add tier_boundary to SMConfig for T0/T1 memory split"` 70 70 + 71 71 + <!-- END_TASK_1 --> 72 72 + 73 73 + <!-- START_TASK_2 --> 74 74 + ### Task 2: Update PE to handle IRAMWriteToken instead of CfgToken 75 75 + 76 76 + **Verifies:** token-migration.AC2.2, token-migration.AC2.3, token-migration.AC2.4 77 77 + 78 78 + **Files:** 79 79 + - Modify: `emu/pe.py:1-13` (imports) 80 80 + - Modify: `emu/pe.py:52-70` (_run dispatch) 81 81 + - Modify: `emu/pe.py:87-103` (_handle_cfg → _handle_iram_write) 82 82 + 83 83 + **Implementation:** 84 84 + 85 85 + **Step 1: Update imports (lines 1-13)** 86 86 + 87 87 + Replace the import block. Remove CfgOp, CfgToken, LoadInstToken, RouteSetToken. Add IRAMWriteToken. 88 88 + 89 89 + Old: 90 90 + ```python 91 91 + from cm_inst import ALUInst, Addr, ArithOp, CfgOp, LogicOp, MemOp, Port, RoutingOp, SMInst, is_monadic_alu 92 92 + from emu.alu import execute 93 93 + from emu.types import MatchEntry 94 94 + from tokens import ( 95 95 + CMToken, CfgToken, DyadToken, LoadInstToken, 96 96 + MonadToken, RouteSetToken, SMToken, 97 97 + ) 98 98 + ``` 99 99 + 100 100 + New: 101 101 + ```python 102 102 + from cm_inst import ALUInst, Addr, ArithOp, LogicOp, MemOp, Port, RoutingOp, SMInst, is_monadic_alu 103 103 + from emu.alu import execute 104 104 + from emu.types import MatchEntry 105 105 + from tokens import ( 106 106 + CMToken, DyadToken, IRAMWriteToken, 107 107 + MonadToken, SMToken, 108 108 + ) 109 109 + ``` 110 110 + 111 111 + **Step 2: Update _run() dispatch (lines 52-70)** 112 112 + 113 113 + Replace the CfgToken isinstance check with IRAMWriteToken. The IRAMWriteToken check must come *before* the MonadToken/DyadToken checks because IRAMWriteToken is a CMToken subclass. 114 114 + 115 115 + Old (lines 56-58): 116 116 + ```python 117 117 + if isinstance(token, CfgToken): 118 118 + self._handle_cfg(token) 119 119 + continue 120 120 + ``` 121 121 + 122 122 + New: 123 123 + ```python 124 124 + if isinstance(token, IRAMWriteToken): 125 125 + self._handle_iram_write(token) 126 126 + continue 127 127 + ``` 128 128 + 129 129 + **Step 3: Replace _handle_cfg with _handle_iram_write (lines 87-103)** 130 130 + 131 131 + Delete the entire `_handle_cfg` method and replace with `_handle_iram_write`. 132 132 + 133 133 + Old: 134 134 + ```python 135 135 + def _handle_cfg(self, token: CfgToken) -> None: 136 136 + """Handle configuration tokens for dynamic IRAM updates and routing setup.""" 137 137 + if isinstance(token, LoadInstToken): 138 138 + base_addr = token.addr if token.addr is not None else 0 139 139 + for i, inst in enumerate(token.instructions): 140 140 + self.iram[base_addr + i] = inst 141 141 + elif isinstance(token, RouteSetToken): 142 142 + self.route_table = { 143 143 + pid: store for pid, store in self.route_table.items() 144 144 + if pid in token.pe_routes 145 145 + } 146 146 + self.sm_routes = { 147 147 + sid: store for sid, store in self.sm_routes.items() 148 148 + if sid in token.sm_routes 149 149 + } 150 150 + else: 151 151 + logger.warning("PE%d: unknown CfgOp: %s", self.pe_id, token.op) 152 152 + ``` 153 153 + 154 154 + New: 155 155 + ```python 156 156 + def _handle_iram_write(self, token: IRAMWriteToken) -> None: 157 157 + """Write instructions into IRAM at the offset specified by the token.""" 158 158 + for i, inst in enumerate(token.instructions): 159 159 + self.iram[token.offset + i] = inst 160 160 + ``` 161 161 + 162 162 + Note: `token.offset` replaces `token.addr` because IRAMWriteToken inherits `offset` from CMToken (which serves as `iram_addr` per the design). RouteSetToken handling is deleted entirely — route restriction is deferred to a future design. 163 163 + 164 164 + **Testing:** 165 165 + 166 166 + Tests must verify each AC listed above: 167 167 + - token-migration.AC2.2: PE receives IRAMWriteToken and IRAM contents match `token.instructions` starting at `token.offset` 168 168 + - token-migration.AC2.3: After loading via IRAMWriteToken, PE correctly executes a token arriving at the loaded offset 169 169 + - token-migration.AC2.4: IRAMWriteToken with a target PE that doesn't exist in the topology — System._target_store raises TypeError or token is dropped by routing 170 170 + 171 171 + Follow project testing patterns: class-based pytest with SimPy environments, `_inject_token` helpers. See `/home/orual/Projects/or1-design/tests/test_pe.py` for existing patterns. 172 172 + 173 173 + **Verification:** 174 174 + 175 175 + ```bash 176 176 + python -m pytest tests/test_pe.py -v -x 177 177 + ``` 178 178 + 179 179 + **Commit:** `jj commit -m "feat: replace CfgToken handling with IRAMWriteToken in PE"` 180 180 + 181 181 + <!-- END_TASK_2 --> 182 182 + 183 183 + <!-- END_SUBCOMPONENT_A --> 184 184 + 185 185 + <!-- START_SUBCOMPONENT_B (tasks 3-4) --> 186 186 + 187 187 + <!-- START_TASK_3 --> 188 188 + ### Task 3: Add T0/T1 tier support and EXEC to StructureMemory 189 189 + 190 190 + **Verifies:** token-migration.AC4.1, token-migration.AC4.2, token-migration.AC4.3, token-migration.AC4.5, token-migration.AC5.1, token-migration.AC5.4 191 191 + 192 192 + **Files:** 193 193 + - Modify: `emu/sm.py:1-36` (imports and constructor) 194 194 + - Modify: `emu/sm.py:38-69` (_run dispatch) 195 195 + - Add new methods: `_is_t0`, `_handle_t0_read`, `_handle_t0_write`, `_handle_exec` 196 196 + 197 197 + **Implementation:** 198 198 + 199 199 + **Step 1: Update imports and add System forward reference (lines 1-15)** 200 200 + 201 201 + Add `from __future__ import annotations` at the very top of the file (before all other imports) to enable forward references for the `System` type annotation. Add `TYPE_CHECKING` block for the System import: 202 202 + 203 203 + ```python 204 204 + from __future__ import annotations 205 205 + 206 206 + import logging 207 207 + from dataclasses import replace 208 208 + from typing import TYPE_CHECKING, Optional 209 209 + 210 210 + import simpy 211 211 + 212 212 + from cm_inst import MemOp 213 213 + from emu.alu import UINT16_MASK 214 214 + from emu.types import DeferredRead 215 215 + from sm_mod import Presence, SMCell 216 216 + from tokens import CMToken, MonadToken, SMToken, Token 217 217 + 218 218 + if TYPE_CHECKING: 219 219 + from emu.network import System 220 220 + 221 221 + logger = logging.getLogger(__name__) 222 222 + 223 223 + ATOMIC_CELL_LIMIT = 256 224 224 + ``` 225 225 + 226 226 + **Step 2: Update constructor (lines 18-36)** 227 227 + 228 228 + Add `tier_boundary`, `t0_store`, and `system` parameters. `t0_store` is a shared `list[Token]` and `system` is a `System` reference — both set after construction by `build_topology()`. 229 229 + 230 230 + ```python 231 231 + class StructureMemory: 232 232 + def __init__( 233 233 + self, 234 234 + env: simpy.Environment, 235 235 + sm_id: int, 236 236 + cell_count: int = 512, 237 237 + fifo_capacity: int = 8, 238 238 + tier_boundary: int = 256, 239 239 + ): 240 240 + self.env = env 241 241 + self.sm_id = sm_id 242 242 + self.tier_boundary = tier_boundary 243 243 + self.cells: list[SMCell] = [ 244 244 + SMCell(Presence.EMPTY, None, None) for _ in range(cell_count) 245 245 + ] 246 246 + self.deferred_read: Optional[DeferredRead] = None 247 247 + self._deferred_satisfied: Optional[simpy.Event] = None 248 248 + self._deferred_cancelled: bool = False 249 249 + self.input_store: simpy.Store = simpy.Store(env, capacity=fifo_capacity) 250 250 + self.route_table: dict[int, simpy.Store] = {} 251 251 + self.t0_store: list[Token] = [] 252 252 + self.system: Optional[System] = None 253 253 + self.process = env.process(self._run()) 254 254 + ``` 255 255 + 256 256 + **Step 3: Add tier-check helper** 257 257 + 258 258 + ```python 259 259 + def _is_t0(self, addr: int) -> bool: 260 260 + """Return True if address falls in the T0 (shared raw storage) region.""" 261 261 + return addr >= self.tier_boundary 262 262 + ``` 263 263 + 264 264 + **Step 4: Update _run() dispatch (lines 38-69)** 265 265 + 266 266 + Add T0 dispatch at the top of the match block. For T0 addresses, only READ, WRITE, and EXEC are valid. All I-structure ops on T0 addresses log a warning and are dropped. 267 267 + 268 268 + Replace the entire match block: 269 269 + 270 270 + ```python 271 271 + def _run(self): 272 272 + while True: 273 273 + token = yield self.input_store.get() 274 274 + 275 275 + if not isinstance(token, SMToken): 276 276 + logger.warning( 277 277 + "SM%d: unexpected token type: %s", self.sm_id, type(token) 278 278 + ) 279 279 + continue 280 280 + 281 281 + addr = token.addr 282 282 + op = token.op 283 283 + 284 284 + if self._is_t0(addr): 285 285 + match op: 286 286 + case MemOp.READ: 287 287 + yield from self._handle_t0_read(addr, token) 288 288 + case MemOp.WRITE: 289 289 + self._handle_t0_write(addr, token) 290 290 + case MemOp.EXEC: 291 291 + yield from self._handle_exec(addr) 292 292 + case _: 293 293 + logger.warning( 294 294 + "SM%d: I-structure op %s on T0 address %d", 295 295 + self.sm_id, op.name, addr, 296 296 + ) 297 297 + continue 298 298 + 299 299 + match op: 300 300 + case MemOp.READ: 301 301 + yield from self._handle_read(addr, token) 302 302 + case MemOp.WRITE: 303 303 + yield from self._handle_write(addr, token) 304 304 + case MemOp.CLEAR: 305 305 + self._handle_clear(addr) 306 306 + case MemOp.RD_INC: 307 307 + yield from self._handle_atomic(addr, token, delta=1) 308 308 + case MemOp.RD_DEC: 309 309 + yield from self._handle_atomic(addr, token, delta=-1) 310 310 + case MemOp.CMP_SW: 311 311 + yield from self._handle_cas(addr, token) 312 312 + case MemOp.ALLOC: 313 313 + self._handle_alloc(addr) 314 314 + case MemOp.FREE: 315 315 + self._handle_clear(addr) 316 316 + case MemOp.EXEC: 317 317 + logger.warning( 318 318 + "SM%d: EXEC on T1 address %d (must be T0)", 319 319 + self.sm_id, addr, 320 320 + ) 321 321 + case MemOp.SET_PAGE | MemOp.WRITE_IMM | MemOp.RAW_READ | MemOp.EXT: 322 322 + raise NotImplementedError( 323 323 + f"SM{self.sm_id}: {op.name} not yet implemented" 324 324 + ) 325 325 + case _: 326 326 + logger.warning("SM%d: unknown op %s", self.sm_id, op) 327 327 + ``` 328 328 + 329 329 + **Step 5: Add T0 operation handlers** 330 330 + 331 331 + ```python 332 332 + def _handle_t0_read(self, addr: int, token: SMToken): 333 333 + """T0 READ: return stored data immediately, no presence tracking or deferral.""" 334 334 + if token.ret is None: 335 335 + return 336 336 + t0_idx = addr - self.tier_boundary 337 337 + if t0_idx < len(self.t0_store): 338 338 + entry = self.t0_store[t0_idx] 339 339 + if isinstance(entry, int): 340 340 + yield from self._send_result(token.ret, entry) 341 341 + elif entry is not None: 342 342 + yield from self._send_result(token.ret, 0) 343 343 + else: 344 344 + yield from self._send_result(token.ret, 0) 345 345 + else: 346 346 + yield from self._send_result(token.ret, 0) 347 347 + 348 348 + def _handle_t0_write(self, addr: int, token: SMToken): 349 349 + """T0 WRITE: store raw integer data without presence checking. 350 350 + 351 351 + Note: This stores token.data (an int) into t0_store. For Token object 352 352 + storage (needed by EXEC), populate t0_store directly before simulation 353 353 + starts (e.g., via build_topology or test setup). This asymmetry is 354 354 + intentional — SM WRITE provides data-level access (lookup tables, 355 355 + shared counters), while EXEC consumes pre-loaded Token objects for 356 356 + bootstrap. Future work will unify T0 as list[int] with token 357 357 + serialisation/deserialisation. 358 358 + """ 359 359 + t0_idx = addr - self.tier_boundary 360 360 + while len(self.t0_store) <= t0_idx: 361 361 + self.t0_store.append(None) 362 362 + self.t0_store[t0_idx] = token.data 363 363 + 364 364 + def _handle_exec(self, addr: int): 365 365 + """EXEC: read Token objects from T0 starting at addr and inject into network.""" 366 366 + if self.system is None: 367 367 + logger.warning("SM%d: EXEC but no system reference", self.sm_id) 368 368 + return 369 369 + t0_idx = addr - self.tier_boundary 370 370 + if t0_idx >= len(self.t0_store): 371 371 + return 372 372 + for entry in self.t0_store[t0_idx:]: 373 373 + if entry is None: 374 374 + break 375 375 + if isinstance(entry, Token): 376 376 + yield from self.system.send(entry) 377 377 + ``` 378 378 + 379 379 + Note on unimplemented opcodes: SET_PAGE, WRITE_IMM, RAW_READ, and EXT exist as MemOp enum members. They have an explicit `case` branch in the T1 match block that raises `NotImplementedError`, matching the design plan's specification. The `case _:` branch handles truly unknown values with a warning. In SimPy, the `NotImplementedError` will terminate the SM process with a traceback, which is the correct behaviour for hitting an unimplemented opcode — it signals programmer error rather than silently dropping operations. 380 380 + 381 381 + **Testing:** 382 382 + 383 383 + Tests must verify each AC listed above: 384 384 + - token-migration.AC4.1: SM operations on addresses below tier_boundary (e.g., addr=0) use full I-structure semantics (test READ on EMPTY defers, WRITE satisfies deferred read) 385 385 + - token-migration.AC4.2: SM WRITE to T0 address (e.g., addr=256) stores token.data into t0_store without any presence state changes 386 386 + - token-migration.AC4.3: SM READ on T0 address returns immediately — no deferred_read set 387 387 + - token-migration.AC4.5: I-structure ops (CLEAR, ALLOC, FREE, RD_INC, RD_DEC, CMP_SW) on T0 address log warning and are dropped 388 388 + - token-migration.AC5.1: EXEC reads Token objects from t0_store starting at addr and injects them via system.send() 389 389 + - token-migration.AC5.4: EXEC on addr beyond t0_store length is a no-op (no tokens injected) 390 390 + 391 391 + Follow project testing patterns: class-based pytest with SimPy environments, token injection helpers. See `/home/orual/Projects/or1-design/tests/test_sm.py` for existing patterns. 392 392 + 393 393 + **Verification:** 394 394 + 395 395 + ```bash 396 396 + python -m pytest tests/test_sm.py -v -x 397 397 + ``` 398 398 + 399 399 + **Commit:** `jj commit -m "feat: add T0/T1 tier split and EXEC opcode to StructureMemory"` 400 400 + 401 401 + <!-- END_TASK_3 --> 402 402 + 403 403 + <!-- START_TASK_4 --> 404 404 + ### Task 4: Simplify network routing and wire T0 store 405 405 + 406 406 + **Verifies:** token-migration.AC4.4, token-migration.AC5.2, token-migration.AC5.3 407 407 + 408 408 + **Files:** 409 409 + - Modify: `emu/network.py:1-7` (imports) 410 410 + - Modify: `emu/network.py:49-55` (_target_store) 411 411 + - Modify: `emu/network.py:58-116` (build_topology) 412 412 + 413 413 + **Implementation:** 414 414 + 415 415 + **Step 1: Update imports (lines 1-7)** 416 416 + 417 417 + Remove CfgToken import. Add Token import (already imported on line 7). 418 418 + 419 419 + Old: 420 420 + ```python 421 421 + from tokens import CMToken, CfgToken, SMToken, Token 422 422 + ``` 423 423 + 424 424 + New: 425 425 + ```python 426 426 + from tokens import CMToken, SMToken, Token 427 427 + ``` 428 428 + 429 429 + **Step 2: Simplify _target_store (lines 49-55)** 430 430 + 431 431 + Remove the CfgToken special case. IRAMWriteToken is a CMToken subclass and routes to PEs automatically via the `isinstance(token, CMToken)` check. 432 432 + 433 433 + Old: 434 434 + ```python 435 435 + def _target_store(self, token: Token) -> simpy.Store: 436 436 + """Resolve the destination store for a token.""" 437 437 + if isinstance(token, SMToken): 438 438 + return self.sms[token.target].input_store 439 439 + if isinstance(token, (CMToken, CfgToken)): 440 440 + return self.pes[token.target].input_store 441 441 + raise TypeError(f"Unknown token type: {type(token).__name__}") 442 442 + ``` 443 443 + 444 444 + New: 445 445 + ```python 446 446 + def _target_store(self, token: Token) -> simpy.Store: 447 447 + """Resolve the destination store for a token.""" 448 448 + if isinstance(token, SMToken): 449 449 + return self.sms[token.target].input_store 450 450 + if isinstance(token, CMToken): 451 451 + return self.pes[token.target].input_store 452 452 + raise TypeError(f"Unknown token type: {type(token).__name__}") 453 453 + ``` 454 454 + 455 455 + **Step 3: Update build_topology to create shared T0 store and wire System reference (lines 58-116)** 456 456 + 457 457 + After creating SMs from configs, create a shared `t0_store` list and assign it to all SMs. After creating the System object, set the system reference on all SMs. 458 458 + 459 459 + Add `tier_boundary` passthrough from SMConfig to StructureMemory constructor. Create the shared T0 store and wire it. 460 460 + 461 461 + Replace the SM creation loop (lines 80-90) with: 462 462 + 463 463 + ```python 464 464 + t0_store: list[Token] = [] 465 465 + 466 466 + for cfg in sm_configs: 467 467 + sm = StructureMemory( 468 468 + env=env, 469 469 + sm_id=cfg.sm_id, 470 470 + cell_count=cfg.cell_count, 471 471 + fifo_capacity=fifo_capacity, 472 472 + tier_boundary=cfg.tier_boundary, 473 473 + ) 474 474 + sm.t0_store = t0_store 475 475 + if cfg.initial_cells is not None: 476 476 + for addr, (pres, data) in cfg.initial_cells.items(): 477 477 + sm.cells[addr] = SMCell(pres, data, None) 478 478 + sms[cfg.sm_id] = sm 479 479 + ``` 480 480 + 481 481 + After the `return System(env, pes, sms)` line (line 116), wire the system reference. Replace: 482 482 + 483 483 + ```python 484 484 + return System(env, pes, sms) 485 485 + ``` 486 486 + 487 487 + With: 488 488 + 489 489 + ```python 490 490 + system = System(env, pes, sms) 491 491 + for sm in sms.values(): 492 492 + sm.system = system 493 493 + return system 494 494 + ``` 495 495 + 496 496 + **Testing:** 497 497 + 498 498 + Tests must verify each AC listed above: 499 499 + - token-migration.AC4.4: Create topology with 2 SMs. Both SMs' `t0_store` is the same object (`sm0.t0_store is sm1.t0_store`). Write via SM0 to T0, verify SM1 sees the same data. 500 500 + - token-migration.AC5.2: EXEC injects tokens that are processed normally — set up T0 with a DyadToken targeting a PE, EXEC from an SM, verify PE receives and processes the token. 501 501 + - token-migration.AC5.3: Full bootstrap test — populate T0 with IRAMWriteToken(s) and seed DyadToken(s), EXEC from SM, verify PE loads IRAM and executes the program. 502 502 + 503 503 + Follow project testing patterns: `build_topology()` with PEConfig/SMConfig, SimPy environments. See `/home/orual/Projects/or1-design/tests/test_integration.py` for existing patterns. 504 504 + 505 505 + **Verification:** 506 506 + 507 507 + ```bash 508 508 + python -m pytest tests/test_integration.py tests/test_network.py -v -x 509 509 + ``` 510 510 + 511 511 + **Commit:** `jj commit -m "feat: simplify network routing to 1-bit SM/CM dispatch, wire shared T0 store"` 512 512 + 513 513 + <!-- END_TASK_4 --> 514 514 + 515 515 + <!-- END_SUBCOMPONENT_B -->

+381

docs/implementation-plans/2026-02-26-token-migration/phase_03.md

··· 1 1 + # Token Format Migration Implementation Plan 2 2 + 3 3 + **Goal:** Migrate OR1 dataflow CPU emulator and assembler from old 2-bit type field token encoding to 1-bit SM/CM split with prefix encoding. 4 4 + 5 5 + **Architecture:** Bottom-up migration: type definitions first, then emulator core, assembler/tools, tests, and documentation. Old system token category eliminated. IRAMWriteToken added as CMToken subclass. SM gains T0/T1 memory tier split. 6 6 + 7 7 + **Tech Stack:** Python 3.12, SimPy 4.1, Lark, pytest + hypothesis 8 8 + 9 9 + **Scope:** 5 phases from original design (phases 1-5) 10 10 + 11 11 + **Codebase verified:** 2026-02-26 12 12 + 13 13 + --- 14 14 + 15 15 + ## Acceptance Criteria Coverage 16 16 + 17 17 + This phase implements and tests: 18 18 + 19 19 + ### token-migration.AC1: Old token types removed 20 20 + - **token-migration.AC1.3 Success:** No module in the codebase imports any deleted type 21 21 + 22 22 + ### token-migration.AC3: MemOp enum updated 23 23 + - **token-migration.AC3.3 Success:** Assembler mnemonic mapping includes all new opcodes 24 24 + 25 25 + ### token-migration.AC7: Assembler updated 26 26 + - **token-migration.AC7.1 Success:** Token stream mode emits IRAMWriteToken (not LoadInstToken) 27 27 + - **token-migration.AC7.2 Success:** Token stream mode does not emit RouteSetToken 28 28 + - **token-migration.AC7.3 Success:** Direct mode (PEConfig/SMConfig) still works 29 29 + - **token-migration.AC7.4 Success:** Assembler round-trip (serialize -> parse -> assemble) works with updated types 30 30 + 31 31 + --- 32 32 + 33 33 + <!-- START_SUBCOMPONENT_A (tasks 1-2) --> 34 34 + 35 35 + <!-- START_TASK_1 --> 36 36 + ### Task 1: Update asm/opcodes.py — remove CfgOp, add new MemOp mnemonics 37 37 + 38 38 + **Verifies:** token-migration.AC1.3, token-migration.AC3.3 39 39 + 40 40 + **Files:** 41 41 + - Modify: `asm/opcodes.py:10-11` (imports) 42 42 + - Modify: `asm/opcodes.py:15` (MNEMONIC_TO_OP type annotation) 43 43 + - Modify: `asm/opcodes.py:58-61` (remove CfgOp entries, add new MemOp entries) 44 44 + - Modify: `asm/opcodes.py:86` (TypeAwareOpToMnemonicDict.__getitem__ type) 45 45 + - Modify: `asm/opcodes.py:103` (TypeAwareOpToMnemonicDict.__contains__ type) 46 46 + - Modify: `asm/opcodes.py:172` (is_monadic type and CfgOp branch) 47 47 + - Modify: `asm/opcodes.py:235` (is_dyadic type) 48 48 + 49 49 + **Implementation:** 50 50 + 51 51 + **Step 1: Update imports (line 11)** 52 52 + 53 53 + Old: 54 54 + ```python 55 55 + from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp, is_monadic_alu 56 56 + ``` 57 57 + 58 58 + New: 59 59 + ```python 60 60 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp, is_monadic_alu 61 61 + ``` 62 62 + 63 63 + **Step 2: Update type alias throughout** 64 64 + 65 65 + Every `Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]` becomes `Union[ArithOp, LogicOp, RoutingOp, MemOp]`. This applies to: 66 66 + - Line 15: `MNEMONIC_TO_OP` type annotation 67 67 + - Line 86: `__getitem__` parameter 68 68 + - Line 103: `__contains__` parameter 69 69 + - Line 172: `is_monadic` first parameter (`op`) 70 70 + - Line 201: `is_monadic` docstring 71 71 + - Line 235: `is_dyadic` first parameter (`op`) 72 72 + 73 73 + For `is_dyadic` specifically (line 235): 74 74 + 75 75 + Old: 76 76 + ```python 77 77 + def is_dyadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp], const: Optional[int] = None) -> bool: 78 78 + ``` 79 79 + 80 80 + New: 81 81 + ```python 82 82 + def is_dyadic(op: Union[ArithOp, LogicOp, RoutingOp, MemOp], const: Optional[int] = None) -> bool: 83 83 + ``` 84 84 + 85 85 + Also update the `is_dyadic` docstring to remove CfgOp references. 86 86 + 87 87 + **Step 3: Remove CfgOp entries, add new MemOp mnemonics (lines 58-61)** 88 88 + 89 89 + Old: 90 90 + ```python 91 91 + # Configuration operations (system-level) 92 92 + "load_inst": CfgOp.LOAD_INST, 93 93 + "route_set": CfgOp.ROUTE_SET, 94 94 + ``` 95 95 + 96 96 + Replace with new MemOp entries: 97 97 + ```python 98 98 + "exec": MemOp.EXEC, 99 99 + "raw_read": MemOp.RAW_READ, 100 100 + "set_page": MemOp.SET_PAGE, 101 101 + "write_imm": MemOp.WRITE_IMM, 102 102 + "ext": MemOp.EXT, 103 103 + ``` 104 104 + 105 105 + **Step 4: Update _MONADIC_OPS_TUPLES (lines 131-153)** 106 106 + 107 107 + Add new monadic MemOp entries. EXEC, RAW_READ, SET_PAGE, WRITE_IMM, and EXT are all monadic (single operand or no ALU involvement): 108 108 + 109 109 + Add after the existing `(MemOp, int(MemOp.RD_DEC)),` entry: 110 110 + ```python 111 111 + (MemOp, int(MemOp.EXEC)), 112 112 + (MemOp, int(MemOp.RAW_READ)), 113 113 + (MemOp, int(MemOp.SET_PAGE)), 114 114 + (MemOp, int(MemOp.WRITE_IMM)), 115 115 + (MemOp, int(MemOp.EXT)), 116 116 + ``` 117 117 + 118 118 + **Step 5: Remove CfgOp branch from is_monadic (lines 215-217)** 119 119 + 120 120 + Delete: 121 121 + ```python 122 122 + # CfgOp operations are always monadic (system-level configuration) 123 123 + if type(op) is CfgOp: 124 124 + return True 125 125 + ``` 126 126 + 127 127 + CfgOp no longer exists. The function signature and docstring should also be updated to remove CfgOp references. 128 128 + 129 129 + **Verification:** 130 130 + 131 131 + ```bash 132 132 + python -c "from asm.opcodes import MNEMONIC_TO_OP; assert 'exec' in MNEMONIC_TO_OP; assert 'load_inst' not in MNEMONIC_TO_OP; print('Mnemonics updated')" 133 133 + python -c "from asm.opcodes import is_monadic; from cm_inst import MemOp; assert is_monadic(MemOp.EXEC); print('EXEC is monadic')" 134 134 + ``` 135 135 + 136 136 + **Commit:** `jj commit -m "refactor: remove CfgOp from opcodes, add new MemOp mnemonics"` 137 137 + 138 138 + <!-- END_TASK_1 --> 139 139 + 140 140 + <!-- START_TASK_2 --> 141 141 + ### Task 2: Update asm/lower.py — remove CfgOp from opcode() return type 142 142 + 143 143 + **Verifies:** token-migration.AC1.3 144 144 + 145 145 + **Files:** 146 146 + - Modify: `asm/lower.py:24` (imports) 147 147 + - Modify: `asm/lower.py:753` (opcode return type) 148 148 + 149 149 + **Implementation:** 150 150 + 151 151 + **Step 1: Update import (line 24)** 152 152 + 153 153 + Old: 154 154 + ```python 155 155 + from cm_inst import ALUOp, CfgOp, MemOp, Port 156 156 + ``` 157 157 + 158 158 + New: 159 159 + ```python 160 160 + from cm_inst import ALUOp, MemOp, Port 161 161 + ``` 162 162 + 163 163 + **Step 2: Update opcode() return type (line 753)** 164 164 + 165 165 + Old: 166 166 + ```python 167 167 + def opcode(self, token: LarkToken) -> Optional[Union[ALUOp, MemOp, CfgOp]]: 168 168 + """Map opcode token to ALUOp, MemOp, or CfgOp enum, or None if invalid.""" 169 169 + ``` 170 170 + 171 171 + New: 172 172 + ```python 173 173 + def opcode(self, token: LarkToken) -> Optional[Union[ALUOp, MemOp]]: 174 174 + """Map opcode token to ALUOp or MemOp enum, or None if invalid.""" 175 175 + ``` 176 176 + 177 177 + The `MNEMONIC_TO_OP` lookup (line 766) still works because CfgOp entries have been removed from that dict in Task 1. 178 178 + 179 179 + **Verification:** 180 180 + 181 181 + ```bash 182 182 + python -c "from asm.lower import LowerTransformer; print('lower.py imports clean')" 183 183 + ``` 184 184 + 185 185 + **Commit:** `jj commit -m "refactor: remove CfgOp from lower.py opcode return type"` 186 186 + 187 187 + <!-- END_TASK_2 --> 188 188 + 189 189 + <!-- END_SUBCOMPONENT_A --> 190 190 + 191 191 + <!-- START_SUBCOMPONENT_B (tasks 3-5) --> 192 192 + 193 193 + <!-- START_TASK_3 --> 194 194 + ### Task 3: Update asm/codegen.py — emit IRAMWriteToken instead of LoadInstToken/RouteSetToken 195 195 + 196 196 + **Verifies:** token-migration.AC7.1, token-migration.AC7.2, token-migration.AC7.3 197 197 + 198 198 + **Files:** 199 199 + - Modify: `asm/codegen.py:19` (cm_inst import) 200 200 + - Modify: `asm/codegen.py:21` (tokens import) 201 201 + - Modify: `asm/codegen.py:231-288` (generate_tokens function) 202 202 + 203 203 + **Implementation:** 204 204 + 205 205 + **Step 1: Update imports (lines 19, 21)** 206 206 + 207 207 + Old: 208 208 + ```python 209 209 + from cm_inst import ALUInst, CfgOp, MemOp, RoutingOp, SMInst 210 210 + from tokens import LoadInstToken, MonadToken, RouteSetToken, SMToken 211 211 + ``` 212 212 + 213 213 + New: 214 214 + ```python 215 215 + from cm_inst import ALUInst, MemOp, RoutingOp, SMInst 216 216 + from tokens import IRAMWriteToken, MonadToken, SMToken 217 217 + ``` 218 218 + 219 219 + **Step 2: Update generate_tokens function (lines 231-288)** 220 220 + 221 221 + The token stream ordering changes from `SM init -> ROUTE_SET -> LOAD_INST -> seeds` to `SM init -> IRAM writes -> seeds`. RouteSetToken emission is entirely removed. 222 222 + 223 223 + Replace the docstring on line 234: 224 224 + ```python 225 225 + """Generate bootstrap token sequence from an allocated IRGraph. 226 226 + 227 227 + Produces tokens in order: SM init → IRAM writes → seeds 228 228 + ``` 229 229 + 230 230 + Replace the return type annotation on line 240: 231 231 + ```python 232 232 + List of tokens (SMToken, IRAMWriteToken, MonadToken) in bootstrap order 233 233 + ``` 234 234 + 235 235 + Delete the ROUTE_SET section (lines 261-270). 236 236 + 237 237 + Replace the LOAD_INST section (lines 272-283) with: 238 238 + 239 239 + ```python 240 240 + # 2. IRAM write tokens 241 241 + for pe_config in result.pe_configs: 242 242 + offsets = sorted(pe_config.iram.keys()) 243 243 + iram_instructions = [pe_config.iram[offset] for offset in offsets] 244 244 + token = IRAMWriteToken( 245 245 + target=pe_config.pe_id, 246 246 + offset=0, 247 247 + ctx=0, 248 248 + data=0, 249 249 + instructions=tuple(iram_instructions), 250 250 + ) 251 251 + tokens.append(token) 252 252 + ``` 253 253 + 254 254 + Note: `offset=0` is the IRAM base address (start writing at offset 0). `ctx=0` and `data=0` satisfy the frozen dataclass contract but are unused by the PE's `_handle_iram_write`. 255 255 + 256 256 + **Testing:** 257 257 + 258 258 + Tests must verify each AC listed above: 259 259 + - token-migration.AC7.1: `generate_tokens()` produces IRAMWriteToken instances (not LoadInstToken) 260 260 + - token-migration.AC7.2: No RouteSetToken in the output of `generate_tokens()` 261 261 + - token-migration.AC7.3: `generate_direct()` still works — produces valid PEConfig/SMConfig with correct IRAM, route restrictions, and seed tokens 262 262 + 263 263 + Follow project testing patterns: construct IRGraph manually, call `generate_direct()` and `generate_tokens()`, assert on output types. See `/home/orual/Projects/or1-design/tests/test_codegen.py` for existing patterns. 264 264 + 265 265 + **Verification:** 266 266 + 267 267 + ```bash 268 268 + python -m pytest tests/test_codegen.py -v -x 269 269 + ``` 270 270 + 271 271 + **Commit:** `jj commit -m "feat: emit IRAMWriteToken in token stream mode, remove RouteSetToken emission"` 272 272 + 273 273 + <!-- END_TASK_3 --> 274 274 + 275 275 + <!-- START_TASK_4 --> 276 276 + ### Task 4: Update dfgraph/categories.py — remove CfgOp branch 277 277 + 278 278 + **Verifies:** token-migration.AC1.3 279 279 + 280 280 + **Files:** 281 281 + - Modify: `dfgraph/categories.py:12` (imports) 282 282 + - Modify: `dfgraph/categories.py:45` (categorise signature) 283 283 + - Modify: `dfgraph/categories.py:72-73` (CfgOp branch) 284 284 + 285 285 + **Implementation:** 286 286 + 287 287 + **Step 1: Update import (line 12)** 288 288 + 289 289 + Old: 290 290 + ```python 291 291 + from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp 292 292 + ``` 293 293 + 294 294 + New: 295 295 + ```python 296 296 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp 297 297 + ``` 298 298 + 299 299 + **Step 2: Update categorise signature (line 45)** 300 300 + 301 301 + Old: 302 302 + ```python 303 303 + def categorise(op: Union[ArithOp, LogicOp, RoutingOp, MemOp, CfgOp]) -> OpcodeCategory: 304 304 + ``` 305 305 + 306 306 + New: 307 307 + ```python 308 308 + def categorise(op: Union[ArithOp, LogicOp, RoutingOp, MemOp]) -> OpcodeCategory: 309 309 + ``` 310 310 + 311 311 + Also update the docstring to remove CfgOp references. 312 312 + 313 313 + **Step 3: Delete CfgOp branch (lines 72-73)** 314 314 + 315 315 + Delete: 316 316 + ```python 317 317 + if isinstance(op, CfgOp): 318 318 + return OpcodeCategory.CONFIG 319 319 + ``` 320 320 + 321 321 + The new MemOp values (EXEC, SET_PAGE, etc.) are covered by the existing `isinstance(op, MemOp)` check on line 70, which returns `OpcodeCategory.MEMORY`. This is correct — these are SM operations, not configuration operations. 322 322 + 323 323 + **Verification:** 324 324 + 325 325 + ```bash 326 326 + python -c "from dfgraph.categories import categorise; from cm_inst import MemOp; assert categorise(MemOp.EXEC).value == 'memory'; print('New MemOps categorised as MEMORY')" 327 327 + ``` 328 328 + 329 329 + **Commit:** `jj commit -m "refactor: remove CfgOp from dfgraph categories"` 330 330 + 331 331 + <!-- END_TASK_4 --> 332 332 + 333 333 + <!-- START_TASK_5 --> 334 334 + ### Task 5: Update asm/__init__.py — fix stale docstrings 335 335 + 336 336 + **Verifies:** token-migration.AC1.3 337 337 + 338 338 + **Files:** 339 339 + - Modify: `asm/__init__.py:89,96` 340 340 + 341 341 + **Implementation:** 342 342 + 343 343 + Update the `assemble_to_tokens()` docstring to reflect the new token stream ordering and types. 344 344 + 345 345 + **Step 1: Update line 89** 346 346 + 347 347 + Old: 348 348 + ```python 349 349 + Returns an ordered sequence: SM init tokens → ROUTE_SET tokens → LOAD_INST tokens → seed tokens. 350 350 + ``` 351 351 + 352 352 + New: 353 353 + ```python 354 354 + Returns an ordered sequence: SM init tokens → IRAM write tokens → seed tokens. 355 355 + ``` 356 356 + 357 357 + **Step 2: Update line 96** 358 358 + 359 359 + Old: 360 360 + ```python 361 361 + List of tokens (SMToken, CfgToken, MonadToken) in bootstrap order 362 362 + ``` 363 363 + 364 364 + New: 365 365 + ```python 366 366 + List of tokens (SMToken, IRAMWriteToken, MonadToken) in bootstrap order 367 367 + ``` 368 368 + 369 369 + **Verification:** 370 370 + 371 371 + ```bash 372 372 + grep -n "CfgToken\|ROUTE_SET\|LOAD_INST" asm/__init__.py 373 373 + ``` 374 374 + 375 375 + Expected: No matches. 376 376 + 377 377 + **Commit:** `jj commit -m "docs: update asm/__init__.py docstrings for token migration"` 378 378 + 379 379 + <!-- END_TASK_5 --> 380 380 + 381 381 + <!-- END_SUBCOMPONENT_B -->

+579

docs/implementation-plans/2026-02-26-token-migration/phase_04.md

··· 1 1 + # Token Format Migration Implementation Plan 2 2 + 3 3 + **Goal:** Migrate OR1 dataflow CPU emulator and assembler from old 2-bit type field token encoding to 1-bit SM/CM split with prefix encoding. 4 4 + 5 5 + **Architecture:** Bottom-up migration: type definitions first, then emulator core, assembler/tools, tests, and documentation. Old system token category eliminated. IRAMWriteToken added as CMToken subclass. SM gains T0/T1 memory tier split. 6 6 + 7 7 + **Tech Stack:** Python 3.12, SimPy 4.1, Lark, pytest + hypothesis 8 8 + 9 9 + **Scope:** 5 phases from original design (phases 1-5) 10 10 + 11 11 + **Codebase verified:** 2026-02-26 12 12 + 13 13 + --- 14 14 + 15 15 + ## Acceptance Criteria Coverage 16 16 + 17 17 + This phase implements and tests: 18 18 + 19 19 + ### token-migration.AC2: IRAMWriteToken works 20 20 + - **token-migration.AC2.1 Success:** IRAMWriteToken routes to target PE via network (isinstance CMToken) 21 21 + - **token-migration.AC2.2 Success:** PE receives IRAMWriteToken and writes instructions to IRAM at the specified offset 22 22 + - **token-migration.AC2.3 Success:** PE executes instructions loaded via IRAMWriteToken correctly 23 23 + - **token-migration.AC2.4 Failure:** IRAMWriteToken with invalid target PE raises or is dropped 24 24 + 25 25 + ### token-migration.AC3: MemOp enum updated 26 26 + - **token-migration.AC3.2 Success:** ALLOC, FREE remain in tier 1 (3-bit); CLEAR in tier 2 (5-bit) 27 27 + - **token-migration.AC3.3 Success:** Assembler mnemonic mapping includes all new opcodes 28 28 + 29 29 + ### token-migration.AC4: SM T0/T1 tier split 30 30 + - **token-migration.AC4.1 Success:** SM operations on addresses below tier_boundary use I-structure semantics (presence tracking, deferred reads) 31 31 + - **token-migration.AC4.2 Success:** SM WRITE to T0 address stores data without presence checking 32 32 + - **token-migration.AC4.3 Success:** SM READ on T0 address returns immediately (no deferral) 33 33 + - **token-migration.AC4.4 Success:** T0 storage is shared — all SMs reference the same T0 store 34 34 + - **token-migration.AC4.5 Failure:** I-structure ops (CLEAR, ALLOC, FREE, atomics) on T0 address produce error 35 35 + - **token-migration.AC4.6 Edge:** Tier boundary is configurable via SMConfig; default is 256 36 36 + 37 37 + ### token-migration.AC5: EXEC opcode 38 38 + - **token-migration.AC5.1 Success:** EXEC reads Token objects from T0 starting at given address and injects them into the network 39 39 + - **token-migration.AC5.2 Success:** Injected tokens are processed normally by target PEs/SMs 40 40 + - **token-migration.AC5.3 Success:** EXEC can load a program (IRAM writes + seed tokens) from T0 that executes correctly 41 41 + - **token-migration.AC5.4 Edge:** EXEC on empty T0 region is a no-op 42 42 + 43 43 + ### token-migration.AC6: Presence metadata widened 44 44 + - **token-migration.AC6.2 Success:** Existing I-structure behaviour unchanged (is_wide=False path) 45 45 + 46 46 + ### token-migration.AC7: Assembler updated 47 47 + - **token-migration.AC7.1 Success:** Token stream mode emits IRAMWriteToken (not LoadInstToken) 48 48 + - **token-migration.AC7.2 Success:** Token stream mode does not emit RouteSetToken 49 49 + - **token-migration.AC7.3 Success:** Direct mode (PEConfig/SMConfig) still works 50 50 + - **token-migration.AC7.4 Success:** Assembler round-trip (serialize -> parse -> assemble) works with updated types 51 51 + 52 52 + ### token-migration.AC8: All tests pass 53 53 + - **token-migration.AC8.1 Success:** `python -m pytest tests/ -v` exits with zero failures 54 54 + 55 55 + --- 56 56 + 57 57 + <!-- START_SUBCOMPONENT_A (tasks 1-3) --> 58 58 + 59 59 + <!-- START_TASK_1 --> 60 60 + ### Task 1: Update tests/test_pe.py — delete RouteSetToken tests, update LoadInstToken to IRAMWriteToken 61 61 + 62 62 + **Verifies:** token-migration.AC2.2, token-migration.AC2.3 63 63 + 64 64 + **Files:** 65 65 + - Modify: `tests/test_pe.py` 66 66 + 67 67 + **Implementation:** 68 68 + 69 69 + **Step 1: Delete the entire `TestRouteSet` class (lines 460-707)** 70 70 + 71 71 + This class tests RouteSetToken handling which is deleted. All 5 test methods (`test_ac71` through `test_ac75`) are removed. Route restriction will return in a future design via a different mechanism. 72 72 + 73 73 + **Step 2: Update `TestBoundaryEdgeCases.test_load_inst_at_non_zero_base_address` (lines 932-972)** 74 74 + 75 75 + This test verifies loading instructions at a non-zero IRAM base address. Update it to use IRAMWriteToken instead of LoadInstToken. 76 76 + 77 77 + Change the local imports inside the test method from: 78 78 + ```python 79 79 + from cm_inst import CfgOp 80 80 + from tokens import LoadInstToken 81 81 + ``` 82 82 + to: 83 83 + ```python 84 84 + from tokens import IRAMWriteToken 85 85 + ``` 86 86 + 87 87 + Replace the LoadInstToken construction: 88 88 + ```python 89 89 + cfg_token = LoadInstToken( 90 90 + target=0, 91 91 + addr=base_addr, 92 92 + op=CfgOp.LOAD_INST, 93 93 + instructions=tuple(instructions), 94 94 + ) 95 95 + ``` 96 96 + with: 97 97 + ```python 98 98 + cfg_token = IRAMWriteToken( 99 99 + target=0, 100 100 + offset=base_addr, 101 101 + ctx=0, 102 102 + data=0, 103 103 + instructions=tuple(instructions), 104 104 + ) 105 105 + ``` 106 106 + 107 107 + Note: `addr` becomes `offset` (inherited from CMToken). The field rename matters. 108 108 + 109 109 + Replace `pe._handle_cfg(cfg_token)` with `pe._handle_iram_write(cfg_token)`. 110 110 + 111 111 + Update the test class and method names/docstrings to reference IRAMWriteToken. 112 112 + 113 113 + **Step 3: Add new IRAMWriteToken test class** 114 114 + 115 115 + Add a new test class that covers IRAMWriteToken specifically: 116 116 + 117 117 + - Test that IRAMWriteToken loads instructions correctly at offset 0 118 118 + - Test that IRAMWriteToken loads instructions at non-zero offset 119 119 + - Test that PE executes instructions loaded via IRAMWriteToken (inject IRAMWriteToken, then inject a DyadToken at the loaded offset, verify correct output) 120 120 + 121 121 + Follow existing test patterns: `env = simpy.Environment()`, inject via `_inject_token`, run simulation, check `pe.output_log`. 122 122 + 123 123 + **Testing:** 124 124 + 125 125 + Tests must verify: 126 126 + - token-migration.AC2.2: IRAM contents match after IRAMWriteToken handled 127 127 + - token-migration.AC2.3: Instruction loaded via IRAMWriteToken executes correctly with token input 128 128 + 129 129 + **Verification:** 130 130 + 131 131 + ```bash 132 132 + python -m pytest tests/test_pe.py -v -x 133 133 + ``` 134 134 + 135 135 + **Commit:** `jj commit -m "test: update PE tests for IRAMWriteToken, remove RouteSetToken tests"` 136 136 + 137 137 + <!-- END_TASK_1 --> 138 138 + 139 139 + <!-- START_TASK_2 --> 140 140 + ### Task 2: Update tests/test_codegen.py — replace CfgToken/RouteSetToken/LoadInstToken assertions 141 141 + 142 142 + **Verifies:** token-migration.AC7.1, token-migration.AC7.2, token-migration.AC7.3, token-migration.AC7.4 143 143 + 144 144 + **Files:** 145 145 + - Modify: `tests/test_codegen.py` 146 146 + 147 147 + **Implementation:** 148 148 + 149 149 + **Step 1: Update imports (lines 26-27)** 150 150 + 151 151 + Old: 152 152 + ```python 153 153 + from cm_inst import ALUInst, Addr, ArithOp, CfgOp, MemOp, Port, RoutingOp, SMInst 154 154 + from tokens import CfgToken, LoadInstToken, MonadToken, RouteSetToken, SMToken 155 155 + ``` 156 156 + 157 157 + New: 158 158 + ```python 159 159 + from cm_inst import ALUInst, Addr, ArithOp, MemOp, Port, RoutingOp, SMInst 160 160 + from tokens import IRAMWriteToken, MonadToken, SMToken 161 161 + ``` 162 162 + 163 163 + **Step 2: Update TestTokenStream token ordering verification (lines 272-279)** 164 164 + 165 165 + Replace the old ordering checks that look for `CfgOp.ROUTE_SET` and `CfgOp.LOAD_INST`: 166 166 + 167 167 + Old ordering logic: 168 168 + ```python 169 169 + if isinstance(t, CfgToken) and t.op == CfgOp.ROUTE_SET: 170 170 + ... 171 171 + if isinstance(t, CfgToken) and t.op == CfgOp.LOAD_INST: 172 172 + ... 173 173 + ``` 174 174 + 175 175 + New ordering logic — the sequence is now `SMToken < IRAMWriteToken < MonadToken`: 176 176 + ```python 177 177 + if isinstance(t, IRAMWriteToken): 178 178 + ... 179 179 + ``` 180 180 + 181 181 + Remove all assertions about RouteSetToken presence in the token stream. 182 182 + 183 183 + **Step 3: Update test_ac88_tokens_are_valid (lines 293-374)** 184 184 + 185 185 + Replace all CfgToken isinstance checks with IRAMWriteToken checks. Remove RouteSetToken structure validation. Update LoadInstToken structure validation to IRAMWriteToken validation. 186 186 + 187 187 + Old assertions: 188 188 + ```python 189 189 + elif isinstance(token, RouteSetToken): 190 190 + assert isinstance(token.op, CfgOp) 191 191 + ... 192 192 + elif isinstance(token, LoadInstToken): 193 193 + assert isinstance(token.op, CfgOp) 194 194 + ``` 195 195 + 196 196 + New assertions: 197 197 + ```python 198 198 + elif isinstance(token, IRAMWriteToken): 199 199 + assert isinstance(token.instructions, tuple) 200 200 + assert len(token.instructions) > 0 201 201 + ``` 202 202 + 203 203 + **Step 4: Update TestEdgeCases (lines 376-568)** 204 204 + 205 205 + Replace `if isinstance(t, RouteSetToken)` filtering (line 430) with appropriate new type checks (this was used to extract route info — now just remove route-related assertions). 206 206 + 207 207 + **Testing:** 208 208 + 209 209 + Tests must verify: 210 210 + - token-migration.AC7.1: Token stream contains IRAMWriteToken instances 211 211 + - token-migration.AC7.2: No RouteSetToken in token stream 212 212 + - token-migration.AC7.3: Direct mode still produces valid PEConfig/SMConfig 213 213 + - token-migration.AC7.4: Round-trip test still works with updated types 214 214 + 215 215 + **Verification:** 216 216 + 217 217 + ```bash 218 218 + python -m pytest tests/test_codegen.py -v -x 219 219 + ``` 220 220 + 221 221 + **Commit:** `jj commit -m "test: update codegen tests for IRAMWriteToken, remove CfgToken assertions"` 222 222 + 223 223 + <!-- END_TASK_2 --> 224 224 + 225 225 + <!-- START_TASK_3 --> 226 226 + ### Task 3: Update tests/test_integration.py — TestTask5CfgTokenLoadInst to IRAMWriteToken 227 227 + 228 228 + **Verifies:** token-migration.AC2.2, token-migration.AC2.3 229 229 + 230 230 + **Files:** 231 231 + - Modify: `tests/test_integration.py:12,15,628-689` 232 232 + 233 233 + **Implementation:** 234 234 + 235 235 + **Step 1: Update imports (lines 12, 15)** 236 236 + 237 237 + Old: 238 238 + ```python 239 239 + from cm_inst import Addr, ALUInst, ArithOp, CfgOp, MemOp, Port, RoutingOp, SMInst 240 240 + ... 241 241 + from tokens import CfgToken, CMToken, DyadToken, LoadInstToken, MonadToken, SMToken 242 242 + ``` 243 243 + 244 244 + New: 245 245 + ```python 246 246 + from cm_inst import Addr, ALUInst, ArithOp, MemOp, Port, RoutingOp, SMInst 247 247 + ... 248 248 + from tokens import CMToken, DyadToken, IRAMWriteToken, MonadToken, SMToken 249 249 + ``` 250 250 + 251 251 + **Step 2: Update TestTask5CfgTokenLoadInst class (lines 628-689)** 252 252 + 253 253 + Rename class to `TestIRAMWriteToken`. Update the docstring. 254 254 + 255 255 + Replace LoadInstToken construction (line 658): 256 256 + 257 257 + Old: 258 258 + ```python 259 259 + cfg_token = LoadInstToken( 260 260 + target=0, 261 261 + addr=0, 262 262 + op=CfgOp.LOAD_INST, 263 263 + instructions=(add_inst,), 264 264 + ) 265 265 + ``` 266 266 + 267 267 + New: 268 268 + ```python 269 269 + iram_token = IRAMWriteToken( 270 270 + target=0, 271 271 + offset=0, 272 272 + ctx=0, 273 273 + data=0, 274 274 + instructions=(add_inst,), 275 275 + ) 276 276 + ``` 277 277 + 278 278 + Update all references from `cfg_token` to `iram_token` throughout the test method. Update comments to say "IRAMWriteToken" instead of "CfgToken". 279 279 + 280 280 + **Verification:** 281 281 + 282 282 + ```bash 283 283 + python -m pytest tests/test_integration.py -v -x 284 284 + ``` 285 285 + 286 286 + **Commit:** `jj commit -m "test: update integration test for IRAMWriteToken"` 287 287 + 288 288 + <!-- END_TASK_3 --> 289 289 + 290 290 + <!-- END_SUBCOMPONENT_A --> 291 291 + 292 292 + <!-- START_SUBCOMPONENT_B (tasks 4-6) --> 293 293 + 294 294 + <!-- START_TASK_4 --> 295 295 + ### Task 4: Update tests/test_e2e.py — replace CfgToken isinstance check 296 296 + 297 297 + **Verifies:** token-migration.AC8.1 298 298 + 299 299 + **Files:** 300 300 + - Modify: `tests/test_e2e.py:16,70` 301 301 + 302 302 + **Implementation:** 303 303 + 304 304 + **Step 1: Update import (line 16)** 305 305 + 306 306 + Old: 307 307 + ```python 308 308 + from tokens import CfgToken, MonadToken, SMToken 309 309 + ``` 310 310 + 311 311 + New: 312 312 + ```python 313 313 + from tokens import IRAMWriteToken, MonadToken, SMToken 314 314 + ``` 315 315 + 316 316 + **Step 2: Update run_program_tokens function (line 70)** 317 317 + 318 318 + The function uses `isinstance(token, CfgToken)` to identify PE-targeting config tokens and extract the PE ID for topology creation. Replace with `isinstance(token, IRAMWriteToken)`. 319 319 + 320 320 + Old: 321 321 + ```python 322 322 + elif isinstance(token, CfgToken): 323 323 + ``` 324 324 + 325 325 + New: 326 326 + ```python 327 327 + elif isinstance(token, IRAMWriteToken): 328 328 + ``` 329 329 + 330 330 + The `.target` field works the same way (inherited from CMToken via Token). 331 331 + 332 332 + **Verification:** 333 333 + 334 334 + ```bash 335 335 + python -m pytest tests/test_e2e.py -v -x 336 336 + ``` 337 337 + 338 338 + **Commit:** `jj commit -m "test: update e2e test for IRAMWriteToken"` 339 339 + 340 340 + <!-- END_TASK_4 --> 341 341 + 342 342 + <!-- START_TASK_5 --> 343 343 + ### Task 5: Update tests/test_opcodes.py — remove CfgOp assertions, add new MemOp assertions 344 344 + 345 345 + **Verifies:** token-migration.AC3.2, token-migration.AC3.3 346 346 + 347 347 + **Files:** 348 348 + - Modify: `tests/test_opcodes.py:5,76-77,227,279` 349 349 + 350 350 + **Implementation:** 351 351 + 352 352 + **Step 1: Update import (line 5)** 353 353 + 354 354 + Old: 355 355 + ```python 356 356 + from tokens import MemOp, CfgOp 357 357 + ``` 358 358 + 359 359 + New: 360 360 + ```python 361 361 + from cm_inst import MemOp 362 362 + ``` 363 363 + 364 364 + (Note: the import was from `tokens` which re-exports from cm_inst. Import directly from cm_inst.) 365 365 + 366 366 + **Step 2: Remove CfgOp mnemonic assertions (lines 76-77)** 367 367 + 368 368 + Delete: 369 369 + ```python 370 370 + assert MNEMONIC_TO_OP["load_inst"] == CfgOp.LOAD_INST 371 371 + assert MNEMONIC_TO_OP["route_set"] == CfgOp.ROUTE_SET 372 372 + ``` 373 373 + 374 374 + Add new MemOp mnemonic assertions: 375 375 + ```python 376 376 + assert MNEMONIC_TO_OP["exec"] == MemOp.EXEC 377 377 + assert MNEMONIC_TO_OP["raw_read"] == MemOp.RAW_READ 378 378 + assert MNEMONIC_TO_OP["set_page"] == MemOp.SET_PAGE 379 379 + assert MNEMONIC_TO_OP["write_imm"] == MemOp.WRITE_IMM 380 380 + assert MNEMONIC_TO_OP["ext"] == MemOp.EXT 381 381 + ``` 382 382 + 383 383 + **Step 3: Update is_monadic parametrize lists (line 227)** 384 384 + 385 385 + Remove `CfgOp.LOAD_INST, CfgOp.ROUTE_SET` from the always-monadic parametrize list. 386 386 + 387 387 + Add the new MemOp values that are monadic: 388 388 + `MemOp.EXEC, MemOp.RAW_READ, MemOp.SET_PAGE, MemOp.WRITE_IMM, MemOp.EXT` 389 389 + 390 390 + **Step 4: Update is_dyadic parametrize lists (line 279)** 391 391 + 392 392 + Remove `CfgOp.LOAD_INST, CfgOp.ROUTE_SET` from the not-dyadic parametrize list. 393 393 + 394 394 + **Verification:** 395 395 + 396 396 + ```bash 397 397 + python -m pytest tests/test_opcodes.py -v -x 398 398 + ``` 399 399 + 400 400 + **Commit:** `jj commit -m "test: update opcode tests for new MemOp values, remove CfgOp"` 401 401 + 402 402 + <!-- END_TASK_5 --> 403 403 + 404 404 + <!-- START_TASK_6 --> 405 405 + ### Task 6: Update tests/test_dfgraph_categories.py — remove TestCategoriseCfgOp, add new MemOp tests 406 406 + 407 407 + **Verifies:** token-migration.AC3.3 408 408 + 409 409 + **Files:** 410 410 + - Modify: `tests/test_dfgraph_categories.py:17,112-121` 411 411 + 412 412 + **Implementation:** 413 413 + 414 414 + **Step 1: Update import (line 17)** 415 415 + 416 416 + Old: 417 417 + ```python 418 418 + from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp 419 419 + ``` 420 420 + 421 421 + New: 422 422 + ```python 423 423 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp 424 424 + ``` 425 425 + 426 426 + **Step 2: Delete TestCategoriseCfgOp class (lines 112-121)** 427 427 + 428 428 + Remove the entire class. CfgOp no longer exists. 429 429 + 430 430 + **Step 3: Add test for new MemOp values** 431 431 + 432 432 + Add a parametrized test that verifies the new MemOp values (EXEC, RAW_READ, SET_PAGE, WRITE_IMM, EXT) are categorised as MEMORY by the `categorise()` function. 433 433 + 434 434 + ```python 435 435 + class TestCategoriseNewMemOps: 436 436 + """Tests for new MemOp values -> MEMORY category mapping.""" 437 437 + 438 438 + @pytest.mark.parametrize("op", [ 439 439 + MemOp.EXEC, MemOp.RAW_READ, MemOp.SET_PAGE, MemOp.WRITE_IMM, MemOp.EXT, 440 440 + ]) 441 441 + def test_new_mem_ops_map_to_memory(self, op): 442 442 + assert categorise(op) == OpcodeCategory.MEMORY 443 443 + ``` 444 444 + 445 445 + **Verification:** 446 446 + 447 447 + ```bash 448 448 + python -m pytest tests/test_dfgraph_categories.py -v -x 449 449 + ``` 450 450 + 451 451 + **Commit:** `jj commit -m "test: update dfgraph category tests for new MemOp values, remove CfgOp"` 452 452 + 453 453 + <!-- END_TASK_6 --> 454 454 + 455 455 + <!-- END_SUBCOMPONENT_B --> 456 456 + 457 457 + <!-- START_SUBCOMPONENT_C (tasks 7-9) --> 458 458 + 459 459 + <!-- START_TASK_7 --> 460 460 + ### Task 7: Update tests/conftest.py — add new MemOp values to Hypothesis strategies 461 461 + 462 462 + **Verifies:** token-migration.AC3.3 463 463 + 464 464 + **Files:** 465 465 + - Modify: `tests/conftest.py:51` 466 466 + 467 467 + **Implementation:** 468 468 + 469 469 + Line 51 currently has: 470 470 + ```python 471 471 + sm_all_ops = st.sampled_from(list(MemOp)) 472 472 + ``` 473 473 + 474 474 + This already dynamically picks up all MemOp members, so it will automatically include the new opcodes (EXEC, EXT, SET_PAGE, WRITE_IMM, RAW_READ) after Phase 1 changes. **No code change needed** — just verify it works. 475 475 + 476 476 + However, the `sm_token` strategy (lines 54-67) generates SMToken objects with randomly chosen MemOp values. The new opcodes (EXEC, SET_PAGE, etc.) are unimplemented in the SM and would log warnings. This is acceptable for property testing — the SM logs warnings for unknown ops and continues. 477 477 + 478 478 + **Verification:** 479 479 + 480 480 + ```bash 481 481 + python -c "from tests.conftest import sm_all_ops; print('conftest strategies import cleanly')" 482 482 + ``` 483 483 + 484 484 + **Commit:** No commit needed — this is a verification-only task. 485 485 + 486 486 + <!-- END_TASK_7 --> 487 487 + 488 488 + <!-- START_TASK_8 --> 489 489 + ### Task 8: Add new tests for SM T0/T1 tier split 490 490 + 491 491 + **Verifies:** token-migration.AC4.1, token-migration.AC4.2, token-migration.AC4.3, token-migration.AC4.4, token-migration.AC4.5, token-migration.AC4.6, token-migration.AC6.2 492 492 + 493 493 + **Files:** 494 494 + - Create: `tests/test_sm_tiers.py` 495 495 + 496 496 + **Implementation:** 497 497 + 498 498 + Create a new test file for T0/T1 tier functionality. This is new functionality not covered by existing tests. 499 499 + 500 500 + Tests must verify each AC listed above: 501 501 + - token-migration.AC4.1: SM READ on T1 address (below tier_boundary) uses I-structure semantics — READ on EMPTY defers, WRITE satisfies deferred read. Reuse patterns from `tests/test_sm.py`. 502 502 + - token-migration.AC4.2: SM WRITE to T0 address (>= tier_boundary) stores data in `t0_store` without changing any presence state. 503 503 + - token-migration.AC4.3: SM READ on T0 address returns immediately. Verify no `deferred_read` is set. 504 504 + - token-migration.AC4.4: Create topology with 2 SMs via `build_topology()`. Assert `sms[0].t0_store is sms[1].t0_store`. Write via SM0 to T0, verify SM1's t0_store has the data. 505 505 + - token-migration.AC4.5: For each I-structure op (CLEAR, ALLOC, FREE, RD_INC, RD_DEC, CMP_SW), send to T0 address, verify warning is logged (use `caplog` fixture) and operation is dropped. 506 506 + - token-migration.AC4.6: Create SM with `tier_boundary=128`. Verify addr=127 is T1 (I-structure) and addr=128 is T0 (raw). 507 507 + - token-migration.AC6.2: Existing I-structure tests continue to pass — `is_wide=False` is the default and doesn't affect behaviour. 508 508 + 509 509 + Follow project testing patterns: class-based pytest with SimPy environments, `inject_token` helpers from `tests/test_sm.py`. See `/home/orual/Projects/or1-design/tests/test_sm.py` for existing patterns. 510 510 + 511 511 + **Verification:** 512 512 + 513 513 + ```bash 514 514 + python -m pytest tests/test_sm_tiers.py -v -x 515 515 + ``` 516 516 + 517 517 + **Commit:** `jj commit -m "test: add SM T0/T1 tier split tests"` 518 518 + 519 519 + <!-- END_TASK_8 --> 520 520 + 521 521 + <!-- START_TASK_9 --> 522 522 + ### Task 9: Add new tests for EXEC opcode and IRAMWriteToken routing 523 523 + 524 524 + **Verifies:** token-migration.AC2.1, token-migration.AC2.4, token-migration.AC5.1, token-migration.AC5.2, token-migration.AC5.3, token-migration.AC5.4 525 525 + 526 526 + **Files:** 527 527 + - Create: `tests/test_exec_bootstrap.py` 528 528 + 529 529 + **Implementation:** 530 530 + 531 531 + Create a new test file for EXEC and bootstrap functionality. 532 532 + 533 533 + Tests must verify each AC listed above: 534 534 + - token-migration.AC2.1: Create topology, inject IRAMWriteToken via `system.inject()`. Verify it arrives at correct PE's `input_store` (routed as CMToken). 535 535 + - token-migration.AC2.4: Inject IRAMWriteToken with `target=99` (nonexistent PE). `System._target_store` should raise `KeyError`. Test via `pytest.raises(KeyError)`. 536 536 + - token-migration.AC5.1: Set up T0 with Token objects. Send EXEC SMToken to an SM. Verify each Token from T0 was injected into the network via `system.send()`. 537 537 + - token-migration.AC5.2: Populate T0 with a DyadToken targeting PE0. EXEC from SM. Run simulation. Verify PE0 received and processed the token (check `pe.output_log`). 538 538 + - token-migration.AC5.3: Full bootstrap test — populate T0 with: IRAMWriteToken (loads ADD instruction into PE0) + DyadToken pair (seed tokens for the ADD). EXEC from SM. Run simulation. Verify PE0 loaded IRAM and produced correct ADD result. 539 539 + - token-migration.AC5.4: EXEC on addr beyond t0_store length. Verify no tokens are injected (output stores remain empty). 540 540 + 541 541 + Follow project testing patterns: `build_topology()` with PEConfig/SMConfig, SimPy environments, collector stores. See `/home/orual/Projects/or1-design/tests/test_integration.py` for existing patterns. 542 542 + 543 543 + **Verification:** 544 544 + 545 545 + ```bash 546 546 + python -m pytest tests/test_exec_bootstrap.py -v -x 547 547 + ``` 548 548 + 549 549 + **Commit:** `jj commit -m "test: add EXEC opcode and IRAMWriteToken routing tests"` 550 550 + 551 551 + <!-- END_TASK_9 --> 552 552 + 553 553 + <!-- END_SUBCOMPONENT_C --> 554 554 + 555 555 + <!-- START_TASK_10 --> 556 556 + ### Task 10: Run full test suite and verify zero failures 557 557 + 558 558 + **Verifies:** token-migration.AC8.1 559 559 + 560 560 + **Files:** None (verification only) 561 561 + 562 562 + **Implementation:** 563 563 + 564 564 + Run the complete test suite: 565 565 + 566 566 + ```bash 567 567 + python -m pytest tests/ -v 568 568 + ``` 569 569 + 570 570 + Expected: All tests pass with zero failures. 571 571 + 572 572 + If any tests fail, investigate and fix. Common issues: 573 573 + - Missed CfgOp/CfgToken imports in a file not covered above 574 574 + - Hypothesis strategies generating tokens that hit unimplemented MemOp cases in SM 575 575 + - Type annotation mismatches in tests 576 576 + 577 577 + **Commit:** `jj commit -m "test: verify all tests pass after token migration"` (only if fixes were needed) 578 578 + 579 579 + <!-- END_TASK_10 -->

+225

docs/implementation-plans/2026-02-26-token-migration/phase_05.md

··· 1 1 + # Token Format Migration Implementation Plan 2 2 + 3 3 + **Goal:** Migrate OR1 dataflow CPU emulator and assembler from old 2-bit type field token encoding to 1-bit SM/CM split with prefix encoding. 4 4 + 5 5 + **Architecture:** Bottom-up migration: type definitions first, then emulator core, assembler/tools, tests, and documentation. 6 6 + 7 7 + **Tech Stack:** Python 3.12, SimPy 4.1, Lark, pytest + hypothesis 8 8 + 9 9 + **Scope:** 5 phases from original design (phases 1-5) 10 10 + 11 11 + **Codebase verified:** 2026-02-26 12 12 + 13 13 + --- 14 14 + 15 15 + ## Acceptance Criteria Coverage 16 16 + 17 17 + This is an infrastructure phase (documentation updates). No acceptance criteria are directly tested — this phase ensures documentation accuracy. 18 18 + 19 19 + **Verifies: None** (documentation only) 20 20 + 21 21 + --- 22 22 + 23 23 + <!-- START_TASK_1 --> 24 24 + ### Task 1: Update CLAUDE.md — Token Hierarchy and Architecture Contracts 25 25 + 26 26 + **Files:** 27 27 + - Modify: `CLAUDE.md` 28 28 + 29 29 + **Implementation:** 30 30 + 31 31 + **Step 1: Remove the "Architecture Contracts > Note" block** 32 32 + 33 33 + Delete the note block (currently around lines after "## Architecture Contracts") that says: 34 34 + ``` 35 35 + > **Note:** The design documents in `design-notes/` have been updated to 36 36 + > reflect the 1-bit SM/CM token split... 37 37 + > ...See `design-notes/sm-and-token-format-discussion.md` for the design rationale. 38 38 + ``` 39 39 + 40 40 + This note is obsolete — the code now matches the design. 41 41 + 42 42 + **Step 2: Update Token Hierarchy section** 43 43 + 44 44 + Replace the current Token Hierarchy listing with: 45 45 + 46 46 + ``` 47 47 + ### Token Hierarchy (tokens.py) 48 48 + 49 49 + All tokens inherit from `Token(target: int)`. The hierarchy: 50 50 + 51 51 + - `CMToken(Token)` -- adds `offset`, `ctx`, `data` (frozen dataclass) 52 52 + - `DyadToken(CMToken)` -- adds `port: Port`, `gen: int`, `wide: bool` 53 53 + - `MonadToken(CMToken)` -- adds `inline: bool` 54 54 + - `IRAMWriteToken(CMToken)` -- adds `instructions: tuple[ALUInst | SMInst, ...]` 55 55 + - `SMToken(Token)` -- `addr: int`, `op: MemOp`, `flags`, `data`, `ret: Optional[CMToken]` 56 56 + ``` 57 57 + 58 58 + Delete all SysToken, CfgToken, IOToken, LoadInstToken, RouteSetToken entries. 59 59 + 60 60 + **Step 3: Update Instruction Set section** 61 61 + 62 62 + In the `cm_inst.py` description, remove `CfgOp` from the list. Update MemOp description to note the new opcodes (EXEC, EXT, SET_PAGE, WRITE_IMM, RAW_READ). 63 63 + 64 64 + **Step 4: Update Processing Element section** 65 65 + 66 66 + Replace the CfgToken handling subsection: 67 67 + 68 68 + Old: 69 69 + ``` 70 70 + **CfgToken handling:** 71 71 + - `LoadInstToken`: writes `token.instructions` into IRAM at `token.addr` base offset 72 72 + - `RouteSetToken`: restricts `route_table` to `token.pe_routes` and `sm_routes` to `token.sm_routes` 73 73 + ``` 74 74 + 75 75 + New: 76 76 + ``` 77 77 + **IRAMWriteToken handling:** 78 78 + - `IRAMWriteToken`: writes `token.instructions` into IRAM starting at `token.offset` 79 79 + ``` 80 80 + 81 81 + **Step 5: Update Network Topology section** 82 82 + 83 83 + In System API section, replace: 84 84 + ``` 85 85 + `System.inject(token: Token)` -- route token by type: SMToken → target SM, CMToken/CfgToken → target PE 86 86 + ``` 87 87 + with: 88 88 + ``` 89 89 + `System.inject(token: Token)` -- route token by type: SMToken → target SM, CMToken → target PE (IRAMWriteToken routes to PE automatically as CMToken subclass) 90 90 + ``` 91 91 + 92 92 + **Step 6: Update Structure Memory section** 93 93 + 94 94 + Add T0/T1 tier split documentation: 95 95 + 96 96 + ``` 97 97 + **Memory Tiers:** 98 98 + - **T1 (below tier_boundary):** Per-SM I-structure cells with presence tracking, deferred reads, atomic ops. Default tier_boundary: 256. 99 99 + - **T0 (at/above tier_boundary):** Shared raw storage across all SMs. No presence tracking. `list[Token]` shared by all SM instances. 100 100 + - T0 operations: READ (immediate return), WRITE (no presence check), EXEC (inject tokens from T0 into network) 101 101 + - I-structure ops on T0 addresses are errors (logged and dropped) 102 102 + ``` 103 103 + 104 104 + **Step 7: Update Module Dependency Graph** 105 105 + 106 106 + Ensure `tokens.py` no longer lists SysToken/CfgToken imports. Update the `asm/codegen.py` dependencies to reference IRAMWriteToken instead of LoadInstToken/RouteSetToken. 107 107 + 108 108 + **Step 8: Update freshness date** 109 109 + 110 110 + Change `<!-- freshness: 2026-02-24 -->` to `<!-- freshness: 2026-02-26 -->`. 111 111 + 112 112 + **Verification:** 113 113 + 114 114 + ```bash 115 115 + grep -rn "CfgOp\|CfgToken\|LoadInstToken\|RouteSetToken\|IOToken\|SysToken" CLAUDE.md 116 116 + ``` 117 117 + 118 118 + Expected: No matches. 119 119 + 120 120 + **Commit:** `jj commit -m "docs: update CLAUDE.md for token migration"` 121 121 + 122 122 + <!-- END_TASK_1 --> 123 123 + 124 124 + <!-- START_TASK_2 --> 125 125 + ### Task 2: Update asm/CLAUDE.md — Dependencies and Invariants 126 126 + 127 127 + **Files:** 128 128 + - Modify: `asm/CLAUDE.md` 129 129 + 130 130 + **Implementation:** 131 131 + 132 132 + **Step 1: Update Dependencies section (line 25)** 133 133 + 134 134 + Old: 135 135 + ``` 136 136 + - **Uses**: `cm_inst` (Port, MemOp, CfgOp, ALUOp, ALUInst, SMInst, Addr), `tokens` (MonadToken, SMToken, CfgToken, LoadInstToken, RouteSetToken), ... 137 137 + ``` 138 138 + 139 139 + New: 140 140 + ``` 141 141 + - **Uses**: `cm_inst` (Port, MemOp, ALUOp, ALUInst, SMInst, Addr), `tokens` (MonadToken, SMToken, IRAMWriteToken), ... 142 142 + ``` 143 143 + 144 144 + **Step 2: Update Pipeline Passes section (line 21)** 145 145 + 146 146 + Replace codegen description: 147 147 + ``` 148 148 + Two modes: direct (PEConfig/SMConfig + seeds) and token stream (SM init -> IRAM writes -> seeds). 149 149 + ``` 150 150 + 151 151 + **Step 3: Update Invariants section (line 41)** 152 152 + 153 153 + Old: 154 154 + ``` 155 155 + - Token stream order is always: SM init -> ROUTE_SET -> LOAD_INST -> seed tokens 156 156 + ``` 157 157 + 158 158 + New: 159 159 + ``` 160 160 + - Token stream order is always: SM init -> IRAM writes -> seed tokens 161 161 + ``` 162 162 + 163 163 + **Step 4: Update freshness date** 164 164 + 165 165 + Change `<!-- freshness: 2026-02-24 -->` to `<!-- freshness: 2026-02-26 -->`. 166 166 + 167 167 + **Verification:** 168 168 + 169 169 + ```bash 170 170 + grep -rn "CfgOp\|CfgToken\|LoadInstToken\|RouteSetToken\|ROUTE_SET\|LOAD_INST" asm/CLAUDE.md 171 171 + ``` 172 172 + 173 173 + Expected: No matches. 174 174 + 175 175 + **Commit:** `jj commit -m "docs: update asm/CLAUDE.md for token migration"` 176 176 + 177 177 + <!-- END_TASK_2 --> 178 178 + 179 179 + <!-- START_TASK_3 --> 180 180 + ### Task 3: Update dfgraph/CLAUDE.md — remove CfgOp references 181 181 + 182 182 + **Files:** 183 183 + - Modify: `dfgraph/CLAUDE.md` 184 184 + 185 185 + **Implementation:** 186 186 + 187 187 + **Step 1: Update Dependencies section** 188 188 + 189 189 + Old: 190 190 + ``` 191 191 + - **Uses**: `cm_inst` (ArithOp, LogicOp, RoutingOp, MemOp, CfgOp, Addr), ... 192 192 + ``` 193 193 + 194 194 + New: 195 195 + ``` 196 196 + - **Uses**: `cm_inst` (ArithOp, LogicOp, RoutingOp, MemOp, Addr), ... 197 197 + ``` 198 198 + 199 199 + **Step 2: Update Invariants section** 200 200 + 201 201 + Old: 202 202 + ``` 203 203 + - `categorise()` covers all ALUOp/MemOp/CfgOp values; raises `ValueError` on unknown types 204 204 + ``` 205 205 + 206 206 + New: 207 207 + ``` 208 208 + - `categorise()` covers all ALUOp/MemOp values; raises `ValueError` on unknown types 209 209 + ``` 210 210 + 211 211 + **Step 3: Update freshness date** 212 212 + 213 213 + Change `<!-- freshness: 2026-02-24 -->` to `<!-- freshness: 2026-02-26 -->`. 214 214 + 215 215 + **Verification:** 216 216 + 217 217 + ```bash 218 218 + grep -rn "CfgOp" dfgraph/CLAUDE.md 219 219 + ``` 220 220 + 221 221 + Expected: No matches. 222 222 + 223 223 + **Commit:** `jj commit -m "docs: update dfgraph/CLAUDE.md for token migration"` 224 224 + 225 225 + <!-- END_TASK_3 -->

+376

docs/implementation-plans/2026-02-26-token-migration/test-requirements.md

··· 1 1 + # Token Migration: Test Requirements 2 2 + 3 3 + This document maps every acceptance criterion from the token format migration 4 4 + design plan to either an automated test or a documented human verification 5 5 + procedure. Each entry is rationalized against the implementation decisions made 6 6 + in the phase plans. 7 7 + 8 8 + **Design plan:** `docs/design-plans/2026-02-26-token-migration.md` 9 9 + **Implementation phases:** `docs/implementation-plans/2026-02-26-token-migration/phase_01.md` through `phase_05.md` 10 10 + 11 11 + --- 12 12 + 13 13 + ## Table of Contents 14 14 + 15 15 + - [AC1: Old token types removed](#ac1-old-token-types-removed) 16 16 + - [AC2: IRAMWriteToken works](#ac2-iramwritetoken-works) 17 17 + - [AC3: MemOp enum updated](#ac3-memop-enum-updated) 18 18 + - [AC4: SM T0/T1 tier split](#ac4-sm-t0t1-tier-split) 19 19 + - [AC5: EXEC opcode](#ac5-exec-opcode) 20 20 + - [AC6: Presence metadata widened](#ac6-presence-metadata-widened) 21 21 + - [AC7: Assembler updated](#ac7-assembler-updated) 22 22 + - [AC8: All tests pass](#ac8-all-tests-pass) 23 23 + - [Summary Matrix](#summary-matrix) 24 24 + 25 25 + --- 26 26 + 27 27 + ## AC1: Old token types removed 28 28 + 29 29 + ### AC1.1: `tokens.py` has no SysToken, CfgToken, IOToken, LoadInstToken, or RouteSetToken classes 30 30 + 31 31 + | Field | Value | 32 32 + |---|---| 33 33 + | **Type** | Automated (unit) | 34 34 + | **Test file** | `tests/test_exec_bootstrap.py` or inline in phase verification | 35 35 + | **Test approach** | Assert `not hasattr(tokens, 'SysToken')` and similarly for the other four deleted classes. Alternatively, a targeted parametrized test that attempts `getattr(tokens, name)` for each deleted name and expects `AttributeError`. | 36 36 + | **Implementation phase** | Phase 1, Task 2 | 37 37 + | **Rationale** | Phase 1 Task 2 rewrites `tokens.py` to contain only Token, CMToken, DyadToken, MonadToken, IRAMWriteToken, SMToken. The verification script in the task confirms the deleted types are absent. A persistent automated test ensures no regression re-introduces them. | 38 38 + 39 39 + ### AC1.2: `cm_inst.py` has no CfgOp enum 40 40 + 41 41 + | Field | Value | 42 42 + |---|---| 43 43 + | **Type** | Automated (unit) | 44 44 + | **Test file** | `tests/test_opcodes.py` | 45 45 + | **Test approach** | Assert `not hasattr(cm_inst, 'CfgOp')`. Phase 4 Task 5 already removes CfgOp assertions from this file and adds new MemOp assertions; a negative assertion that CfgOp does not exist should be included alongside. | 46 46 + | **Implementation phase** | Phase 1, Task 1 | 47 47 + | **Rationale** | Phase 1 Task 1 deletes the CfgOp class. The verification script confirms absence. The existing opcode test file is the natural home for a persistent regression check since it already validates the opcode namespace. | 48 48 + 49 49 + ### AC1.3: No module in the codebase imports any deleted type 50 50 + 51 51 + | Field | Value | 52 52 + |---|---| 53 53 + | **Type** | Automated (unit + static) | 54 54 + | **Test file** | `tests/test_exec_bootstrap.py` (or a dedicated `tests/test_migration_cleanup.py`) | 55 55 + | **Test approach** | A test that uses `grep` or `ast` module to scan all `.py` files under the project root for import statements referencing `SysToken`, `CfgToken`, `IOToken`, `LoadInstToken`, `RouteSetToken`, or `CfgOp`. Assert zero matches. | 56 56 + | **Implementation phase** | Phase 3 (Tasks 1-5 systematically remove all imports); Phase 4 (Tasks 1-6 update all test imports) | 57 57 + | **Rationale** | Phase 3 removes CfgOp from `asm/opcodes.py`, `asm/lower.py`, `asm/codegen.py`, `dfgraph/categories.py`, and `asm/__init__.py`. Phase 4 removes old imports from all test files. A codebase-wide scan test ensures completeness and catches files not explicitly listed in the phase plans. This is more robust than relying solely on per-file verification commands. | 58 58 + 59 59 + --- 60 60 + 61 61 + ## AC2: IRAMWriteToken works 62 62 + 63 63 + ### AC2.1: IRAMWriteToken routes to target PE via network (isinstance CMToken) 64 64 + 65 65 + | Field | Value | 66 66 + |---|---| 67 67 + | **Type** | Automated (integration) | 68 68 + | **Test file** | `tests/test_exec_bootstrap.py` | 69 69 + | **Test approach** | Create a topology with `build_topology()`. Inject an IRAMWriteToken via `system.inject()` with a specific `target` PE ID. Verify the token arrives in the correct PE's `input_store`. Assert `isinstance(token, CMToken)` is True. | 70 70 + | **Implementation phase** | Phase 2, Task 4 (network routing simplification) | 71 71 + | **Rationale** | Phase 2 Task 4 simplifies `_target_store` to a 2-branch isinstance check (SMToken vs CMToken). IRAMWriteToken inherits CMToken, so it routes to PEs automatically. The test confirms the inheritance-based routing works end-to-end through the System object. | 72 72 + 73 73 + ### AC2.2: PE receives IRAMWriteToken and writes instructions to IRAM at the specified offset 74 74 + 75 75 + | Field | Value | 76 76 + |---|---| 77 77 + | **Type** | Automated (unit) | 78 78 + | **Test file** | `tests/test_pe.py` | 79 79 + | **Test approach** | Construct a PE, inject an IRAMWriteToken with known `instructions` and `offset`. Run the SimPy environment. Assert `pe.iram[offset + i] == instructions[i]` for each instruction. | 80 80 + | **Implementation phase** | Phase 2, Task 2 | 81 81 + | **Rationale** | Phase 2 Task 2 replaces `_handle_cfg` with `_handle_iram_write` in the PE. The new method writes `token.instructions` into IRAM at `token.offset`. Phase 4 Task 1 updates existing PE tests and adds a new IRAMWriteToken test class covering offset=0 and non-zero offset cases. | 82 82 + 83 83 + ### AC2.3: PE executes instructions loaded via IRAMWriteToken correctly 84 84 + 85 85 + | Field | Value | 86 86 + |---|---| 87 87 + | **Type** | Automated (integration) | 88 88 + | **Test file** | `tests/test_pe.py` and `tests/test_integration.py` | 89 89 + | **Test approach** | (1) In `test_pe.py`: inject IRAMWriteToken to load an ADD instruction, then inject a DyadToken pair at the loaded offset. Verify `pe.output_log` contains the correct ADD result. (2) In `test_integration.py`: the renamed `TestIRAMWriteToken` class (formerly `TestTask5CfgTokenLoadInst`) tests the full load-then-execute path through the topology. | 90 90 + | **Implementation phase** | Phase 2, Task 2 (PE implementation); Phase 4, Tasks 1 and 3 (test updates) | 91 91 + | **Rationale** | This criterion requires end-to-end verification: IRAM load via token, then instruction execution via matching. Phase 4 Task 1 adds a test that injects IRAMWriteToken followed by a DyadToken and checks output. Phase 4 Task 3 updates the existing integration test that did the same with LoadInstToken. | 92 92 + 93 93 + ### AC2.4: IRAMWriteToken with invalid target PE raises or is dropped 94 94 + 95 95 + | Field | Value | 96 96 + |---|---| 97 97 + | **Type** | Automated (unit) | 98 98 + | **Test file** | `tests/test_exec_bootstrap.py` | 99 99 + | **Test approach** | Create a topology with PE IDs {0, 1}. Inject IRAMWriteToken with `target=99`. Assert `pytest.raises(KeyError)` from `System._target_store`. | 100 100 + | **Implementation phase** | Phase 2, Task 4 (network routing) | 101 101 + | **Rationale** | Phase 2 Task 4 implementation of `_target_store` accesses `self.pes[token.target]`, which raises KeyError for nonexistent PE IDs. Phase 4 Task 9 specifies this as a test case. The KeyError is the expected failure mode per the implementation plan. | 102 102 + 103 103 + --- 104 104 + 105 105 + ## AC3: MemOp enum updated 106 106 + 107 107 + ### AC3.1: MemOp contains EXEC, EXT, SET_PAGE, WRITE_IMM, RAW_READ, CLEAR with correct tier grouping 108 108 + 109 109 + | Field | Value | 110 110 + |---|---| 111 111 + | **Type** | Automated (unit) | 112 112 + | **Test file** | `tests/test_opcodes.py` | 113 113 + | **Test approach** | Assert each new member exists on the MemOp enum: `MemOp.EXEC`, `MemOp.EXT`, `MemOp.SET_PAGE`, `MemOp.WRITE_IMM`, `MemOp.RAW_READ`, `MemOp.CLEAR`. Assert sequential integer values match the design (READ=0 through WRITE_IMM=12). Assert tier 1 members (READ, WRITE, EXEC, ALLOC, FREE, EXT) have values 0-5 and tier 2 members (CLEAR through WRITE_IMM) have values 6-12. | 114 114 + | **Implementation phase** | Phase 1, Task 1 | 115 115 + | **Rationale** | Phase 1 Task 1 defines the new MemOp with exact sequential values. The tier grouping is documented in comments but the test should verify the value ranges to ensure hardware encoding assumptions hold. Phase 4 Task 5 adds mnemonic assertions for the new members. | 116 116 + 117 117 + ### AC3.2: ALLOC, FREE remain in tier 1 (3-bit); CLEAR in tier 2 (5-bit) 118 118 + 119 119 + | Field | Value | 120 120 + |---|---| 121 121 + | **Type** | Automated (unit) | 122 122 + | **Test file** | `tests/test_opcodes.py` | 123 123 + | **Test approach** | Assert `MemOp.ALLOC.value == 3`, `MemOp.FREE.value == 4` (tier 1, values 0-5), and `MemOp.CLEAR.value == 6` (tier 2, values 6-12). A helper `is_tier1(op)` that checks `op.value <= 5` could make this more readable. | 124 124 + | **Implementation phase** | Phase 1, Task 1 | 125 125 + | **Rationale** | The design specifies ALLOC and FREE in the 3-bit tier for full 1024-cell address range, and CLEAR in the 5-bit tier because it only applies to T1 cells in lower address space. The integer values encode this tier assignment. | 126 126 + 127 127 + ### AC3.3: Assembler mnemonic mapping includes all new opcodes 128 128 + 129 129 + | Field | Value | 130 130 + |---|---| 131 131 + | **Type** | Automated (unit) | 132 132 + | **Test file** | `tests/test_opcodes.py` and `tests/test_dfgraph_categories.py` | 133 133 + | **Test approach** | (1) In `test_opcodes.py`: assert `MNEMONIC_TO_OP["exec"] == MemOp.EXEC` and similarly for `raw_read`, `set_page`, `write_imm`, `ext`. Assert `"load_inst"` and `"route_set"` are NOT in `MNEMONIC_TO_OP`. (2) In `test_dfgraph_categories.py`: assert `categorise(MemOp.EXEC) == OpcodeCategory.MEMORY` and similarly for all new MemOp members. | 134 134 + | **Implementation phase** | Phase 3, Task 1 (opcodes.py update); Phase 4, Tasks 5 and 6 (test updates) | 135 135 + | **Rationale** | Phase 3 Task 1 adds new mnemonics to `MNEMONIC_TO_OP` and removes old CfgOp entries. Phase 3 Task 4 updates dfgraph categorisation. Phase 4 Tasks 5 and 6 add the corresponding test assertions. | 136 136 + 137 137 + --- 138 138 + 139 139 + ## AC4: SM T0/T1 tier split 140 140 + 141 141 + ### AC4.1: SM operations on addresses below tier_boundary use I-structure semantics 142 142 + 143 143 + | Field | Value | 144 144 + |---|---| 145 145 + | **Type** | Automated (unit) | 146 146 + | **Test file** | `tests/test_sm_tiers.py` | 147 147 + | **Test approach** | Send READ on address 0 (T1) to an SM where cell is EMPTY. Verify `deferred_read` is set (deferral semantics). Then WRITE to the same address. Verify deferred read is satisfied and result token is emitted. Reuses patterns from existing `tests/test_sm.py`. | 148 148 + | **Implementation phase** | Phase 2, Task 3 | 149 149 + | **Rationale** | Phase 2 Task 3 adds T0/T1 dispatch at the top of the SM `_run()` match block. Addresses below `tier_boundary` fall through to the existing T1 match block, preserving all I-structure semantics unchanged. The test verifies that T1 path is not broken by the new dispatch. | 150 150 + 151 151 + ### AC4.2: SM WRITE to T0 address stores data without presence checking 152 152 + 153 153 + | Field | Value | 154 154 + |---|---| 155 155 + | **Type** | Automated (unit) | 156 156 + | **Test file** | `tests/test_sm_tiers.py` | 157 157 + | **Test approach** | Send WRITE SMToken with `addr=256` (default tier boundary). Verify `sm.t0_store[0] == token.data`. Verify no cell in `sm.cells` was modified (no presence state change). | 158 158 + | **Implementation phase** | Phase 2, Task 3 | 159 159 + | **Rationale** | Phase 2 Task 3 implements `_handle_t0_write` which stores `token.data` (an int) into `t0_store` at `addr - tier_boundary`. The design explicitly states T0 WRITE has no presence checking. The test confirms both the storage and the absence of side effects on T1 cells. | 160 160 + 161 161 + ### AC4.3: SM READ on T0 address returns immediately (no deferral) 162 162 + 163 163 + | Field | Value | 164 164 + |---|---| 165 165 + | **Type** | Automated (unit) | 166 166 + | **Test file** | `tests/test_sm_tiers.py` | 167 167 + | **Test approach** | Send READ SMToken with `addr=256` and a return route. Verify that `sm.deferred_read` is None (no deferral). Verify the return token is emitted in the same simulation step. | 168 168 + | **Implementation phase** | Phase 2, Task 3 | 169 169 + | **Rationale** | Phase 2 Task 3 implements `_handle_t0_read` which returns immediately. The design states T0 READ has no deferral. The test checks both the positive (result emitted) and negative (no deferred_read set) aspects. | 170 170 + 171 171 + ### AC4.4: T0 storage is shared -- all SMs reference the same T0 store 172 172 + 173 173 + | Field | Value | 174 174 + |---|---| 175 175 + | **Type** | Automated (integration) | 176 176 + | **Test file** | `tests/test_sm_tiers.py` | 177 177 + | **Test approach** | Create topology with 2 SMs via `build_topology()`. Assert `sms[0].t0_store is sms[1].t0_store` (identity check, not equality). Write via SM0 to T0 address, verify SM1's `t0_store` has the same data. | 178 178 + | **Implementation phase** | Phase 2, Task 4 | 179 179 + | **Rationale** | Phase 2 Task 4 creates a single `t0_store: list[Token] = []` in `build_topology()` and assigns it to all SM instances. The identity check (`is`) confirms it is the same Python object, not a copy. The cross-SM write test confirms functional sharing. | 180 180 + 181 181 + ### AC4.5: I-structure ops on T0 address produce error 182 182 + 183 183 + | Field | Value | 184 184 + |---|---| 185 185 + | **Type** | Automated (unit) | 186 186 + | **Test file** | `tests/test_sm_tiers.py` | 187 187 + | **Test approach** | For each I-structure op (CLEAR, ALLOC, FREE, RD_INC, RD_DEC, CMP_SW), send an SMToken with that op and `addr >= tier_boundary` to the SM. Use `caplog` fixture to verify a warning is logged containing "I-structure op" and the op name. Verify the operation is dropped (no state change in t0_store or cells). Parametrize over the six ops. | 188 188 + | **Implementation phase** | Phase 2, Task 3 | 189 189 + | **Rationale** | Phase 2 Task 3 implements a catch-all `case _:` in the T0 match block that logs a warning for any op other than READ, WRITE, and EXEC. The design specifies these as errors. The implementation uses logging (not exceptions) to avoid crashing the SimPy process, which is the correct pattern for the emulator -- the SM continues processing subsequent tokens. | 190 190 + 191 191 + ### AC4.6: Tier boundary is configurable via SMConfig; default is 256 192 192 + 193 193 + | Field | Value | 194 194 + |---|---| 195 195 + | **Type** | Automated (unit) | 196 196 + | **Test file** | `tests/test_sm_tiers.py` | 197 197 + | **Test approach** | (1) Create SMConfig with default parameters; assert `tier_boundary == 256`. (2) Create SM with `tier_boundary=128`. Verify addr=127 is T1 (I-structure: READ defers) and addr=128 is T0 (raw: READ returns immediately). | 198 198 + | **Implementation phase** | Phase 2, Task 1 | 199 199 + | **Rationale** | Phase 2 Task 1 adds `tier_boundary: int = 256` to SMConfig. Phase 2 Task 3 passes it through to the StructureMemory constructor. The test with a non-default boundary value (128) confirms the boundary is actually used by the dispatch logic, not just stored. | 200 200 + 201 201 + --- 202 202 + 203 203 + ## AC5: EXEC opcode 204 204 + 205 205 + ### AC5.1: EXEC reads Token objects from T0 starting at given address and injects them into the network 206 206 + 207 207 + | Field | Value | 208 208 + |---|---| 209 209 + | **Type** | Automated (integration) | 210 210 + | **Test file** | `tests/test_exec_bootstrap.py` | 211 211 + | **Test approach** | Populate `t0_store` with Token objects (e.g., DyadTokens). Send EXEC SMToken with `addr=tier_boundary` to an SM. Verify each Token from t0_store was injected via `system.send()` by checking that target PEs/SMs received the tokens. | 212 212 + | **Implementation phase** | Phase 2, Task 3 | 213 213 + | **Rationale** | Phase 2 Task 3 implements `_handle_exec` which iterates `t0_store[addr - tier_boundary:]` and calls `system.send(entry)` for each Token. The test must pre-populate t0_store directly (not via SM WRITE, which stores ints not Tokens) per the design asymmetry documented in Phase 2 Task 3. | 214 214 + 215 215 + ### AC5.2: Injected tokens are processed normally by target PEs/SMs 216 216 + 217 217 + | Field | Value | 218 218 + |---|---| 219 219 + | **Type** | Automated (integration) | 220 220 + | **Test file** | `tests/test_exec_bootstrap.py` | 221 221 + | **Test approach** | Populate T0 with a DyadToken targeting PE0 (which has an ADD instruction loaded). EXEC from SM. Run simulation. Verify PE0 received the token and produced the expected output in `pe.output_log`. | 222 222 + | **Implementation phase** | Phase 2, Task 4 | 223 223 + | **Rationale** | Phase 2 Task 4 wires the System reference to all SMs, enabling EXEC to call `system.send()`. The injected tokens go through the normal routing path (`_target_store`), so they are processed identically to directly injected tokens. The test verifies the full chain: EXEC -> send -> route -> PE process -> output. | 224 224 + 225 225 + ### AC5.3: EXEC can load a program (IRAM writes + seed tokens) from T0 that executes correctly 226 226 + 227 227 + | Field | Value | 228 228 + |---|---| 229 229 + | **Type** | Automated (e2e) | 230 230 + | **Test file** | `tests/test_exec_bootstrap.py` | 231 231 + | **Test approach** | Full bootstrap test: (1) Populate T0 with IRAMWriteToken (loads ADD instruction into PE0 IRAM) and a DyadToken pair (seed operands for the ADD). (2) EXEC from SM. (3) Run simulation. (4) Verify PE0 loaded IRAM, matched the operand pair, executed ADD, and produced the correct result. | 232 232 + | **Implementation phase** | Phase 2, Tasks 3 and 4 | 233 233 + | **Rationale** | This is the highest-fidelity test of the bootstrap mechanism. It exercises the full path that replaces the old CfgToken-based program loading. The design describes EXEC as the bootstrap path (loading programs from ROM at reset). This test simulates that scenario in miniature. | 234 234 + 235 235 + ### AC5.4: EXEC on empty T0 region is a no-op 236 236 + 237 237 + | Field | Value | 238 238 + |---|---| 239 239 + | **Type** | Automated (unit) | 240 240 + | **Test file** | `tests/test_exec_bootstrap.py` | 241 241 + | **Test approach** | Send EXEC SMToken with `addr` beyond the current `t0_store` length. Verify no tokens are injected (all PE/SM input stores remain empty). Also test with `addr` pointing to a `None` entry in t0_store. | 242 242 + | **Implementation phase** | Phase 2, Task 3 | 243 243 + | **Rationale** | Phase 2 Task 3 implements `_handle_exec` with a bounds check (`t0_idx >= len(self.t0_store)` returns early) and a `None` sentinel break (`if entry is None: break`). Both paths are tested. | 244 244 + 245 245 + --- 246 246 + 247 247 + ## AC6: Presence metadata widened 248 248 + 249 249 + ### AC6.1: SMCell has is_wide field (default False) 250 250 + 251 251 + | Field | Value | 252 252 + |---|---| 253 253 + | **Type** | Automated (unit) | 254 254 + | **Test file** | `tests/test_sm_tiers.py` (or `tests/test_sm.py`) | 255 255 + | **Test approach** | Construct `SMCell(Presence.EMPTY, None, None)`. Assert `cell.is_wide is False`. Construct `SMCell(Presence.EMPTY, None, None, is_wide=True)`. Assert `cell.is_wide is True`. | 256 256 + | **Implementation phase** | Phase 1, Task 3 | 257 257 + | **Rationale** | Phase 1 Task 3 adds `is_wide: bool = False` to the SMCell dataclass. The default value ensures backward compatibility. The test confirms both the default and explicit-True paths. | 258 258 + 259 259 + ### AC6.2: Existing I-structure behaviour unchanged (is_wide=False path) 260 260 + 261 261 + | Field | Value | 262 262 + |---|---| 263 263 + | **Type** | Automated (integration) | 264 264 + | **Test file** | `tests/test_sm.py` (existing tests) and `tests/test_sm_tiers.py` | 265 265 + | **Test approach** | All existing `tests/test_sm.py` tests continue to pass without modification (they exercise I-structure semantics with the default `is_wide=False`). In `test_sm_tiers.py`, explicitly verify T1 operations with `is_wide=False` produce identical behaviour to pre-migration. | 266 266 + | **Implementation phase** | Phase 1, Task 3 (field addition); Phase 2, Task 3 (T1 path preservation) | 267 267 + | **Rationale** | The `is_wide` field has no behavioral effect in the current implementation -- it is a metadata placeholder for future use. The existing SM test suite serves as a regression suite ensuring I-structure semantics are unchanged. Phase 4 Task 8 explicitly includes AC6.2 in the SM tier test plan. | 268 268 + 269 269 + --- 270 270 + 271 271 + ## AC7: Assembler updated 272 272 + 273 273 + ### AC7.1: Token stream mode emits IRAMWriteToken (not LoadInstToken) 274 274 + 275 275 + | Field | Value | 276 276 + |---|---| 277 277 + | **Type** | Automated (unit) | 278 278 + | **Test file** | `tests/test_codegen.py` | 279 279 + | **Test approach** | Call `generate_tokens()` on an assembled IRGraph. Filter output tokens by type. Assert at least one `IRAMWriteToken` is present. Assert zero `LoadInstToken` instances (import would fail if attempted, but check by string name or absence). | 280 280 + | **Implementation phase** | Phase 3, Task 3 | 281 281 + | **Rationale** | Phase 3 Task 3 replaces LoadInstToken construction with IRAMWriteToken in `generate_tokens()`. Phase 4 Task 2 updates the test assertions. The test checks both the positive (IRAMWriteToken present) and negative (no LoadInstToken) conditions. | 282 282 + 283 283 + ### AC7.2: Token stream mode does not emit RouteSetToken 284 284 + 285 285 + | Field | Value | 286 286 + |---|---| 287 287 + | **Type** | Automated (unit) | 288 288 + | **Test file** | `tests/test_codegen.py` | 289 289 + | **Test approach** | Call `generate_tokens()`. Assert no token in the output has type name `RouteSetToken`. The old ordering was `SM init -> ROUTE_SET -> LOAD_INST -> seeds`; new ordering is `SM init -> IRAM writes -> seeds`. Verify the ordering: all SMTokens precede all IRAMWriteTokens, which precede all seed MonadTokens. | 290 290 + | **Implementation phase** | Phase 3, Task 3 | 291 291 + | **Rationale** | Phase 3 Task 3 deletes the ROUTE_SET emission section entirely. The design defers route restriction to a future mechanism. Phase 4 Task 2 removes RouteSetToken assertions and updates ordering checks. | 292 292 + 293 293 + ### AC7.3: Direct mode (PEConfig/SMConfig) still works 294 294 + 295 295 + | Field | Value | 296 296 + |---|---| 297 297 + | **Type** | Automated (unit) | 298 298 + | **Test file** | `tests/test_codegen.py` | 299 299 + | **Test approach** | Call `generate_direct()` on an assembled IRGraph. Assert the result contains valid PEConfig and SMConfig objects with correct IRAM contents and seed tokens. This is an existing test that must continue to pass. | 300 300 + | **Implementation phase** | Phase 3, Task 3 (implementation preserves direct mode) | 301 301 + | **Rationale** | Phase 3 Task 3 only modifies `generate_tokens()`. The `generate_direct()` path is unchanged because it produces PEConfig/SMConfig objects directly, not token streams. The existing tests serve as regression coverage. | 302 302 + 303 303 + ### AC7.4: Assembler round-trip (serialize -> parse -> assemble) works with updated types 304 304 + 305 305 + | Field | Value | 306 306 + |---|---| 307 307 + | **Type** | Automated (integration) | 308 308 + | **Test file** | `tests/test_codegen.py` and `tests/test_serialize.py` | 309 309 + | **Test approach** | Use `round_trip()` API: serialize an IRGraph to dfasm source, parse it back, re-assemble, and compare the output. Assert the round-tripped result matches the original. | 310 310 + | **Implementation phase** | Phase 3, Tasks 1-3 (assembler pipeline updates) | 311 311 + | **Rationale** | The round-trip test exercises the full pipeline: serialize -> parse -> lower -> resolve -> place -> allocate -> codegen. If any step has stale CfgOp references, the round-trip will fail. Existing round-trip tests in `test_codegen.py` and `test_serialize.py` provide this coverage. | 312 312 + 313 313 + --- 314 314 + 315 315 + ## AC8: All tests pass 316 316 + 317 317 + ### AC8.1: `python -m pytest tests/ -v` exits with zero failures 318 318 + 319 319 + | Field | Value | 320 320 + |---|---| 321 321 + | **Type** | Automated (e2e) | 322 322 + | **Test file** | Full test suite: `tests/` | 323 323 + | **Test approach** | Run `python -m pytest tests/ -v`. Assert exit code 0 and zero failures in output. | 324 324 + | **Implementation phase** | Phase 4, Task 10 | 325 325 + | **Rationale** | This is the global gate criterion. Phase 4 Task 10 is explicitly a verification-only task that runs the full suite. All preceding phases build toward this. If any phase introduces a regression, this criterion catches it. This should be the final check before Phase 5 (documentation). | 326 326 + 327 327 + --- 328 328 + 329 329 + ## Human Verification Criteria 330 330 + 331 331 + No acceptance criteria in this migration require human verification that cannot be automated. All criteria are testable through automated means: 332 332 + 333 333 + - **Type deletions** (AC1): testable via `hasattr` checks and codebase-wide grep tests 334 334 + - **Runtime behaviour** (AC2, AC4, AC5): testable via SimPy simulation with assertions on state 335 335 + - **Enum values** (AC3): testable via direct value comparison 336 336 + - **Metadata fields** (AC6): testable via dataclass construction and attribute checks 337 337 + - **Assembler output** (AC7): testable via token stream inspection and round-trip 338 338 + - **Suite health** (AC8): testable via pytest exit code 339 339 + 340 340 + If any criterion were to require human verification, it would be documented here with justification and a step-by-step verification procedure. The design plan intentionally scoped this migration to changes with deterministic, observable outcomes. 341 341 + 342 342 + --- 343 343 + 344 344 + ## Summary Matrix 345 345 + 346 346 + | AC | Criterion | Test Type | Test File | Phase | 347 347 + |---|---|---|---|---| 348 348 + | AC1.1 | Old token classes deleted from `tokens.py` | Unit | `tests/test_exec_bootstrap.py` | P1T2, P4T9 | 349 349 + | AC1.2 | CfgOp deleted from `cm_inst.py` | Unit | `tests/test_opcodes.py` | P1T1, P4T5 | 350 350 + | AC1.3 | No imports of deleted types in codebase | Unit (static) | `tests/test_exec_bootstrap.py` | P3, P4 | 351 351 + | AC2.1 | IRAMWriteToken routes to PE via CMToken isinstance | Integration | `tests/test_exec_bootstrap.py` | P2T4, P4T9 | 352 352 + | AC2.2 | PE writes IRAM from IRAMWriteToken | Unit | `tests/test_pe.py` | P2T2, P4T1 | 353 353 + | AC2.3 | PE executes IRAMWriteToken-loaded instructions | Integration | `tests/test_pe.py`, `tests/test_integration.py` | P2T2, P4T1/T3 | 354 354 + | AC2.4 | Invalid target PE raises KeyError | Unit | `tests/test_exec_bootstrap.py` | P2T4, P4T9 | 355 355 + | AC3.1 | MemOp has all new opcodes with correct values | Unit | `tests/test_opcodes.py` | P1T1, P4T5 | 356 356 + | AC3.2 | ALLOC/FREE tier 1, CLEAR tier 2 | Unit | `tests/test_opcodes.py` | P1T1, P4T5 | 357 357 + | AC3.3 | Mnemonic mapping includes new opcodes | Unit | `tests/test_opcodes.py`, `tests/test_dfgraph_categories.py` | P3T1, P4T5/T6 | 358 358 + | AC4.1 | T1 addresses use I-structure semantics | Unit | `tests/test_sm_tiers.py` | P2T3, P4T8 | 359 359 + | AC4.2 | T0 WRITE stores without presence checking | Unit | `tests/test_sm_tiers.py` | P2T3, P4T8 | 360 360 + | AC4.3 | T0 READ returns immediately | Unit | `tests/test_sm_tiers.py` | P2T3, P4T8 | 361 361 + | AC4.4 | T0 store shared across all SMs | Integration | `tests/test_sm_tiers.py` | P2T4, P4T8 | 362 362 + | AC4.5 | I-structure ops on T0 produce error | Unit | `tests/test_sm_tiers.py` | P2T3, P4T8 | 363 363 + | AC4.6 | Tier boundary configurable, default 256 | Unit | `tests/test_sm_tiers.py` | P2T1, P4T8 | 364 364 + | AC5.1 | EXEC reads from T0 and injects into network | Integration | `tests/test_exec_bootstrap.py` | P2T3, P4T9 | 365 365 + | AC5.2 | Injected tokens processed normally | Integration | `tests/test_exec_bootstrap.py` | P2T4, P4T9 | 366 366 + | AC5.3 | EXEC bootstraps a full program | E2E | `tests/test_exec_bootstrap.py` | P2T3/T4, P4T9 | 367 367 + | AC5.4 | EXEC on empty T0 is no-op | Unit | `tests/test_exec_bootstrap.py` | P2T3, P4T9 | 368 368 + | AC6.1 | SMCell has is_wide field | Unit | `tests/test_sm_tiers.py` | P1T3, P4T8 | 369 369 + | AC6.2 | Existing I-structure behaviour unchanged | Integration | `tests/test_sm.py` (existing), `tests/test_sm_tiers.py` | P1T3, P4T8 | 370 370 + | AC7.1 | Token stream emits IRAMWriteToken | Unit | `tests/test_codegen.py` | P3T3, P4T2 | 371 371 + | AC7.2 | No RouteSetToken in token stream | Unit | `tests/test_codegen.py` | P3T3, P4T2 | 372 372 + | AC7.3 | Direct mode still works | Unit | `tests/test_codegen.py` | P3T3, P4T2 | 373 373 + | AC7.4 | Round-trip works with updated types | Integration | `tests/test_codegen.py`, `tests/test_serialize.py` | P3, P4T2 | 374 374 + | AC8.1 | Full test suite passes | E2E | `tests/` (all) | P4T10 | 375 375 + 376 376 + **Legend:** P = Phase, T = Task. E.g., P2T3 = Phase 2, Task 3.

+11 -4

emu/network.py

··· 4 4 from emu.sm import StructureMemory 5 5 from emu.types import PEConfig, SMConfig 6 6 from sm_mod import SMCell 7 7 - from tokens import CMToken, CfgToken, SMToken, Token 7 7 + from tokens import CMToken, SMToken, Token 8 8 9 9 10 10 class System: ··· 21 21 def inject(self, token: Token) -> None: 22 22 """Inject a token into the appropriate element's input store (direct append). 23 23 24 24 - Routes by type: SMToken → target SM, CMToken/CfgToken → target PE. 24 24 + Routes by type: SMToken → target SM, CMToken → target PE. 25 25 Uses direct list append (bypasses SimPy put), suitable for pre-simulation setup. 26 26 """ 27 27 store = self._target_store(token) ··· 50 50 """Resolve the destination store for a token.""" 51 51 if isinstance(token, SMToken): 52 52 return self.sms[token.target].input_store 53 53 - if isinstance(token, (CMToken, CfgToken)): 53 53 + if isinstance(token, CMToken): 54 54 return self.pes[token.target].input_store 55 55 raise TypeError(f"Unknown token type: {type(token).__name__}") 56 56 ··· 76 76 if cfg.gen_counters is not None: 77 77 pe.gen_counters = list(cfg.gen_counters) 78 78 pes[cfg.pe_id] = pe 79 79 + 80 80 + t0_store: list[Token] = [] 79 81 80 82 for cfg in sm_configs: 81 83 sm = StructureMemory( ··· 83 85 sm_id=cfg.sm_id, 84 86 cell_count=cfg.cell_count, 85 87 fifo_capacity=fifo_capacity, 88 88 + tier_boundary=cfg.tier_boundary, 86 89 ) 90 90 + sm.t0_store = t0_store 87 91 if cfg.initial_cells is not None: 88 92 for addr, (pres, data) in cfg.initial_cells.items(): 89 93 sm.cells[addr] = SMCell(pres, data, None) ··· 113 117 if sid in cfg.allowed_sm_routes 114 118 } 115 119 116 116 - return System(env, pes, sms) 120 120 + system = System(env, pes, sms) 121 121 + for sm in sms.values(): 122 122 + sm.system = system 123 123 + return system

+9 -22

emu/pe.py

··· 4 4 5 5 import simpy 6 6 7 7 - from cm_inst import ALUInst, Addr, ArithOp, CfgOp, LogicOp, MemOp, Port, RoutingOp, SMInst, is_monadic_alu 7 7 + from cm_inst import ALUInst, Addr, ArithOp, LogicOp, MemOp, Port, RoutingOp, SMInst, is_monadic_alu 8 8 from emu.alu import execute 9 9 from emu.types import MatchEntry 10 10 from tokens import ( 11 11 - CMToken, CfgToken, DyadToken, LoadInstToken, 12 12 - MonadToken, RouteSetToken, SMToken, 11 11 + CMToken, DyadToken, IRAMWriteToken, 12 12 + MonadToken, SMToken, 13 13 ) 14 14 15 15 logger = logging.getLogger(__name__) ··· 53 53 while True: 54 54 token = yield self.input_store.get() 55 55 56 56 - if isinstance(token, CfgToken): 57 57 - self._handle_cfg(token) 56 56 + if isinstance(token, IRAMWriteToken): 57 57 + self._handle_iram_write(token) 58 58 continue 59 59 60 60 if isinstance(token, MonadToken): ··· 84 84 result, bool_out = execute(inst.op, left, right, inst.const) 85 85 yield from self._emit(inst, result, bool_out, token.ctx) 86 86 87 87 - def _handle_cfg(self, token: CfgToken) -> None: 88 88 - """Handle configuration tokens for dynamic IRAM updates and routing setup.""" 89 89 - if isinstance(token, LoadInstToken): 90 90 - base_addr = token.addr if token.addr is not None else 0 91 91 - for i, inst in enumerate(token.instructions): 92 92 - self.iram[base_addr + i] = inst 93 93 - elif isinstance(token, RouteSetToken): 94 94 - self.route_table = { 95 95 - pid: store for pid, store in self.route_table.items() 96 96 - if pid in token.pe_routes 97 97 - } 98 98 - self.sm_routes = { 99 99 - sid: store for sid, store in self.sm_routes.items() 100 100 - if sid in token.sm_routes 101 101 - } 102 102 - else: 103 103 - logger.warning("PE%d: unknown CfgOp: %s", self.pe_id, token.op) 87 87 + def _handle_iram_write(self, token: IRAMWriteToken) -> None: 88 88 + """Write instructions into IRAM at the offset specified by the token.""" 89 89 + for i, inst in enumerate(token.instructions): 90 90 + self.iram[token.offset + i] = inst 104 91 105 92 def _match_monadic(self, token: MonadToken) -> tuple[int, None]: 106 93 return (token.data, None)

+90 -2

emu/sm.py

··· 1 1 + from __future__ import annotations 2 2 + 1 3 import logging 2 4 from dataclasses import replace 3 3 - from typing import Optional 5 5 + from typing import TYPE_CHECKING, Optional 4 6 5 7 import simpy 6 8 ··· 8 10 from emu.alu import UINT16_MASK 9 11 from emu.types import DeferredRead 10 12 from sm_mod import Presence, SMCell 11 11 - from tokens import CMToken, MonadToken, SMToken 13 13 + from tokens import CMToken, MonadToken, SMToken, Token 14 14 + 15 15 + if TYPE_CHECKING: 16 16 + from emu.network import System 12 17 13 18 logger = logging.getLogger(__name__) 14 19 ··· 22 27 sm_id: int, 23 28 cell_count: int = 512, 24 29 fifo_capacity: int = 8, 30 30 + tier_boundary: int = 256, 25 31 ): 26 32 self.env = env 27 33 self.sm_id = sm_id 34 34 + self.tier_boundary = tier_boundary 28 35 self.cells: list[SMCell] = [ 29 36 SMCell(Presence.EMPTY, None, None) for _ in range(cell_count) 30 37 ] ··· 33 40 self._deferred_cancelled: bool = False 34 41 self.input_store: simpy.Store = simpy.Store(env, capacity=fifo_capacity) 35 42 self.route_table: dict[int, simpy.Store] = {} 43 43 + self.t0_store: list[Token] = [] 44 44 + self.system: Optional[System] = None 36 45 self.process = env.process(self._run()) 46 46 + 47 47 + def _is_t0(self, addr: int) -> bool: 48 48 + """Return True if address falls in the T0 (shared raw storage) region.""" 49 49 + return addr >= self.tier_boundary 37 50 38 51 def _run(self): 39 52 while True: ··· 48 61 addr = token.addr 49 62 op = token.op 50 63 64 64 + if self._is_t0(addr): 65 65 + match op: 66 66 + case MemOp.READ: 67 67 + yield from self._handle_t0_read(addr, token) 68 68 + case MemOp.WRITE: 69 69 + self._handle_t0_write(addr, token) 70 70 + case MemOp.EXEC: 71 71 + yield from self._handle_exec(addr) 72 72 + case _: 73 73 + logger.warning( 74 74 + "SM%d: I-structure op %s on T0 address %d", 75 75 + self.sm_id, op.name, addr, 76 76 + ) 77 77 + continue 78 78 + 51 79 match op: 52 80 case MemOp.READ: 53 81 yield from self._handle_read(addr, token) ··· 65 93 self._handle_alloc(addr) 66 94 case MemOp.FREE: 67 95 self._handle_clear(addr) 96 96 + case MemOp.EXEC: 97 97 + logger.warning( 98 98 + "SM%d: EXEC on T1 address %d (must be T0)", 99 99 + self.sm_id, addr, 100 100 + ) 101 101 + case MemOp.SET_PAGE | MemOp.WRITE_IMM | MemOp.RAW_READ | MemOp.EXT: 102 102 + raise NotImplementedError( 103 103 + f"SM{self.sm_id}: {op.name} not yet implemented" 104 104 + ) 68 105 case _: 69 106 logger.warning("SM%d: unknown op %s", self.sm_id, op) 70 107 ··· 165 202 def _send_result(self, return_route: CMToken, data: int): 166 203 result = replace(return_route, data=data) 167 204 yield self.route_table[return_route.target].put(result) 205 205 + 206 206 + def _handle_t0_read(self, addr: int, token: SMToken): 207 207 + """T0 READ: return stored data immediately, no presence tracking or deferral.""" 208 208 + if token.ret is None: 209 209 + return 210 210 + t0_idx = addr - self.tier_boundary 211 211 + if t0_idx < len(self.t0_store): 212 212 + entry = self.t0_store[t0_idx] 213 213 + if isinstance(entry, int): 214 214 + yield from self._send_result(token.ret, entry) 215 215 + elif entry is not None: 216 216 + yield from self._send_result(token.ret, 0) 217 217 + else: 218 218 + yield from self._send_result(token.ret, 0) 219 219 + else: 220 220 + yield from self._send_result(token.ret, 0) 221 221 + 222 222 + def _handle_t0_write(self, addr: int, token: SMToken): 223 223 + """T0 WRITE: store raw integer data without presence checking. 224 224 + 225 225 + Note: This stores token.data (an int) into t0_store. For Token object 226 226 + storage (needed by EXEC), populate t0_store directly before simulation 227 227 + starts (e.g., via build_topology or test setup). This asymmetry is 228 228 + intentional — SM WRITE provides data-level access (lookup tables, 229 229 + shared counters), while EXEC consumes pre-loaded Token objects for 230 230 + bootstrap. Future work will unify T0 as list[int] with token 231 231 + serialisation/deserialisation. 232 232 + """ 233 233 + t0_idx = addr - self.tier_boundary 234 234 + while len(self.t0_store) <= t0_idx: 235 235 + self.t0_store.append(None) 236 236 + self.t0_store[t0_idx] = token.data 237 237 + 238 238 + def _handle_exec(self, addr: int): 239 239 + """EXEC: read Token objects from T0 starting at addr and inject into network. 240 240 + 241 241 + Uses send() to properly trigger SimPy Store.put() events, ensuring that 242 242 + tokens wake up pending get() operations in target PEs/SMs. 243 243 + """ 244 244 + if self.system is None: 245 245 + logger.warning("SM%d: EXEC but no system reference", self.sm_id) 246 246 + return 247 247 + t0_idx = addr - self.tier_boundary 248 248 + if t0_idx >= len(self.t0_store): 249 249 + return 250 250 + for entry in self.t0_store[t0_idx:]: 251 251 + if entry is None: 252 252 + break 253 253 + if isinstance(entry, Token): 254 254 + # Use send() which properly triggers SimPy Store.put() events 255 255 + yield from self.system.send(entry)

+1

emu/types.py

··· 35 35 sm_id: int 36 36 cell_count: int = 512 37 37 initial_cells: Optional[dict[int, tuple[Presence, Optional[int]]]] = None 38 38 + tier_boundary: int = 256

+1

sm_mod.py

··· 15 15 pres: Presence 16 16 data_l: Optional[int] # data or length 17 17 data_r: Optional[List[int]] # optional data 18 18 + is_wide: bool = False

+10 -2

tests/conftest.py

··· 48 48 ) 49 49 50 50 51 51 - sm_all_ops = st.sampled_from(list(MemOp)) 51 51 + # SM ops: implemented for T1 are READ, WRITE, CLEAR, RD_INC, RD_DEC, CMP_SW, ALLOC, FREE, EXEC 52 52 + # Unimplemented: SET_PAGE, WRITE_IMM, RAW_READ, EXT 53 53 + sm_implemented_ops = [ 54 54 + MemOp.READ, MemOp.WRITE, MemOp.CLEAR, MemOp.RD_INC, MemOp.RD_DEC, 55 55 + MemOp.CMP_SW, MemOp.ALLOC, MemOp.FREE, MemOp.EXEC 56 56 + ] 57 57 + sm_all_ops = st.sampled_from(sm_implemented_ops) 52 58 53 59 54 60 @st.composite 55 61 def sm_token(draw, addr=None, op=None, data=None): 56 56 - _addr = draw(st.integers(min_value=0, max_value=511)) if addr is None else addr 62 62 + # By default, generate T1 addresses (below tier_boundary of 256) 63 63 + # Tests that specifically need T0 addresses should pass them explicitly 64 64 + _addr = draw(st.integers(min_value=0, max_value=255)) if addr is None else addr 57 65 _op = draw(sm_all_ops) if op is None else op 58 66 _data = draw(uint16) if data is None else data 59 67 ret = CMToken(target=0, offset=0, ctx=0, data=0)

+193 -42

tests/test_codegen.py

··· 1 1 """Tests for code generation. 2 2 3 3 Tests verify: 4 4 + - token-migration.AC7.1: Token stream mode emits IRAMWriteToken (not LoadInstToken) 5 5 + - token-migration.AC7.2: Token stream mode does not emit RouteSetToken 6 6 + - token-migration.AC7.3: Direct mode (PEConfig/SMConfig) still works 7 7 + 8 8 + Also tests original codegen AC8 criteria: 4 9 - or1-asm.AC8.1: Direct mode produces valid PEConfig with correct IRAM contents 5 10 - or1-asm.AC8.2: Direct mode produces valid SMConfig with initial cell values 6 11 - or1-asm.AC8.3: Direct mode produces seed MonadTokens for const nodes with no incoming edges 7 12 - or1-asm.AC8.4: Direct mode PEConfig includes route restrictions matching edge analysis 8 8 - - or1-asm.AC8.5: Token stream mode emits SM init tokens before ROUTE_SET tokens 9 9 - - or1-asm.AC8.6: Token stream mode emits ROUTE_SET tokens before LOAD_INST tokens 10 10 - - or1-asm.AC8.7: Token stream mode emits LOAD_INST tokens before seed tokens 11 11 - - or1-asm.AC8.8: Token stream mode produces valid tokens consumable by emulator 12 13 - or1-asm.AC8.9: Program with no data_defs produces empty SM init section 13 13 - - or1-asm.AC8.10: Single PE program produces ROUTE_SET with only self-routes 14 14 """ 15 15 16 16 from asm.codegen import generate_direct, generate_tokens, AssemblyResult ··· 23 23 SourceLoc, 24 24 ResolvedDest, 25 25 ) 26 26 - from cm_inst import ALUInst, Addr, ArithOp, CfgOp, MemOp, Port, RoutingOp, SMInst 27 27 - from tokens import CfgToken, LoadInstToken, MonadToken, RouteSetToken, SMToken 26 26 + from cm_inst import ALUInst, Addr, ArithOp, MemOp, Port, RoutingOp, SMInst 27 27 + from tokens import IRAMWriteToken, MonadToken, SMToken 28 28 from emu.types import PEConfig, SMConfig 29 29 from sm_mod import Presence 30 30 31 31 32 32 + class TestTokenMigration: 33 33 + """Token migration acceptance criteria (AC7.1, AC7.2, AC7.3).""" 34 34 + 35 35 + def test_ac71_iram_write_token_in_stream(self): 36 36 + """AC7.1: Token stream mode emits IRAMWriteToken (not LoadInstToken). 37 37 + 38 38 + Tests that: 39 39 + - generate_tokens() produces IRAMWriteToken instances 40 40 + - At least one IRAMWriteToken is present for each PE with instructions 41 41 + """ 42 42 + node = IRNode( 43 43 + name="&add", 44 44 + opcode=ArithOp.ADD, 45 45 + pe=0, 46 46 + iram_offset=0, 47 47 + ctx=0, 48 48 + loc=SourceLoc(1, 1), 49 49 + ) 50 50 + system = SystemConfig(pe_count=1, sm_count=1) 51 51 + graph = IRGraph({"&add": node}, system=system) 52 52 + 53 53 + tokens = generate_tokens(graph) 54 54 + 55 55 + # Find IRAMWriteToken instances 56 56 + iram_write_tokens = [ 57 57 + t for t in tokens if isinstance(t, IRAMWriteToken) 58 58 + ] 59 59 + 60 60 + assert len(iram_write_tokens) > 0, "Should emit at least one IRAMWriteToken (AC7.1)" 61 61 + for token in iram_write_tokens: 62 62 + assert isinstance(token.instructions, tuple), "IRAMWriteToken should have instructions tuple" 63 63 + assert len(token.instructions) > 0, "IRAMWriteToken instructions should not be empty" 64 64 + 65 65 + def test_ac72_no_route_set_token(self): 66 66 + """AC7.2: Token stream mode does not emit RouteSetToken. 67 67 + 68 68 + Tests that: 69 69 + - generate_tokens() produces no RouteSetToken instances 70 70 + """ 71 71 + # Multi-PE graph to test routing 72 72 + node_pe0 = IRNode( 73 73 + name="&a", 74 74 + opcode=ArithOp.ADD, 75 75 + pe=0, 76 76 + iram_offset=0, 77 77 + ctx=0, 78 78 + dest_l=ResolvedDest( 79 79 + name="&b", 80 80 + addr=Addr(a=0, port=Port.L, pe=1), 81 81 + ), 82 82 + loc=SourceLoc(1, 1), 83 83 + ) 84 84 + node_pe1 = IRNode( 85 85 + name="&b", 86 86 + opcode=ArithOp.SUB, 87 87 + pe=1, 88 88 + iram_offset=0, 89 89 + ctx=0, 90 90 + loc=SourceLoc(2, 1), 91 91 + ) 92 92 + edge = IREdge(source="&a", dest="&b", port=Port.L, loc=SourceLoc(1, 1)) 93 93 + system = SystemConfig(pe_count=2, sm_count=1) 94 94 + graph = IRGraph( 95 95 + {"&a": node_pe0, "&b": node_pe1}, 96 96 + edges=[edge], 97 97 + system=system, 98 98 + ) 99 99 + 100 100 + tokens = generate_tokens(graph) 101 101 + 102 102 + # Verify no RouteSetToken (check by class name to be robust) 103 103 + route_set_tokens = [ 104 104 + t for t in tokens 105 105 + if type(t).__name__ == 'RouteSetToken' 106 106 + ] 107 107 + 108 108 + assert len(route_set_tokens) == 0, "Should not emit RouteSetToken (AC7.2)" 109 109 + 110 110 + def test_ac73_direct_mode_still_works(self): 111 111 + """AC7.3: Direct mode (PEConfig/SMConfig) still works. 112 112 + 113 113 + Tests that: 114 114 + - generate_direct() produces valid PEConfig with correct IRAM, route restrictions 115 115 + - generate_direct() produces valid SMConfig with initial cell values 116 116 + - Seed tokens are generated correctly 117 117 + """ 118 118 + data_def = IRDataDef( 119 119 + name="@val", 120 120 + sm_id=0, 121 121 + cell_addr=5, 122 122 + value=42, 123 123 + loc=SourceLoc(1, 1), 124 124 + ) 125 125 + node1 = IRNode( 126 126 + name="&a", 127 127 + opcode=ArithOp.ADD, 128 128 + pe=0, 129 129 + iram_offset=0, 130 130 + ctx=0, 131 131 + loc=SourceLoc(1, 1), 132 132 + ) 133 133 + node2 = IRNode( 134 134 + name="&b", 135 135 + opcode=RoutingOp.CONST, 136 136 + pe=0, 137 137 + iram_offset=1, 138 138 + ctx=0, 139 139 + const=99, 140 140 + loc=SourceLoc(2, 1), 141 141 + ) 142 142 + system = SystemConfig(pe_count=1, sm_count=1) 143 143 + graph = IRGraph( 144 144 + {"&a": node1, "&b": node2}, 145 145 + data_defs=[data_def], 146 146 + system=system, 147 147 + ) 148 148 + 149 149 + result = generate_direct(graph) 150 150 + 151 151 + # Verify PEConfig 152 152 + assert len(result.pe_configs) == 1 153 153 + pe_config = result.pe_configs[0] 154 154 + assert pe_config.pe_id == 0 155 155 + assert len(pe_config.iram) == 2 156 156 + assert pe_config.allowed_pe_routes == {0} 157 157 + assert pe_config.allowed_sm_routes == set() 158 158 + 159 159 + # Verify SMConfig 160 160 + assert len(result.sm_configs) == 1 161 161 + sm_config = result.sm_configs[0] 162 162 + assert sm_config.sm_id == 0 163 163 + assert 5 in sm_config.initial_cells 164 164 + pres, val = sm_config.initial_cells[5] 165 165 + assert pres == Presence.FULL 166 166 + assert val == 42 167 167 + 168 168 + # Verify seed tokens 169 169 + assert len(result.seed_tokens) == 1 170 170 + seed = result.seed_tokens[0] 171 171 + assert isinstance(seed, MonadToken) 172 172 + assert seed.data == 99 173 173 + 174 174 + 32 175 class TestDirectMode: 33 176 """AC8.1, AC8.2, AC8.3, AC8.4: Direct mode code generation.""" 34 177 ··· 219 362 220 363 221 364 class TestTokenStream: 222 222 - """AC8.5, AC8.6, AC8.7, AC8.8: Token stream generation and ordering.""" 365 365 + """AC7.1, AC7.2, AC7.3: Token stream generation and ordering.""" 223 366 224 367 def test_ac85_ac86_ac87_token_ordering(self): 225 225 - """AC8.5-8.7: Tokens are emitted in correct order. 368 368 + """AC7.1-7.2: Token stream emits SM init, IRAM writes, then seeds (no ROUTE_SET or LOAD_INST). 226 369 227 370 Tests that: 228 371 - SM init tokens come first 229 229 - - ROUTE_SET tokens come next 230 230 - - LOAD_INST tokens come next 372 372 + - IRAM write tokens come next (IRAMWriteToken, not LoadInstToken) 231 373 - Seed tokens come last 374 374 + - No RouteSetToken is present 232 375 """ 233 376 # Create a multi-PE graph with data_defs 234 377 data_def = IRDataDef( ··· 269 412 i for i, t in enumerate(tokens) 270 413 if isinstance(t, SMToken) 271 414 ] 272 272 - route_set_indices = [ 273 273 - i for i, t in enumerate(tokens) 274 274 - if isinstance(t, CfgToken) and t.op == CfgOp.ROUTE_SET 275 275 - ] 276 276 - load_inst_indices = [ 415 415 + iram_write_indices = [ 277 416 i for i, t in enumerate(tokens) 278 278 - if isinstance(t, CfgToken) and t.op == CfgOp.LOAD_INST 417 417 + if isinstance(t, IRAMWriteToken) 279 418 ] 280 419 seed_indices = [ 281 420 i for i, t in enumerate(tokens) if isinstance(t, MonadToken) 282 421 ] 283 422 284 284 - # Verify order: SM < ROUTE_SET < LOAD_INST < seed 423 423 + # Verify order: SM < IRAM write < seed 285 424 assert smtoken_indices, "Should have at least one SM token" 286 286 - assert route_set_indices, "Should have at least one ROUTE_SET token" 287 287 - assert load_inst_indices, "Should have at least one LOAD_INST token" 425 425 + assert iram_write_indices, "Should have at least one IRAM write token" 288 426 assert seed_indices, "Should have at least one seed token" 289 289 - assert max(smtoken_indices) < min(route_set_indices), "SM tokens should come before ROUTE_SET" 290 290 - assert max(route_set_indices) < min(load_inst_indices), "ROUTE_SET should come before LOAD_INST" 291 291 - assert max(load_inst_indices) < min(seed_indices), "LOAD_INST should come before seed tokens" 427 427 + assert max(smtoken_indices) < min(iram_write_indices), "SM tokens should come before IRAM write tokens" 428 428 + assert max(iram_write_indices) < min(seed_indices), "IRAM write tokens should come before seed tokens" 292 429 293 430 def test_ac88_tokens_are_valid(self): 294 294 - """AC8.8: Generated tokens are valid and consumable by emulator. 431 431 + """AC7.3: Generated tokens in direct mode are valid and direct mode PEConfig/SMConfig still works. 295 432 296 433 Tests that: 297 434 - All tokens have required fields set 298 435 - Token structure matches emulator expectations 436 436 + - Direct mode produces valid PEConfig/SMConfig 299 437 - Tokens can be injected into an emulator System and execution completes 300 438 """ 301 439 from emu.network import build_topology ··· 326 464 result = generate_direct(graph) 327 465 tokens = generate_tokens(graph) 328 466 467 467 + # Verify direct mode result structure 468 468 + assert isinstance(result, AssemblyResult) 469 469 + assert len(result.pe_configs) == 1 470 470 + assert result.pe_configs[0].pe_id == 0 471 471 + assert len(result.sm_configs) == 1 472 472 + assert result.sm_configs[0].sm_id == 0 473 473 + 329 474 # Build emulator system from AssemblyResult configs 330 475 env = simpy.Environment() 331 476 emu_system = build_topology( ··· 335 480 fifo_capacity=16, 336 481 ) 337 482 338 338 - # Inject tokens into the system following the sequence: 339 339 - # 1. SM init tokens (paired with SM ID) 340 340 - # 2. ROUTE_SET and LOAD_INST CfgTokens 483 483 + # Inject tokens into the system following the new sequence: 484 484 + # 1. SM init tokens 485 485 + # 2. IRAM write tokens 341 486 # 3. Seed MonadTokens 342 487 for token in tokens: 343 488 if isinstance(token, SMToken): 344 489 emu_system.inject(token) 345 345 - elif isinstance(token, CfgToken): 490 490 + elif isinstance(token, IRAMWriteToken): 346 491 emu_system.inject(token) 347 492 elif isinstance(token, MonadToken): 348 493 emu_system.inject(token) ··· 357 502 assert isinstance(token.addr, int) 358 503 assert isinstance(token.op, MemOp) 359 504 assert token.op == MemOp.WRITE 360 360 - elif isinstance(token, RouteSetToken): 361 361 - assert isinstance(token.target, int) 362 362 - assert isinstance(token.op, CfgOp) 363 363 - assert isinstance(token.pe_routes, frozenset) 364 364 - assert isinstance(token.sm_routes, frozenset) 365 365 - elif isinstance(token, LoadInstToken): 505 505 + elif isinstance(token, IRAMWriteToken): 366 506 assert isinstance(token.target, int) 367 367 - assert isinstance(token.op, CfgOp) 507 507 + assert isinstance(token.offset, int) 508 508 + assert isinstance(token.ctx, int) 509 509 + assert isinstance(token.data, int) 368 510 assert isinstance(token.instructions, tuple) 369 511 elif isinstance(token, MonadToken): 370 512 assert isinstance(token.target, int) ··· 402 544 assert len(sm_tokens) == 0 403 545 404 546 def test_ac810_single_pe_self_route(self): 405 405 - """AC8.10: Single-PE program ROUTE_SET contains only self-route. 547 547 + """AC7.2: Single-PE program produces IRAM writes with no RouteSetToken. 406 548 407 549 Tests that: 408 408 - - allowed_pe_routes contains only the PE's own ID 409 409 - - Single-PE graph produces ROUTE_SET with pe_routes=[pe_id] 550 550 + - allowed_pe_routes contains only the PE's own ID (in direct mode) 551 551 + - Token stream has no RouteSetToken (route restrictions are not emitted) 552 552 + - Token stream has IRAMWriteToken instead 410 553 """ 411 554 node = IRNode( 412 555 name="&add", ··· 425 568 assert pe_config.allowed_pe_routes == {0} 426 569 427 570 tokens = generate_tokens(graph) 571 571 + # Verify no RouteSetToken (AC7.2) 428 572 route_set_tokens = [ 429 573 t for t in tokens 430 430 - if isinstance(t, RouteSetToken) 574 574 + if type(t).__name__ == 'RouteSetToken' # Check by class name to avoid import 431 575 ] 432 432 - assert len(route_set_tokens) == 1 433 433 - token = route_set_tokens[0] 434 434 - assert token.pe_routes == frozenset({0}) 576 576 + assert len(route_set_tokens) == 0, "RouteSetToken should not be in token stream (AC7.2)" 577 577 + 578 578 + # Verify IRAMWriteToken is present (AC7.1) 579 579 + iram_write_tokens = [ 580 580 + t for t in tokens 581 581 + if isinstance(t, IRAMWriteToken) 582 582 + ] 583 583 + assert len(iram_write_tokens) == 1, "Should have exactly one IRAMWriteToken per PE" 584 584 + token = iram_write_tokens[0] 585 585 + assert token.target == 0 435 586 436 587 def test_multiple_data_defs_same_sm(self): 437 588 """Multiple data_defs targeting same SM produce single SMConfig.

+6 -14

tests/test_dfgraph_categories.py

··· 8 8 - RoutingOp members (except CONST/FREE_CTX) map to ROUTING 9 9 - RoutingOp.CONST and FREE_CTX map to CONFIG 10 10 - MemOp members map to MEMORY 11 11 - - CfgOp members map to CONFIG 12 11 - Every OpcodeCategory has a colour in CATEGORY_COLOURS 13 12 """ 14 13 15 14 import pytest 16 15 17 17 - from cm_inst import ArithOp, CfgOp, LogicOp, MemOp, RoutingOp 16 16 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp 18 17 from asm.opcodes import MNEMONIC_TO_OP 19 18 from dfgraph.categories import categorise, OpcodeCategory, CATEGORY_COLOURS 20 19 ··· 103 102 MemOp.RD_INC, 104 103 MemOp.RD_DEC, 105 104 MemOp.CMP_SW, 105 105 + MemOp.EXEC, 106 106 + MemOp.RAW_READ, 107 107 + MemOp.SET_PAGE, 108 108 + MemOp.WRITE_IMM, 109 109 + MemOp.EXT, 106 110 ]) 107 111 def test_memory_ops_map_to_memory(self, op): 108 112 """All MemOp members map to MEMORY category.""" 109 113 assert categorise(op) == OpcodeCategory.MEMORY 110 110 - 111 111 - 112 112 - class TestCategoriseCfgOp: 113 113 - """Tests for CfgOp -> CONFIG category mapping.""" 114 114 - 115 115 - @pytest.mark.parametrize("op", [ 116 116 - CfgOp.LOAD_INST, 117 117 - CfgOp.ROUTE_SET, 118 118 - ]) 119 119 - def test_config_ops_map_to_config(self, op): 120 120 - """All CfgOp members map to CONFIG category.""" 121 121 - assert categorise(op) == OpcodeCategory.CONFIG 122 114 123 115 124 116 class TestCategoriseMnemonicToOp:

+3 -3

tests/test_e2e.py

··· 13 13 14 14 from asm import assemble, assemble_to_tokens 15 15 from emu import build_topology 16 16 - from tokens import CfgToken, MonadToken, SMToken 16 16 + from tokens import IRAMWriteToken, MonadToken, SMToken 17 17 18 18 19 19 def run_program_direct(source: str, until: int = 1000) -> dict: ··· 67 67 for token in tokens: 68 68 if isinstance(token, SMToken): 69 69 max_sm_id = max(max_sm_id, token.target) 70 70 - elif isinstance(token, CfgToken): 70 70 + elif isinstance(token, IRAMWriteToken): 71 71 max_pe_id = max(max_pe_id, token.target) 72 72 elif isinstance(token, MonadToken): 73 73 max_pe_id = max(max_pe_id, token.target) 74 74 75 75 - # Create minimal PE configs (empty IRAM - will be filled by LOAD_INST) 75 75 + # Create minimal PE configs (empty IRAM - will be filled by IRAMWriteToken) 76 76 from emu.types import PEConfig, SMConfig 77 77 pe_configs = [PEConfig(i, {}) for i in range(max_pe_id + 1)] 78 78 sm_configs = [SMConfig(i) for i in range(max_sm_id + 1)]

+382

tests/test_exec_bootstrap.py

··· 1 1 + """ 2 2 + Tests for EXEC opcode and IRAMWriteToken bootstrap functionality. 3 3 + 4 4 + Verifies acceptance criteria: 5 5 + - token-migration.AC2.1: IRAMWriteToken routes to target PE via network (isinstance CMToken) 6 6 + - token-migration.AC2.4: IRAMWriteToken with invalid target PE raises or is dropped 7 7 + - token-migration.AC5.1: EXEC reads Token objects from T0 and injects them 8 8 + - token-migration.AC5.2: Injected tokens are processed normally by target PEs/SMs 9 9 + - token-migration.AC5.3: EXEC can load a program (IRAM + seed tokens) and execute correctly 10 10 + - token-migration.AC5.4: EXEC on empty T0 region is a no-op 11 11 + """ 12 12 + 13 13 + import pytest 14 14 + import simpy 15 15 + 16 16 + from cm_inst import ALUInst, Addr, MemOp, Port, RoutingOp 17 17 + from emu import build_topology 18 18 + from emu.types import PEConfig, SMConfig 19 19 + from sm_mod import Presence 20 20 + from tokens import DyadToken, IRAMWriteToken, MonadToken, SMToken 21 21 + 22 22 + 23 23 + class TestAC2_1IRAMWriteTokenRouting: 24 24 + """AC2.1: IRAMWriteToken routes to target PE via network (isinstance CMToken).""" 25 25 + 26 26 + def test_iram_write_token_routes_to_target_pe_via_system_inject(self): 27 27 + """IRAMWriteToken is routed to correct target PE when injected via system.inject().""" 28 28 + env = simpy.Environment() 29 29 + 30 30 + sys = build_topology( 31 31 + env, 32 32 + [ 33 33 + PEConfig(pe_id=0, iram={}), 34 34 + PEConfig(pe_id=1, iram={}), 35 35 + PEConfig(pe_id=2, iram={}), 36 36 + ], 37 37 + [], 38 38 + ) 39 39 + 40 40 + # Create IRAMWriteToken targeting PE 2 41 41 + inst = ALUInst( 42 42 + op=RoutingOp.CONST, 43 43 + dest_l=Addr(a=0, port=Port.L, pe=0), 44 44 + dest_r=None, 45 45 + const=0x1234, 46 46 + ) 47 47 + iram_token = IRAMWriteToken( 48 48 + target=2, # Target PE 2 49 49 + offset=10, 50 50 + ctx=0, 51 51 + data=0, 52 52 + instructions=(inst,), 53 53 + ) 54 54 + 55 55 + # Inject via system.inject() which appends to PE 2's input_store.items directly 56 56 + sys.inject(iram_token) 57 57 + 58 58 + # Verify token arrived at PE 2's input_store (inject appends directly to items) 59 59 + assert len(sys.pes[2].input_store.items) > 0 60 60 + received = sys.pes[2].input_store.items[0] 61 61 + assert isinstance(received, IRAMWriteToken) 62 62 + assert received.target == 2 63 63 + assert received.offset == 10 64 64 + 65 65 + # PE 0 and PE 1 should not have received the token 66 66 + assert len(sys.pes[0].input_store.items) == 0 67 67 + assert len(sys.pes[1].input_store.items) == 0 68 68 + 69 69 + def test_iram_write_token_multiple_targets(self): 70 70 + """Multiple IRAMWriteTokens can route to different target PEs.""" 71 71 + env = simpy.Environment() 72 72 + 73 73 + sys = build_topology( 74 74 + env, 75 75 + [ 76 76 + PEConfig(pe_id=0, iram={}), 77 77 + PEConfig(pe_id=1, iram={}), 78 78 + ], 79 79 + [], 80 80 + ) 81 81 + 82 82 + # Create two IRAMWriteTokens targeting different PEs 83 83 + inst0 = ALUInst(op=RoutingOp.CONST, dest_l=None, dest_r=None, const=100) 84 84 + inst1 = ALUInst(op=RoutingOp.CONST, dest_l=None, dest_r=None, const=200) 85 85 + 86 86 + token0 = IRAMWriteToken(target=0, offset=0, ctx=0, data=0, instructions=(inst0,)) 87 87 + token1 = IRAMWriteToken(target=1, offset=5, ctx=0, data=0, instructions=(inst1,)) 88 88 + 89 89 + # Inject both tokens 90 90 + sys.inject(token0) 91 91 + sys.inject(token1) 92 92 + 93 93 + # Verify routing via direct items inspection 94 94 + assert len(sys.pes[0].input_store.items) == 1 95 95 + assert len(sys.pes[1].input_store.items) == 1 96 96 + assert sys.pes[0].input_store.items[0].offset == 0 97 97 + assert sys.pes[1].input_store.items[0].offset == 5 98 98 + 99 99 + 100 100 + class TestAC2_4IRAMWriteTokenInvalidTarget: 101 101 + """AC2.4: IRAMWriteToken with invalid target PE raises or is dropped.""" 102 102 + 103 103 + def test_iram_write_token_invalid_target_raises_key_error(self): 104 104 + """IRAMWriteToken with non-existent target PE raises KeyError via system.send().""" 105 105 + env = simpy.Environment() 106 106 + 107 107 + # Create topology with only PE 0 108 108 + sys = build_topology( 109 109 + env, 110 110 + [PEConfig(pe_id=0, iram={})], 111 111 + [], 112 112 + ) 113 113 + 114 114 + # Create IRAMWriteToken targeting non-existent PE 5 115 115 + inst = ALUInst(op=RoutingOp.CONST, dest_l=None, dest_r=None, const=0x5555) 116 116 + iram_token = IRAMWriteToken( 117 117 + target=5, # PE 5 does not exist 118 118 + offset=0, 119 119 + ctx=0, 120 120 + data=0, 121 121 + instructions=(inst,), 122 122 + ) 123 123 + 124 124 + # Attempting to send should raise KeyError 125 125 + def process_token(): 126 126 + yield from sys.send(iram_token) 127 127 + 128 128 + with pytest.raises(KeyError): 129 129 + env.process(process_token()) 130 130 + env.run(until=100) 131 131 + 132 132 + 133 133 + class TestAC5_1ExecInjectsTokens: 134 134 + """AC5.1: EXEC reads Token objects from T0 and injects them into the network via send().""" 135 135 + 136 136 + def test_exec_injects_single_token_to_pe(self): 137 137 + """EXEC at T0 address reads a DyadToken and injects it via send() which triggers SimPy events.""" 138 138 + env = simpy.Environment() 139 139 + 140 140 + sys = build_topology( 141 141 + env, 142 142 + [ 143 143 + PEConfig(pe_id=0, iram={}), 144 144 + PEConfig(pe_id=1, iram={}), 145 145 + ], 146 146 + [SMConfig(sm_id=0, cell_count=512, tier_boundary=256)], 147 147 + ) 148 148 + 149 149 + # Create a DyadToken to be injected by EXEC 150 150 + seed_token = DyadToken( 151 151 + target=1, 152 152 + offset=0, 153 153 + ctx=0, 154 154 + data=0x4567, 155 155 + port=Port.L, 156 156 + gen=0, 157 157 + wide=False, 158 158 + ) 159 159 + 160 160 + # Pre-populate T0 with the token 161 161 + sys.sms[0].t0_store.append(seed_token) 162 162 + sys.sms[0].system = sys 163 163 + 164 164 + def test_sequence(): 165 165 + # SM0 executes EXEC at T0 address 256 (t0_idx=0) 166 166 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 167 167 + yield sys.sms[0].input_store.put(exec_token) 168 168 + 169 169 + env.process(test_sequence()) 170 170 + env.run(until=100) 171 171 + 172 172 + # Verify token was injected via send() - it will be consumed by PE1's process 173 173 + # and stored in matching_store. The key is that send() triggers the get() event. 174 174 + assert sys.pes[1].matching_store[0][0].occupied is True 175 175 + assert sys.pes[1].matching_store[0][0].data == 0x4567 176 176 + assert sys.pes[1].matching_store[0][0].port == Port.L 177 177 + 178 178 + def test_exec_injects_multiple_tokens(self): 179 179 + """EXEC at T0 address reads multiple tokens and injects them in order via send(). 180 180 + 181 181 + Verifies that send() properly wakes up pending get() operations, allowing multiple 182 182 + tokens to be delivered in sequence through SimPy's event system. 183 183 + """ 184 184 + env = simpy.Environment() 185 185 + 186 186 + sys = build_topology( 187 187 + env, 188 188 + [ 189 189 + PEConfig(pe_id=0, iram={}), 190 190 + PEConfig(pe_id=1, iram={}), 191 191 + ], 192 192 + [ 193 193 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 194 194 + SMConfig(sm_id=1, cell_count=512, tier_boundary=256), 195 195 + ], 196 196 + ) 197 197 + 198 198 + # Create multiple SMTokens to be injected (write operations don't require IRAM) 199 199 + token1 = SMToken(target=1, addr=100, op=MemOp.WRITE, flags=None, data=0x1111, ret=None) 200 200 + token2 = SMToken(target=1, addr=101, op=MemOp.WRITE, flags=None, data=0x2222, ret=None) 201 201 + 202 202 + # Pre-populate T0 203 203 + sys.sms[0].t0_store.extend([token1, token2]) 204 204 + sys.sms[0].system = sys 205 205 + 206 206 + def test_sequence(): 207 207 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 208 208 + yield sys.sms[0].input_store.put(exec_token) 209 209 + 210 210 + env.process(test_sequence()) 211 211 + env.run(until=100) 212 212 + 213 213 + # Verify both tokens were injected via send() and processed by SM1 214 214 + # Both WRITE operations should have updated SM1's cells 215 215 + assert sys.sms[1].cells[100].pres == Presence.FULL 216 216 + assert sys.sms[1].cells[100].data_l == 0x1111 217 217 + assert sys.sms[1].cells[101].pres == Presence.FULL 218 218 + assert sys.sms[1].cells[101].data_l == 0x2222 219 219 + 220 220 + 221 221 + class TestAC5_2ExecTokensProcessedNormally: 222 222 + """AC5.2: Injected tokens are processed normally by target PEs/SMs.""" 223 223 + 224 224 + def test_injected_dyad_token_received_by_pe(self): 225 225 + """DyadToken injected by EXEC via send() is received and processed by target PE.""" 226 226 + env = simpy.Environment() 227 227 + 228 228 + sys = build_topology( 229 229 + env, 230 230 + [ 231 231 + PEConfig(pe_id=0, iram={}), 232 232 + PEConfig(pe_id=1, iram={}), 233 233 + ], 234 234 + [SMConfig(sm_id=0, cell_count=512, tier_boundary=256)], 235 235 + ) 236 236 + 237 237 + # Create dyad token to be injected by EXEC 238 238 + token_l = DyadToken(target=1, offset=0, ctx=0, data=0xABCD, port=Port.L, gen=0, wide=False) 239 239 + 240 240 + # Pre-populate T0 241 241 + sys.sms[0].t0_store.append(token_l) 242 242 + sys.sms[0].system = sys 243 243 + 244 244 + def test_sequence(): 245 245 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 246 246 + yield sys.sms[0].input_store.put(exec_token) 247 247 + 248 248 + env.process(test_sequence()) 249 249 + env.run(until=100) 250 250 + 251 251 + # Verify PE1 received and processed the token via matching_store 252 252 + assert sys.pes[1].matching_store[0][0].occupied is True 253 253 + assert sys.pes[1].matching_store[0][0].data == 0xABCD 254 254 + assert sys.pes[1].matching_store[0][0].port == Port.L 255 255 + 256 256 + 257 257 + class TestAC5_3BootstrapProgram: 258 258 + """AC5.3: EXEC can load a program (IRAM writes + seed tokens) from T0 that executes correctly.""" 259 259 + 260 260 + def test_bootstrap_with_iram_write_and_seed_tokens(self): 261 261 + """Full bootstrap: T0 contains IRAMWriteToken and seed tokens, EXEC loads and runs them. 262 262 + 263 263 + Tests the FULL SimPy execution chain: 264 264 + - Populate T0 with IRAMWriteToken + seed DyadToken pair 265 265 + - Send EXEC SMToken to SM 266 266 + - Run env.run() 267 267 + - Assert on pe.output_log containing the expected ALU result 268 268 + """ 269 269 + env = simpy.Environment() 270 270 + 271 271 + sys = build_topology( 272 272 + env, 273 273 + [ 274 274 + PEConfig(pe_id=0, iram={}), # PE0 starts empty, will be loaded by bootstrap 275 275 + PEConfig(pe_id=1, iram={}), # PE1 is output receiver 276 276 + ], 277 277 + [SMConfig(sm_id=0, cell_count=512, tier_boundary=256)], 278 278 + ) 279 279 + 280 280 + # Create instruction to be loaded: CONST(0xABCD) to PE1 281 281 + const_inst = ALUInst( 282 282 + op=RoutingOp.CONST, 283 283 + dest_l=Addr(a=0, port=Port.L, pe=1), 284 284 + dest_r=None, 285 285 + const=0xABCD, 286 286 + ) 287 287 + 288 288 + # Create IRAMWriteToken to load instruction at offset 0 289 289 + iram_write = IRAMWriteToken( 290 290 + target=0, 291 291 + offset=0, 292 292 + ctx=0, 293 293 + data=0, 294 294 + instructions=(const_inst,), 295 295 + ) 296 296 + 297 297 + # Create seed MonadToken to trigger the loaded instruction at PE0 298 298 + seed_token = MonadToken( 299 299 + target=0, 300 300 + offset=0, 301 301 + ctx=0, 302 302 + data=0, 303 303 + inline=False, 304 304 + ) 305 305 + 306 306 + # Pre-populate T0 with bootstrap sequence 307 307 + sys.sms[0].t0_store.append(iram_write) 308 308 + sys.sms[0].t0_store.append(seed_token) 309 309 + sys.sms[0].system = sys 310 310 + 311 311 + def test_sequence(): 312 312 + # Trigger EXEC to bootstrap 313 313 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 314 314 + yield sys.sms[0].input_store.put(exec_token) 315 315 + 316 316 + env.process(test_sequence()) 317 317 + env.run(until=200) 318 318 + 319 319 + # Verify FULL SimPy execution chain: 320 320 + # 1. PE0 should have received and processed IRAMWriteToken 321 321 + assert 0 in sys.pes[0].iram, "Instruction not loaded into IRAM by bootstrap" 322 322 + assert sys.pes[0].iram[0].op == RoutingOp.CONST 323 323 + assert sys.pes[0].iram[0].const == 0xABCD 324 324 + 325 325 + # 2. PE0 should have received and processed seed token, producing output 326 326 + assert len(sys.pes[0].output_log) > 0, \ 327 327 + "PE0 did not produce output; seed token may not have triggered IRAM execution" 328 328 + 329 329 + # 3. The output should be routed to PE1 with the CONST value 330 330 + output_to_pe1 = [t for t in sys.pes[0].output_log if t.target == 1] 331 331 + assert len(output_to_pe1) > 0, "PE0 did not route output to PE1" 332 332 + assert output_to_pe1[0].data == 0xABCD, f"Expected output data 0xABCD, got {output_to_pe1[0].data}" 333 333 + 334 334 + 335 335 + class TestAC5_4ExecOnEmptyT0: 336 336 + """AC5.4: EXEC on empty T0 region is a no-op.""" 337 337 + 338 338 + def test_exec_on_addr_beyond_t0_store_length_is_noop(self): 339 339 + """EXEC at address beyond t0_store length produces no output.""" 340 340 + env = simpy.Environment() 341 341 + 342 342 + sys = build_topology( 343 343 + env, 344 344 + [PEConfig(pe_id=0, iram={})], 345 345 + [SMConfig(sm_id=0, cell_count=512, tier_boundary=256)], 346 346 + ) 347 347 + 348 348 + # t0_store is empty, EXEC on T0 address beyond current length 349 349 + def test_sequence(): 350 350 + exec_token = SMToken(target=0, addr=300, op=MemOp.EXEC, flags=None, data=None, ret=None) 351 351 + yield sys.sms[0].input_store.put(exec_token) 352 352 + 353 353 + env.process(test_sequence()) 354 354 + env.run(until=100) 355 355 + 356 356 + # Verify no tokens were injected (output stores remain unchanged) 357 357 + assert len(sys.pes[0].input_store.items) == 0 358 358 + 359 359 + def test_exec_on_empty_t0_index(self): 360 360 + """EXEC at T0 index with no token is a no-op.""" 361 361 + env = simpy.Environment() 362 362 + 363 363 + sys = build_topology( 364 364 + env, 365 365 + [PEConfig(pe_id=0, iram={})], 366 366 + [SMConfig(sm_id=0, cell_count=512, tier_boundary=256)], 367 367 + ) 368 368 + 369 369 + # Pre-populate t0_store with one token 370 370 + sys.sms[0].t0_store.append(MonadToken(target=0, offset=0, ctx=0, data=100, inline=False)) 371 371 + 372 372 + # EXEC at index 5 which is beyond current t0_store length (1) 373 373 + def test_sequence(): 374 374 + exec_token = SMToken(target=0, addr=261, op=MemOp.EXEC, flags=None, data=None, ret=None) # t0_idx = 5 375 375 + yield sys.sms[0].input_store.put(exec_token) 376 376 + 377 377 + env.process(test_sequence()) 378 378 + env.run(until=100) 379 379 + 380 380 + # Verify only the pre-existing token remains in PE0 input_store (no injection happened) 381 381 + # The pre-existing token at index 0 should not be re-injected by EXEC at index 5 382 382 + assert len(sys.pes[0].input_store.items) == 0

+160 -37

tests/test_integration.py

··· 9 9 10 10 import simpy 11 11 12 12 - from cm_inst import Addr, ALUInst, ArithOp, CfgOp, MemOp, Port, RoutingOp, SMInst 12 12 + from cm_inst import Addr, ALUInst, ArithOp, MemOp, Port, RoutingOp, SMInst 13 13 from emu import PEConfig, SMConfig, build_topology 14 14 from sm_mod import Presence 15 15 - from tokens import CfgToken, CMToken, DyadToken, LoadInstToken, MonadToken, SMToken 15 15 + from tokens import CMToken, DyadToken, IRAMWriteToken, MonadToken, SMToken 16 16 17 17 18 18 class TestAC51IRAMInitialization: ··· 625 625 assert trigger_token.data == 0 626 626 627 627 628 628 - class TestTask5CfgTokenLoadInst: 629 629 - """Test CfgToken LOAD_INST handling in PE""" 628 628 + # Task 4: T0 Store Sharing and System Wiring Tests 629 629 + 630 630 + class TestAC4_4T0StoreShared: 631 631 + """AC4.4: T0 storage is shared — all SMs reference the same T0 store.""" 630 632 631 631 - def test_cfg_token_load_inst(self): 632 632 - """CfgToken with LOAD_INST dynamically loads instructions into PE IRAM.""" 633 633 + def test_t0_store_shared_across_sms(self): 634 634 + """All SMs share the same t0_store object.""" 633 635 env = simpy.Environment() 634 636 635 635 - # Build topology: PE0 starts with empty IRAM, PE1 is collector 637 637 + # Build topology with 2 SMs 636 638 sys = build_topology( 637 639 env, 640 640 + [], 638 641 [ 639 639 - PEConfig(pe_id=0, iram={}), # Empty IRAM initially 640 640 - PEConfig(pe_id=1, iram={}), # Collector 642 642 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 643 643 + SMConfig(sm_id=1, cell_count=512, tier_boundary=256), 641 644 ], 645 645 + ) 646 646 + 647 647 + # Verify both SMs reference the same t0_store object 648 648 + assert sys.sms[0].t0_store is sys.sms[1].t0_store 649 649 + 650 650 + def test_t0_write_visible_across_sms(self): 651 651 + """Data written to T0 by one SM is visible to another SM.""" 652 652 + env = simpy.Environment() 653 653 + 654 654 + sys = build_topology( 655 655 + env, 642 656 [], 657 657 + [ 658 658 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 659 659 + SMConfig(sm_id=1, cell_count=512, tier_boundary=256), 660 660 + ], 643 661 ) 644 662 645 645 - # Set up collector for output 646 646 - collector_store = simpy.Store(env, capacity=100) 647 647 - sys.pes[0].route_table[1] = collector_store 663 663 + # Create collector for SM1 read result 664 664 + collector = simpy.Store(env) 665 665 + sys.sms[1].route_table[0] = collector # SM1 can return to PE0 (dummy) 666 666 + 667 667 + def test_sequence(): 668 668 + # SM0 writes to T0 address 256 669 669 + write_token = SMToken(target=0, addr=256, op=MemOp.WRITE, flags=None, data=0xDEAD, ret=None) 670 670 + yield sys.sms[0].input_store.put(write_token) 671 671 + 672 672 + # SM1 reads from same T0 address 673 673 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 674 674 + read_token = SMToken(target=1, addr=256, op=MemOp.READ, flags=None, data=None, ret=ret_route) 675 675 + yield sys.sms[1].input_store.put(read_token) 676 676 + 677 677 + env.process(test_sequence()) 678 678 + env.run(until=100) 679 679 + 680 680 + # Verify SM1 read returned the value written by SM0 681 681 + assert len(collector.items) == 1 682 682 + assert collector.items[0].data == 0xDEAD 683 683 + 684 684 + 685 685 + class TestAC5_2ExecInjectsTokensProcessedNormally: 686 686 + """AC5.2: EXEC injects tokens that are processed normally by target PEs/SMs.""" 687 687 + 688 688 + def test_exec_injects_token_to_pe(self): 689 689 + """EXEC reads DyadToken from T0 and PE receives it normally via send().""" 690 690 + env = simpy.Environment() 691 691 + 692 692 + sys = build_topology( 693 693 + env, 694 694 + [ 695 695 + PEConfig(pe_id=0, iram={}), 696 696 + PEConfig(pe_id=1, iram={}), 697 697 + ], 698 698 + [ 699 699 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 700 700 + ], 701 701 + ) 702 702 + 703 703 + # Create DyadToken to be injected by EXEC 704 704 + seed_token = DyadToken( 705 705 + target=1, 706 706 + offset=0, 707 707 + ctx=0, 708 708 + data=0x1234, 709 709 + port=Port.L, 710 710 + gen=0, 711 711 + wide=False, 712 712 + ) 713 713 + 714 714 + # Pre-populate T0 with the token 715 715 + sys.sms[0].t0_store.append(seed_token) 716 716 + sys.sms[0].system = sys # Ensure system reference is set 717 717 + 718 718 + def test_sequence(): 719 719 + # SM0 executes EXEC at T0 address 256 (t0_idx=0, which has seed_token) 720 720 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 721 721 + yield sys.sms[0].input_store.put(exec_token) 722 722 + 723 723 + env.process(test_sequence()) 724 724 + env.run(until=100) 725 725 + 726 726 + # Verify token was injected via send() and received by PE1 in its matching_store 727 727 + # The send() call triggers PE1's _run() loop to wake up and consume the token 728 728 + assert sys.pes[1].matching_store[0][0].occupied is True 729 729 + assert sys.pes[1].matching_store[0][0].data == 0x1234 648 730 649 649 - # Create instruction to load: CONST(42) routing to PE1 731 731 + 732 732 + class TestAC5_3FullBootstrapSequence: 733 733 + """AC5.3: EXEC can load a program (IRAM writes + seed tokens) from T0 that executes correctly.""" 734 734 + 735 735 + def test_exec_bootstrap_with_iram_write_and_seed_token(self): 736 736 + """Full bootstrap sequence: T0 contains IRAMWriteToken and seed DyadToken, EXEC loads and runs them. 737 737 + 738 738 + Tests the FULL SimPy execution chain: 739 739 + - Populate T0 with IRAMWriteToken + seed DyadToken pair 740 740 + - Send EXEC SMToken to SM 741 741 + - Run env.run() 742 742 + - Assert on pe.output_log containing the expected ALU result 743 743 + """ 744 744 + env = simpy.Environment() 745 745 + 746 746 + sys = build_topology( 747 747 + env, 748 748 + [ 749 749 + PEConfig(pe_id=0, iram={}), # PE0 starts empty 750 750 + PEConfig(pe_id=1, iram={}), # PE1 is output receiver 751 751 + ], 752 752 + [ 753 753 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 754 754 + ], 755 755 + ) 756 756 + 757 757 + # Create instruction to be loaded: CONST(0x5555) routing to PE1 650 758 const_inst = ALUInst( 651 759 op=RoutingOp.CONST, 652 760 dest_l=Addr(a=0, port=Port.L, pe=1), 653 761 dest_r=None, 654 654 - const=42, 762 762 + const=0x5555, 655 763 ) 656 764 657 657 - # Create LoadInstToken to load instruction at offset 0 658 658 - cfg_token = LoadInstToken( 765 765 + # Create IRAMWriteToken to load instruction at offset 0 766 766 + iram_write = IRAMWriteToken( 659 767 target=0, 660 660 - addr=0, 661 661 - op=CfgOp.LOAD_INST, 768 768 + offset=0, 769 769 + ctx=0, 770 770 + data=0, 662 771 instructions=(const_inst,), 663 772 ) 664 773 665 665 - # Function to send CfgToken then seed token 666 666 - def injector(): 667 667 - # First, send CfgToken to load the instruction 668 668 - yield sys.pes[0].input_store.put(cfg_token) 669 669 - # Then inject a MonadToken seed to trigger the loaded instruction 670 670 - yield sys.pes[0].input_store.put( 671 671 - MonadToken(target=0, offset=0, ctx=0, data=0, inline=False) 672 672 - ) 774 774 + # Create seed DyadToken to trigger the loaded instruction 775 775 + seed_token = DyadToken( 776 776 + target=0, 777 777 + offset=0, 778 778 + ctx=0, 779 779 + data=0, 780 780 + port=Port.L, 781 781 + gen=0, 782 782 + wide=False, 783 783 + ) 673 784 674 674 - env.process(injector()) 785 785 + # Pre-populate T0 with bootstrap sequence 786 786 + sys.sms[0].t0_store.append(iram_write) 787 787 + sys.sms[0].t0_store.append(seed_token) 788 788 + sys.sms[0].system = sys 675 789 676 676 - # Run simulation 677 677 - env.run(until=1000) 790 790 + def test_sequence(): 791 791 + # Trigger EXEC to bootstrap 792 792 + exec_token = SMToken(target=0, addr=256, op=MemOp.EXEC, flags=None, data=None, ret=None) 793 793 + yield sys.sms[0].input_store.put(exec_token) 678 794 679 679 - # Verify: 680 680 - # 1. The instruction was loaded into IRAM 681 681 - assert 0 in sys.pes[0].iram 795 795 + env.process(test_sequence()) 796 796 + env.run(until=200) 797 797 + 798 798 + # Verify FULL SimPy execution chain: 799 799 + # 1. PE0 should have received and processed IRAMWriteToken 800 800 + assert 0 in sys.pes[0].iram, "Instruction not loaded into IRAM by bootstrap" 682 801 assert sys.pes[0].iram[0].op == RoutingOp.CONST 683 683 - assert sys.pes[0].iram[0].const == 42 802 802 + assert sys.pes[0].iram[0].const == 0x5555 684 803 685 685 - # 2. The loaded instruction executed and produced output 686 686 - assert len(collector_store.items) == 1 687 687 - result_token = collector_store.items[0] 688 688 - assert result_token.data == 42 804 804 + # 2. PE0 should have received and processed seed token, producing output 805 805 + assert len(sys.pes[0].output_log) > 0, \ 806 806 + "PE0 did not produce output; seed token may not have triggered IRAM execution" 807 807 + 808 808 + # 3. The output should be routed to PE1 with the CONST value 809 809 + output_to_pe1 = [t for t in sys.pes[0].output_log if t.target == 1] 810 810 + assert len(output_to_pe1) > 0, "PE0 did not route output to PE1" 811 811 + assert output_to_pe1[0].data == 0x5555, f"Expected output data 0x5555, got {output_to_pe1[0].data}"

+61

tests/test_migration_cleanup.py

··· 1 1 + """Regression tests: deleted types must not be re-introduced.""" 2 2 + 3 3 + import ast 4 4 + from pathlib import Path 5 5 + 6 6 + import cm_inst 7 7 + import tokens 8 8 + 9 9 + PROJECT_ROOT = Path(__file__).resolve().parent.parent 10 10 + DELETED_NAMES = frozenset({ 11 11 + "SysToken", "CfgToken", "IOToken", "LoadInstToken", "RouteSetToken", "CfgOp", 12 12 + }) 13 13 + 14 14 + 15 15 + class TestAC1_DeletedTypesAbsent: 16 16 + """AC1: Old token types and CfgOp must not exist.""" 17 17 + 18 18 + def test_no_sys_token(self): 19 19 + """SysToken must not exist in tokens module.""" 20 20 + assert not hasattr(tokens, 'SysToken') 21 21 + 22 22 + def test_no_cfg_token(self): 23 23 + """CfgToken must not exist in tokens module.""" 24 24 + assert not hasattr(tokens, 'CfgToken') 25 25 + 26 26 + def test_no_io_token(self): 27 27 + """IOToken must not exist in tokens module.""" 28 28 + assert not hasattr(tokens, 'IOToken') 29 29 + 30 30 + def test_no_load_inst_token(self): 31 31 + """LoadInstToken must not exist in tokens module.""" 32 32 + assert not hasattr(tokens, 'LoadInstToken') 33 33 + 34 34 + def test_no_route_set_token(self): 35 35 + """RouteSetToken must not exist in tokens module.""" 36 36 + assert not hasattr(tokens, 'RouteSetToken') 37 37 + 38 38 + def test_no_cfg_op(self): 39 39 + """CfgOp must not exist in cm_inst module.""" 40 40 + assert not hasattr(cm_inst, 'CfgOp') 41 41 + 42 42 + 43 43 + class TestAC1_3NoStaleImports: 44 44 + """AC1.3: No module in the codebase imports any deleted type.""" 45 45 + 46 46 + def test_no_production_code_imports_deleted_types(self): 47 47 + """Scan all production .py files for import statements referencing deleted names.""" 48 48 + violations = [] 49 49 + for py_file in PROJECT_ROOT.rglob("*.py"): 50 50 + if "test" in py_file.name or py_file.is_relative_to(PROJECT_ROOT / "tests"): 51 51 + continue 52 52 + try: 53 53 + tree = ast.parse(py_file.read_text(), filename=str(py_file)) 54 54 + except SyntaxError: 55 55 + continue 56 56 + for node in ast.walk(tree): 57 57 + if isinstance(node, ast.ImportFrom): 58 58 + for alias in node.names: 59 59 + if alias.name in DELETED_NAMES: 60 60 + violations.append(f"{py_file}:{node.lineno} imports {alias.name}") 61 61 + assert violations == [], f"Stale imports found:\n" + "\n".join(violations)

+28 -18

tests/test_opcodes.py

··· 1 1 """Tests for asm.opcodes module — mnemonic mapping and arity classification.""" 2 2 3 3 import pytest 4 4 - from cm_inst import ArithOp, LogicOp, RoutingOp 5 5 - from tokens import MemOp, CfgOp 4 4 + from cm_inst import ArithOp, LogicOp, MemOp, RoutingOp 6 5 from asm.opcodes import ( 7 6 MNEMONIC_TO_OP, 8 7 OP_TO_MNEMONIC, ··· 71 70 assert MNEMONIC_TO_OP["rd_dec"] == MemOp.RD_DEC 72 71 assert MNEMONIC_TO_OP["cmp_sw"] == MemOp.CMP_SW 73 72 74 74 - def test_configuration_opcodes(self): 75 75 - """or1-asm.AC1.3: Configuration opcodes map correctly.""" 76 76 - assert MNEMONIC_TO_OP["load_inst"] == CfgOp.LOAD_INST 77 77 - assert MNEMONIC_TO_OP["route_set"] == CfgOp.ROUTE_SET 78 78 - 79 73 def test_mnemonic_to_op_count(self): 80 74 """Verify all expected mnemonics are present.""" 81 81 - # Total: 7 arithmetic + 9 logic + 4 branch + 4 switch + 6 control + 8 memory + 2 config = 40 82 82 - expected_count = 40 75 75 + # Total: 7 arithmetic + 9 logic + 4 branch + 4 switch + 6 control + 13 memory = 43 76 76 + # (CfgOp removed: -2 load_inst, route_set; new MemOps added: +5 exec, raw_read, set_page, write_imm, ext) 77 77 + expected_count = 43 83 78 assert len(MNEMONIC_TO_OP) == expected_count 84 79 85 80 ··· 101 96 "gate", "sel", "merge", "pass", "const", "free_ctx", 102 97 # Memory 103 98 "read", "write", "clear", "alloc", "free", "rd_inc", "rd_dec", "cmp_sw", 104 104 - # Configuration 105 105 - "load_inst", "route_set", 99 99 + "exec", "raw_read", "set_page", "write_imm", "ext", 106 100 ] 107 101 ) 108 102 def test_round_trip(self, mnemonic): ··· 126 120 "gate", "sel", "merge", "pass", "const", "free_ctx", 127 121 # Memory 128 122 "read", "write", "clear", "alloc", "free", "rd_inc", "rd_dec", "cmp_sw", 129 129 - # Configuration 130 130 - "load_inst", "route_set", 123 123 + "exec", "raw_read", "set_page", "write_imm", "ext", 131 124 ] 132 125 ) 133 126 def test_round_trip_via_dict(self, mnemonic): ··· 146 139 """Verify MONADIC_OPS contains exactly the right opcodes.""" 147 140 148 141 def test_monadic_ops_size(self): 149 149 - """MONADIC_OPS should have exactly 15 opcodes (collision-free). 142 142 + """MONADIC_OPS should have exactly 20 opcodes (collision-free). 150 143 151 151 - Without collision-free implementation, this would be 12 due to IntEnum 144 144 + Without collision-free implementation, this would be lower due to IntEnum 152 145 collisions (e.g., ArithOp.INC colliding with some MemOp value). 146 146 + Count: 5 arithmetic + 1 logic + 3 routing + 5 old memory + 5 new memory = 20 153 147 """ 154 154 - assert len(MONADIC_OPS) == 15 148 148 + assert len(MONADIC_OPS) == 20 155 149 156 150 def test_monadic_opcodes_present(self): 157 151 """All known monadic opcodes should be in the set.""" ··· 224 218 RoutingOp.PASS, RoutingOp.CONST, RoutingOp.FREE_CTX, 225 219 MemOp.READ, MemOp.ALLOC, MemOp.FREE, MemOp.CLEAR, 226 220 MemOp.RD_INC, MemOp.RD_DEC, 227 227 - CfgOp.LOAD_INST, CfgOp.ROUTE_SET, 221 221 + MemOp.EXEC, MemOp.RAW_READ, MemOp.SET_PAGE, MemOp.WRITE_IMM, MemOp.EXT, 228 222 ] 229 223 ) 230 224 def test_always_monadic_opcodes(self, op): ··· 276 270 RoutingOp.PASS, RoutingOp.CONST, RoutingOp.FREE_CTX, 277 271 MemOp.READ, MemOp.ALLOC, MemOp.FREE, MemOp.CLEAR, 278 272 MemOp.RD_INC, MemOp.RD_DEC, 279 279 - CfgOp.LOAD_INST, CfgOp.ROUTE_SET, 273 273 + MemOp.EXEC, MemOp.RAW_READ, MemOp.SET_PAGE, MemOp.WRITE_IMM, MemOp.EXT, 280 274 ] 281 275 ) 282 276 def test_always_monadic_is_not_dyadic(self, op): ··· 307 301 def test_dyadic_opcodes_is_dyadic(self, op): 308 302 """Dyadic opcodes should be dyadic.""" 309 303 assert is_dyadic(op) is True 304 304 + 305 305 + 306 306 + class TestAC3_2TierGrouping: 307 307 + """AC3.2: MemOp integer values preserve tier 1/tier 2 encoding boundaries.""" 308 308 + 309 309 + def test_tier1_ops_have_values_0_through_5(self): 310 310 + """Tier 1 memory operations must have values in range [0, 5].""" 311 311 + tier1 = [MemOp.READ, MemOp.WRITE, MemOp.EXEC, MemOp.ALLOC, MemOp.FREE, MemOp.EXT] 312 312 + for op in tier1: 313 313 + assert op.value <= 5, f"{op.name} has value {op.value}, expected <= 5" 314 314 + 315 315 + def test_tier2_ops_have_values_6_through_12(self): 316 316 + """Tier 2 memory operations must have values in range [6, 12].""" 317 317 + tier2 = [MemOp.CLEAR, MemOp.RD_INC, MemOp.RD_DEC, MemOp.CMP_SW, MemOp.RAW_READ, MemOp.SET_PAGE, MemOp.WRITE_IMM] 318 318 + for op in tier2: 319 319 + assert op.value >= 6 and op.value <= 12, f"{op.name} has value {op.value}, expected in range [6, 12]" 310 320 311 321 312 322 class TestFreeDisambiguation:

+103 -238

tests/test_pe.py

··· 20 20 from cm_inst import ALUInst, Addr, ArithOp, MemOp, Port, RoutingOp, SMInst 21 21 from emu.pe import ProcessingElement 22 22 from tests.conftest import dyad_token 23 23 - from tokens import DyadToken, MonadToken 23 23 + from tokens import DyadToken, IRAMWriteToken, MonadToken 24 24 25 25 26 26 def _inject_token(pe, token): ··· 457 457 assert len(output_store.items) == 0 458 458 459 459 460 460 - class TestRouteSet: 461 461 - """Test ROUTE_SET CfgToken handler (AC7.1–AC7.5). 460 460 + class TestIRAMWriteToken: 461 461 + """Test IRAMWriteToken handler for dynamic IRAM updates. 462 462 463 463 Verifies: 464 464 - - AC7.1: ROUTE_SET CfgToken accepted without warning 465 465 - - AC7.2: PE can route to allowed PE IDs 466 466 - - AC7.3: PE can route to allowed SM IDs 467 467 - - AC7.4: Routing to unlisted PE ID raises KeyError 468 468 - - AC7.5: Routing to unlisted SM ID raises KeyError 464 464 + - AC2.2: PE receives IRAMWriteToken and writes instructions to IRAM at offset 465 465 + - AC2.3: PE executes instructions loaded via IRAMWriteToken correctly 466 466 + - AC2.4: IRAMWriteToken with invalid target PE raises or is dropped 469 467 """ 470 468 471 471 - def test_ac71_route_set_accepted_no_warning(self): 472 472 - """AC7.1: ROUTE_SET CfgToken accepted without warning.""" 473 473 - from cm_inst import CfgOp 474 474 - from tokens import RouteSetToken 475 475 - 469 469 + def test_ac22_iram_write_token_loads_instructions(self): 470 470 + """AC2.2: IRAMWriteToken writes instructions to IRAM at specified offset.""" 476 471 env = simpy.Environment() 477 472 pe = ProcessingElement(env, 0, {}) 478 473 479 479 - # Set up full-mesh routes (3 PEs, 1 SM) 480 480 - pe_store_0 = simpy.Store(env) 481 481 - pe_store_1 = simpy.Store(env) 482 482 - pe_store_2 = simpy.Store(env) 483 483 - sm_store_0 = simpy.Store(env) 484 484 - 485 485 - pe.route_table[0] = pe_store_0 486 486 - pe.route_table[1] = pe_store_1 487 487 - pe.route_table[2] = pe_store_2 488 488 - pe.sm_routes[0] = sm_store_0 489 489 - 490 490 - # Create ROUTE_SET: allow PE 0 and PE 2, allow SM 0 491 491 - route_set_token = RouteSetToken( 492 492 - target=0, 493 493 - addr=None, 494 494 - op=CfgOp.ROUTE_SET, 495 495 - pe_routes=frozenset({0, 2}), 496 496 - sm_routes=frozenset({0}), 474 474 + # Create IRAMWriteToken to load a PASS instruction at offset 5 475 475 + pass_inst = ALUInst( 476 476 + op=RoutingOp.PASS, 477 477 + dest_l=Addr(a=0, port=Port.L, pe=1), 478 478 + dest_r=None, 479 479 + const=None, 497 480 ) 498 498 - 499 499 - # Call _handle_cfg 500 500 - pe._handle_cfg(route_set_token) 501 501 - 502 502 - # No exception, no warning (check with logging) 503 503 - assert len(pe.route_table) == 2 504 504 - assert 0 in pe.route_table 505 505 - assert 2 in pe.route_table 506 506 - assert len(pe.sm_routes) == 1 507 507 - assert 0 in pe.sm_routes 508 508 - 509 509 - def test_ac72_route_to_allowed_pe_succeeds(self): 510 510 - """AC7.2: After ROUTE_SET, PE can route to allowed PE ID.""" 511 511 - from cm_inst import CfgOp 512 512 - from tokens import RouteSetToken 513 513 - 514 514 - env = simpy.Environment() 515 515 - 516 516 - # PE 0 with PASS instruction 517 517 - iram = { 518 518 - 0: ALUInst( 519 519 - op=RoutingOp.PASS, 520 520 - dest_l=Addr(a=0, port=Port.L, pe=0), 521 521 - dest_r=None, 522 522 - const=None, 523 523 - ) 524 524 - } 525 525 - pe = ProcessingElement(env, 0, iram) 526 526 - 527 527 - # Full-mesh route_table 528 528 - pe_store_0 = simpy.Store(env) 529 529 - pe_store_1 = simpy.Store(env) 530 530 - pe_store_2 = simpy.Store(env) 531 531 - pe.route_table[0] = pe_store_0 532 532 - pe.route_table[1] = pe_store_1 533 533 - pe.route_table[2] = pe_store_2 534 534 - 535 535 - # ROUTE_SET: allow PE 0 and PE 2 536 536 - route_set_token = RouteSetToken( 481 481 + write_token = IRAMWriteToken( 537 482 target=0, 538 538 - addr=None, 539 539 - op=CfgOp.ROUTE_SET, 540 540 - pe_routes=frozenset({0, 2}), 541 541 - sm_routes=frozenset(), 483 483 + offset=5, 484 484 + ctx=0, 485 485 + data=0, 486 486 + instructions=(pass_inst,), 542 487 ) 543 543 - pe._handle_cfg(route_set_token) 544 488 545 545 - # Inject token and route to PE 0 (allowed) 546 546 - token = MonadToken(target=0, offset=0, ctx=0, data=42, inline=False) 547 547 - 489 489 + # Inject the IRAMWriteToken 548 490 def inject(): 549 549 - yield pe.input_store.put(route_set_token) # Apply restriction first 491 491 + yield pe.input_store.put(write_token) 550 492 yield env.timeout(10) 551 551 - yield pe.input_store.put(token) 552 493 553 494 env.process(inject()) 554 554 - env.run(until=200) 555 555 - 556 556 - # Token should have been routed to PE 0 with correct data and target 557 557 - assert len(pe_store_0.items) > 0 558 558 - routed_token = pe_store_0.items[0] 559 559 - assert isinstance(routed_token, DyadToken) 560 560 - assert routed_token.data == 42 # Token data preserved 561 561 - assert routed_token.target == 0 # Routed to PE 0 495 495 + env.run(until=100) 562 496 563 563 - def test_ac73_route_to_allowed_sm_succeeds(self): 564 564 - """AC7.3: After ROUTE_SET, PE can route to allowed SM ID.""" 565 565 - from cm_inst import CfgOp 566 566 - from tokens import RouteSetToken 497 497 + # Verify IRAM was updated at offset 5 498 498 + assert 5 in pe.iram 499 499 + assert pe.iram[5] == pass_inst 567 500 501 501 + def test_ac23_loaded_instructions_execute_correctly(self): 502 502 + """AC2.3: After loading via IRAMWriteToken, PE executes the loaded instruction.""" 568 503 env = simpy.Environment() 569 569 - 570 570 - # PE 0 with READ to SM 0 571 571 - iram = { 572 572 - 0: SMInst( 573 573 - op=MemOp.READ, 574 574 - sm_id=0, 575 575 - const=10, 576 576 - ret=Addr(a=0, port=Port.L, pe=0), 577 577 - ) 578 578 - } 579 579 - pe = ProcessingElement(env, 0, iram) 504 504 + pe = ProcessingElement(env, 0, {}) 580 505 581 581 - # Full-mesh sm_routes 582 582 - sm_store_0 = simpy.Store(env) 583 583 - sm_store_1 = simpy.Store(env) 584 584 - pe.sm_routes[0] = sm_store_0 585 585 - pe.sm_routes[1] = sm_store_1 506 506 + # Set up output store 507 507 + output_store = simpy.Store(env, capacity=10) 508 508 + pe.route_table[1] = output_store 586 509 587 587 - # ROUTE_SET: allow SM 0 588 588 - route_set_token = RouteSetToken( 510 510 + # Create IRAMWriteToken to load an ADD instruction at offset 3 511 511 + add_inst = ALUInst( 512 512 + op=ArithOp.ADD, 513 513 + dest_l=Addr(a=0, port=Port.L, pe=1), 514 514 + dest_r=None, 515 515 + const=None, 516 516 + ) 517 517 + write_token = IRAMWriteToken( 589 518 target=0, 590 590 - addr=None, 591 591 - op=CfgOp.ROUTE_SET, 592 592 - pe_routes=frozenset(), 593 593 - sm_routes=frozenset({0}), 519 519 + offset=3, 520 520 + ctx=0, 521 521 + data=0, 522 522 + instructions=(add_inst,), 594 523 ) 595 595 - pe._handle_cfg(route_set_token) 596 524 597 597 - # Inject token to trigger SM READ 598 598 - token = MonadToken(target=0, offset=0, ctx=0, data=5, inline=False) 599 599 - 525 525 + # Inject the IRAMWriteToken, then dyadic tokens to trigger ADD 600 526 def inject(): 601 601 - yield pe.input_store.put(route_set_token) 527 527 + yield pe.input_store.put(write_token) 602 528 yield env.timeout(10) 603 603 - yield pe.input_store.put(token) 529 529 + # Now inject dyadic tokens for the loaded ADD instruction 530 530 + token_l = DyadToken(target=0, offset=3, ctx=0, data=0x10, port=Port.L, gen=0, wide=False) 531 531 + token_r = DyadToken(target=0, offset=3, ctx=0, data=0x20, port=Port.R, gen=0, wide=False) 532 532 + yield pe.input_store.put(token_l) 533 533 + yield pe.input_store.put(token_r) 604 534 605 535 env.process(inject()) 606 536 env.run(until=200) 607 537 608 608 - # SM token should have been routed to SM 0 with correct target 609 609 - assert len(sm_store_0.items) > 0 610 610 - routed_token = sm_store_0.items[0] 611 611 - from tokens import SMToken 612 612 - assert isinstance(routed_token, SMToken) 613 613 - assert routed_token.addr == 10 # Cell address from const 538 538 + # Verify ADD result: 0x10 + 0x20 = 0x30 539 539 + assert len(output_store.items) == 1 540 540 + result = output_store.items[0] 541 541 + assert result.data == 0x30 614 542 615 615 - def test_ac74_route_to_unlisted_pe_raises_keyerror(self): 616 616 - """AC7.4: After ROUTE_SET, routing to unlisted PE ID raises KeyError.""" 617 617 - from cm_inst import CfgOp 618 618 - from tokens import RouteSetToken 543 543 + def test_ac24_write_to_invalid_target_pe_raises_or_drops(self): 544 544 + """AC2.4: IRAMWriteToken with non-existent target PE raises TypeError or is dropped.""" 545 545 + from emu.network import System 619 546 620 547 env = simpy.Environment() 548 548 + # Create topology with only PE 0 549 549 + pe0 = ProcessingElement(env, 0, {}) 550 550 + system = System(env, {0: pe0}, {}) 621 551 622 622 - # PE 0 with PASS instruction routing to PE 1 623 623 - iram = { 624 624 - 0: ALUInst( 625 625 - op=RoutingOp.PASS, 626 626 - dest_l=Addr(a=0, port=Port.L, pe=1), 627 627 - dest_r=None, 628 628 - const=None, 629 629 - ) 630 630 - } 631 631 - pe = ProcessingElement(env, 0, iram) 632 632 - 633 633 - # Full-mesh route_table (PEs 0, 1, 2) 634 634 - pe.route_table[0] = simpy.Store(env) 635 635 - pe.route_table[1] = simpy.Store(env) 636 636 - pe.route_table[2] = simpy.Store(env) 637 637 - 638 638 - # ROUTE_SET: allow only PE 0 and 2 (exclude PE 1) 639 639 - route_set_token = RouteSetToken( 640 640 - target=0, 641 641 - addr=None, 642 642 - op=CfgOp.ROUTE_SET, 643 643 - pe_routes=frozenset({0, 2}), 644 644 - sm_routes=frozenset(), 552 552 + # Create IRAMWriteToken targeting non-existent PE 5 553 553 + pass_inst = ALUInst( 554 554 + op=RoutingOp.PASS, 555 555 + dest_l=Addr(a=0, port=Port.L, pe=1), 556 556 + dest_r=None, 557 557 + const=None, 645 558 ) 646 646 - pe._handle_cfg(route_set_token) 647 647 - 648 648 - # Inject token targeting PE 1 (not allowed) 649 649 - token = MonadToken(target=0, offset=0, ctx=0, data=42, inline=False) 650 650 - 651 651 - def inject(): 652 652 - yield pe.input_store.put(route_set_token) 653 653 - yield env.timeout(10) 654 654 - yield pe.input_store.put(token) 655 655 - 656 656 - env.process(inject()) 657 657 - 658 658 - # Should raise KeyError when trying to access route_table[1] 659 659 - with pytest.raises(KeyError): 660 660 - env.run(until=200) 661 661 - 662 662 - def test_ac75_route_to_unlisted_sm_raises_keyerror(self): 663 663 - """AC7.5: After ROUTE_SET, routing to unlisted SM ID raises KeyError.""" 664 664 - from cm_inst import CfgOp 665 665 - from tokens import RouteSetToken 666 666 - 667 667 - env = simpy.Environment() 668 668 - 669 669 - # PE 0 with READ to SM 1 670 670 - iram = { 671 671 - 0: SMInst( 672 672 - op=MemOp.READ, 673 673 - sm_id=1, 674 674 - const=10, 675 675 - ret=Addr(a=0, port=Port.L, pe=0), 676 676 - ) 677 677 - } 678 678 - pe = ProcessingElement(env, 0, iram) 679 679 - 680 680 - # Full-mesh sm_routes (SMs 0, 1) 681 681 - pe.sm_routes[0] = simpy.Store(env) 682 682 - pe.sm_routes[1] = simpy.Store(env) 683 683 - 684 684 - # ROUTE_SET: allow only SM 0 (exclude SM 1) 685 685 - route_set_token = RouteSetToken( 686 686 - target=0, 687 687 - addr=None, 688 688 - op=CfgOp.ROUTE_SET, 689 689 - pe_routes=frozenset(), 690 690 - sm_routes=frozenset({0}), 559 559 + write_token = IRAMWriteToken( 560 560 + target=5, # PE 5 does not exist 561 561 + offset=0, 562 562 + ctx=0, 563 563 + data=0, 564 564 + instructions=(pass_inst,), 691 565 ) 692 692 - pe._handle_cfg(route_set_token) 693 566 694 694 - # Inject token targeting SM 1 (not allowed) 695 695 - token = MonadToken(target=0, offset=0, ctx=0, data=5, inline=False) 696 696 - 697 697 - def inject(): 698 698 - yield pe.input_store.put(route_set_token) 699 699 - yield env.timeout(10) 700 700 - yield pe.input_store.put(token) 701 701 - 702 702 - env.process(inject()) 703 703 - 704 704 - # Should raise KeyError when trying to access sm_routes[1] 567 567 + # Attempting to send to invalid PE should raise TypeError 705 568 with pytest.raises(KeyError): 706 706 - env.run(until=200) 569 569 + def inject(): 570 570 + yield from system.send(write_token) 571 571 + env.process(inject()) 572 572 + env.run(until=100) 707 573 708 574 709 575 class TestMatchingStoreCleared: ··· 929 795 class TestBoundaryEdgeCases: 930 796 """Test boundary and edge cases for PE operations.""" 931 797 932 932 - def test_load_inst_at_non_zero_base_address(self): 933 933 - """LOAD_INST CfgToken can load instructions at non-zero base address.""" 934 934 - from cm_inst import CfgOp 935 935 - from tokens import LoadInstToken 936 936 - 798 798 + def test_iram_write_multiple_instructions(self): 799 799 + """IRAMWriteToken can load multiple instructions sequentially.""" 937 800 env = simpy.Environment() 938 801 pe = ProcessingElement(env, 0, {}) 939 802 940 940 - # Set up output store to collect results 941 941 - output_store = simpy.Store(env, capacity=10) 942 942 - pe.route_table[1] = output_store 803 803 + # Create two instructions to load 804 804 + inst1 = ALUInst( 805 805 + op=RoutingOp.PASS, 806 806 + dest_l=Addr(a=0, port=Port.L, pe=1), 807 807 + dest_r=None, 808 808 + const=None, 809 809 + ) 810 810 + inst2 = ALUInst( 811 811 + op=ArithOp.ADD, 812 812 + dest_l=Addr(a=1, port=Port.L, pe=1), 813 813 + dest_r=None, 814 814 + const=None, 815 815 + ) 943 816 944 944 - # Create LOAD_INST to load PASS instruction at offset 5 945 945 - load_inst_token = LoadInstToken( 817 817 + write_token = IRAMWriteToken( 946 818 target=0, 947 947 - addr=5, # Non-zero base address 948 948 - op=CfgOp.LOAD_INST, 949 949 - instructions=(ALUInst( 950 950 - op=RoutingOp.PASS, 951 951 - dest_l=Addr(a=0, port=Port.L, pe=1), 952 952 - dest_r=None, 953 953 - const=None, 954 954 - ),), 819 819 + offset=10, 820 820 + ctx=0, 821 821 + data=0, 822 822 + instructions=(inst1, inst2), 955 823 ) 956 824 957 825 def inject(): 958 958 - # Inject LOAD_INST config token 959 959 - yield pe.input_store.put(load_inst_token) 826 826 + yield pe.input_store.put(write_token) 960 827 yield env.timeout(10) 961 961 - # Now inject MonadToken targeting offset 5 962 962 - seed_token = MonadToken(target=0, offset=5, ctx=0, data=0x1234, inline=False) 963 963 - yield pe.input_store.put(seed_token) 964 828 965 829 env.process(inject()) 966 830 env.run(until=100) 967 831 968 968 - # Verify instruction was loaded and executed 969 969 - assert len(output_store.items) == 1 970 970 - result = output_store.items[0] 971 971 - assert result.data == 0x1234 # PASS returns left operand 832 832 + # Verify both instructions were loaded at consecutive offsets 833 833 + assert 10 in pe.iram 834 834 + assert 11 in pe.iram 835 835 + assert pe.iram[10] == inst1 836 836 + assert pe.iram[11] == inst2

+4

tests/test_sm.py

··· 866 866 # Verify no result token and cell still EMPTY 867 867 assert len(collector.items) == 0 868 868 assert sm.cells[210].pres == Presence.EMPTY 869 869 + 870 870 + 871 871 + # Task 3: T0/T1 Tier Split and EXEC Opcode Tests 872 872 +

+474

tests/test_sm_tiers.py

··· 1 1 + """ 2 2 + Tests for SM T0/T1 memory tier split. 3 3 + 4 4 + Verifies acceptance criteria: 5 5 + - token-migration.AC4.1: SM operations on T1 (< tier_boundary) use I-structure semantics 6 6 + - token-migration.AC4.2: SM WRITE to T0 (>= tier_boundary) stores data without presence checking 7 7 + - token-migration.AC4.3: SM READ on T0 address returns immediately (no deferral) 8 8 + - token-migration.AC4.4: T0 storage is shared across all SMs 9 9 + - token-migration.AC4.5: I-structure ops (CLEAR, ALLOC, FREE, atomics) on T0 produce error/warning 10 10 + - token-migration.AC4.6: Tier boundary is configurable via SMConfig; default is 256 11 11 + - token-migration.AC6.2: Existing I-structure behaviour unchanged (is_wide=False path) 12 12 + """ 13 13 + 14 14 + import simpy 15 15 + 16 16 + from cm_inst import MemOp 17 17 + from emu import build_topology 18 18 + from emu.sm import StructureMemory 19 19 + from emu.types import SMConfig 20 20 + from sm_mod import Presence 21 21 + from tokens import CMToken, SMToken 22 22 + 23 23 + 24 24 + def inject_token(env: simpy.Environment, store: simpy.Store, token): 25 25 + """Helper to inject token into store via a process.""" 26 26 + def _injector(): 27 27 + yield store.put(token) 28 28 + 29 29 + env.process(_injector()) 30 30 + 31 31 + 32 32 + class TestAC4_1T1IStructureSemantics: 33 33 + """AC4.1: SM READ on T1 (< tier_boundary) uses I-structure semantics.""" 34 34 + 35 35 + def test_t1_read_on_full_returns_immediately(self): 36 36 + """READ on T1 full cell returns data immediately.""" 37 37 + env = simpy.Environment() 38 38 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 39 39 + 40 40 + # Pre-populate T1 cell (addr < 256) to FULL 41 41 + t1_addr = 100 42 42 + sm.cells[t1_addr].pres = Presence.FULL 43 43 + sm.cells[t1_addr].data_l = 0x1111 44 44 + 45 45 + collector = simpy.Store(env) 46 46 + sm.route_table[0] = collector 47 47 + 48 48 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 49 49 + read_token = SMToken(target=0, addr=t1_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 50 50 + inject_token(env, sm.input_store, read_token) 51 51 + 52 52 + env.run(until=100) 53 53 + 54 54 + assert len(collector.items) == 1 55 55 + assert collector.items[0].data == 0x1111 56 56 + 57 57 + def test_t1_read_on_empty_defers_and_sets_waiting(self): 58 58 + """READ on T1 empty cell sets WAITING and stashes return route.""" 59 59 + env = simpy.Environment() 60 60 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 61 61 + 62 62 + t1_addr = 150 63 63 + assert sm.cells[t1_addr].pres == Presence.EMPTY 64 64 + 65 65 + collector = simpy.Store(env) 66 66 + sm.route_table[0] = collector 67 67 + 68 68 + ret_route = CMToken(target=0, offset=5, ctx=1, data=0) 69 69 + read_token = SMToken(target=0, addr=t1_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 70 70 + inject_token(env, sm.input_store, read_token) 71 71 + 72 72 + env.run(until=100) 73 73 + 74 74 + # Verify deferred read is set 75 75 + assert sm.deferred_read is not None 76 76 + assert sm.deferred_read.cell_addr == t1_addr 77 77 + assert sm.cells[t1_addr].pres == Presence.WAITING 78 78 + # No result token yet 79 79 + assert len(collector.items) == 0 80 80 + 81 81 + def test_t1_deferred_read_satisfied_by_write(self): 82 82 + """WRITE on T1 WAITING cell satisfies deferred read.""" 83 83 + env = simpy.Environment() 84 84 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 85 85 + 86 86 + t1_addr = 120 87 87 + collector = simpy.Store(env) 88 88 + sm.route_table[0] = collector 89 89 + 90 90 + def test_sequence(): 91 91 + # First: READ on empty T1 cell to defer 92 92 + ret_route = CMToken(target=0, offset=10, ctx=0, data=0) 93 93 + read_token = SMToken(target=0, addr=t1_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 94 94 + yield sm.input_store.put(read_token) 95 95 + 96 96 + # Second: WRITE to satisfy the deferred read 97 97 + yield env.timeout(10) 98 98 + write_token = SMToken(target=0, addr=t1_addr, op=MemOp.WRITE, flags=None, data=0x2222, ret=None) 99 99 + yield sm.input_store.put(write_token) 100 100 + 101 101 + env.process(test_sequence()) 102 102 + env.run(until=100) 103 103 + 104 104 + # Verify result token was emitted with correct data 105 105 + assert len(collector.items) == 1 106 106 + assert collector.items[0].data == 0x2222 107 107 + 108 108 + 109 109 + class TestAC4_2T0WriteDirect: 110 110 + """AC4.2: SM WRITE to T0 stores data without presence checking.""" 111 111 + 112 112 + def test_t0_write_stores_directly(self): 113 113 + """WRITE to T0 (addr >= tier_boundary) stores in t0_store directly.""" 114 114 + env = simpy.Environment() 115 115 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 116 116 + 117 117 + t0_addr = 256 118 118 + write_token = SMToken(target=0, addr=t0_addr, op=MemOp.WRITE, flags=None, data=0x3333, ret=None) 119 119 + inject_token(env, sm.input_store, write_token) 120 120 + 121 121 + env.run(until=100) 122 122 + 123 123 + # Verify data in t0_store at correct index 124 124 + t0_idx = t0_addr - 256 125 125 + assert len(sm.t0_store) > t0_idx 126 126 + assert sm.t0_store[t0_idx] == 0x3333 127 127 + 128 128 + def test_t0_write_overwrites_previous_value(self): 129 129 + """WRITE to T0 can overwrite previous value.""" 130 130 + env = simpy.Environment() 131 131 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 132 132 + 133 133 + # Pre-populate 134 134 + sm.t0_store.append(0x1111) 135 135 + 136 136 + # Overwrite at same index 137 137 + t0_addr = 256 138 138 + write_token = SMToken(target=0, addr=t0_addr, op=MemOp.WRITE, flags=None, data=0x4444, ret=None) 139 139 + inject_token(env, sm.input_store, write_token) 140 140 + 141 141 + env.run(until=100) 142 142 + 143 143 + # Verify overwrite 144 144 + assert sm.t0_store[0] == 0x4444 145 145 + 146 146 + 147 147 + class TestAC4_3T0ReadImmediate: 148 148 + """AC4.3: SM READ on T0 address returns immediately (no deferral).""" 149 149 + 150 150 + def test_t0_read_immediate_no_deferral(self): 151 151 + """READ on T0 address returns immediately without I-structure blocking.""" 152 152 + env = simpy.Environment() 153 153 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 154 154 + 155 155 + # Pre-populate t0_store 156 156 + sm.t0_store.append(0x5555) 157 157 + 158 158 + collector = simpy.Store(env) 159 159 + sm.route_table[0] = collector 160 160 + 161 161 + t0_addr = 256 162 162 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 163 163 + read_token = SMToken(target=0, addr=t0_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 164 164 + inject_token(env, sm.input_store, read_token) 165 165 + 166 166 + env.run(until=100) 167 167 + 168 168 + # Verify immediate result 169 169 + assert len(collector.items) == 1 170 170 + assert collector.items[0].data == 0x5555 171 171 + # No deferred read 172 172 + assert sm.deferred_read is None 173 173 + 174 174 + def test_t0_read_on_empty_returns_zero(self): 175 175 + """READ on empty T0 address returns 0.""" 176 176 + env = simpy.Environment() 177 177 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 178 178 + 179 179 + # t0_store is empty 180 180 + collector = simpy.Store(env) 181 181 + sm.route_table[0] = collector 182 182 + 183 183 + t0_addr = 256 184 184 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 185 185 + read_token = SMToken(target=0, addr=t0_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 186 186 + inject_token(env, sm.input_store, read_token) 187 187 + 188 188 + env.run(until=100) 189 189 + 190 190 + # Verify 0 returned 191 191 + assert len(collector.items) == 1 192 192 + assert collector.items[0].data == 0 193 193 + 194 194 + 195 195 + class TestAC4_5T0IStructureOpsRejected: 196 196 + """AC4.5: I-structure ops (CLEAR, ALLOC, FREE, RD_INC, RD_DEC, CMP_SW) on T0 are rejected.""" 197 197 + 198 198 + def test_t0_clear_rejected(self): 199 199 + """CLEAR on T0 address is rejected (no-op).""" 200 200 + env = simpy.Environment() 201 201 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 202 202 + 203 203 + # Pre-populate t0_store 204 204 + sm.t0_store.append(0xDEAD) 205 205 + 206 206 + t0_addr = 256 207 207 + clear_token = SMToken(target=0, addr=t0_addr, op=MemOp.CLEAR, flags=None, data=None, ret=None) 208 208 + inject_token(env, sm.input_store, clear_token) 209 209 + 210 210 + env.run(until=100) 211 211 + 212 212 + # Verify t0_store unchanged 213 213 + assert sm.t0_store[0] == 0xDEAD 214 214 + 215 215 + def test_t0_alloc_rejected(self): 216 216 + """ALLOC on T0 address is rejected.""" 217 217 + env = simpy.Environment() 218 218 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 219 219 + 220 220 + t0_addr = 256 221 221 + alloc_token = SMToken(target=0, addr=t0_addr, op=MemOp.ALLOC, flags=None, data=None, ret=None) 222 222 + inject_token(env, sm.input_store, alloc_token) 223 223 + 224 224 + env.run(until=100) 225 225 + 226 226 + # t0_store should remain empty 227 227 + assert len(sm.t0_store) == 0 228 228 + 229 229 + def test_t0_free_rejected(self): 230 230 + """FREE on T0 address is rejected.""" 231 231 + env = simpy.Environment() 232 232 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 233 233 + 234 234 + # Pre-populate 235 235 + sm.t0_store.append(0xBEEF) 236 236 + 237 237 + t0_addr = 256 238 238 + free_token = SMToken(target=0, addr=t0_addr, op=MemOp.FREE, flags=None, data=None, ret=None) 239 239 + inject_token(env, sm.input_store, free_token) 240 240 + 241 241 + env.run(until=100) 242 242 + 243 243 + # t0_store unchanged 244 244 + assert sm.t0_store[0] == 0xBEEF 245 245 + 246 246 + def test_t0_rd_inc_rejected(self): 247 247 + """RD_INC on T0 address is rejected.""" 248 248 + env = simpy.Environment() 249 249 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 250 250 + 251 251 + collector = simpy.Store(env) 252 252 + sm.route_table[0] = collector 253 253 + 254 254 + t0_addr = 256 255 255 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 256 256 + inc_token = SMToken(target=0, addr=t0_addr, op=MemOp.RD_INC, flags=None, data=None, ret=ret_route) 257 257 + inject_token(env, sm.input_store, inc_token) 258 258 + 259 259 + env.run(until=100) 260 260 + 261 261 + # No result token (operation rejected) 262 262 + assert len(collector.items) == 0 263 263 + 264 264 + def test_t0_rd_dec_rejected(self): 265 265 + """RD_DEC on T0 address is rejected.""" 266 266 + env = simpy.Environment() 267 267 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 268 268 + 269 269 + collector = simpy.Store(env) 270 270 + sm.route_table[0] = collector 271 271 + 272 272 + t0_addr = 256 273 273 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 274 274 + dec_token = SMToken(target=0, addr=t0_addr, op=MemOp.RD_DEC, flags=None, data=None, ret=ret_route) 275 275 + inject_token(env, sm.input_store, dec_token) 276 276 + 277 277 + env.run(until=100) 278 278 + 279 279 + # No result token 280 280 + assert len(collector.items) == 0 281 281 + 282 282 + def test_t0_cmp_sw_rejected(self): 283 283 + """CMP_SW on T0 address is rejected.""" 284 284 + env = simpy.Environment() 285 285 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=256) 286 286 + 287 287 + # Pre-populate t0_store 288 288 + sm.t0_store.append(0x100) 289 289 + 290 290 + collector = simpy.Store(env) 291 291 + sm.route_table[0] = collector 292 292 + 293 293 + t0_addr = 256 294 294 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 295 295 + cmp_token = SMToken(target=0, addr=t0_addr, op=MemOp.CMP_SW, flags=0x100, data=0x200, ret=ret_route) 296 296 + inject_token(env, sm.input_store, cmp_token) 297 297 + 298 298 + env.run(until=100) 299 299 + 300 300 + # No result token and t0_store unchanged 301 301 + assert len(collector.items) == 0 302 302 + assert sm.t0_store[0] == 0x100 303 303 + 304 304 + 305 305 + class TestAC4_6ConfigurableTierBoundary: 306 306 + """AC4.6: Tier boundary is configurable via SMConfig; default is 256.""" 307 307 + 308 308 + def test_default_tier_boundary_is_256(self): 309 309 + """SMConfig creates SM with default tier_boundary=256.""" 310 310 + env = simpy.Environment() 311 311 + sm = StructureMemory(env, 0, cell_count=512) 312 312 + assert sm.tier_boundary == 256 313 313 + 314 314 + def test_custom_tier_boundary(self): 315 315 + """SMConfig allows custom tier_boundary.""" 316 316 + env = simpy.Environment() 317 317 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=128) 318 318 + assert sm.tier_boundary == 128 319 319 + 320 320 + def test_addr_below_custom_tier_boundary_is_t1(self): 321 321 + """Addresses below custom tier_boundary use I-structure semantics.""" 322 322 + env = simpy.Environment() 323 323 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=128) 324 324 + 325 325 + # Addr 127 should be T1 (< 128) 326 326 + t1_addr = 127 327 327 + sm.cells[t1_addr].pres = Presence.FULL 328 328 + sm.cells[t1_addr].data_l = 0x7777 329 329 + 330 330 + collector = simpy.Store(env) 331 331 + sm.route_table[0] = collector 332 332 + 333 333 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 334 334 + read_token = SMToken(target=0, addr=t1_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 335 335 + inject_token(env, sm.input_store, read_token) 336 336 + 337 337 + env.run(until=100) 338 338 + 339 339 + # Verify I-structure read worked 340 340 + assert len(collector.items) == 1 341 341 + assert collector.items[0].data == 0x7777 342 342 + 343 343 + def test_addr_at_custom_tier_boundary_is_t0(self): 344 344 + """Addresses at or above custom tier_boundary use T0 semantics.""" 345 345 + env = simpy.Environment() 346 346 + sm = StructureMemory(env, 0, cell_count=512, tier_boundary=128) 347 347 + 348 348 + # Addr 128 should be T0 (>= 128) 349 349 + t0_addr = 128 350 350 + 351 351 + collector = simpy.Store(env) 352 352 + sm.route_table[0] = collector 353 353 + 354 354 + # T0 read on empty should return 0, not defer 355 355 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 356 356 + read_token = SMToken(target=0, addr=t0_addr, op=MemOp.READ, flags=None, data=None, ret=ret_route) 357 357 + inject_token(env, sm.input_store, read_token) 358 358 + 359 359 + env.run(until=100) 360 360 + 361 361 + # Should return 0 immediately 362 362 + assert len(collector.items) == 1 363 363 + assert collector.items[0].data == 0 364 364 + 365 365 + 366 366 + class TestAC4_4T0SharedAcrossTopology: 367 367 + """AC4.4: T0 storage is shared — all SMs reference the same T0 store.""" 368 368 + 369 369 + def test_all_sms_share_same_t0_store(self): 370 370 + """All SMs in topology reference identical t0_store object.""" 371 371 + env = simpy.Environment() 372 372 + 373 373 + sys = build_topology( 374 374 + env, 375 375 + [], 376 376 + [ 377 377 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 378 378 + SMConfig(sm_id=1, cell_count=512, tier_boundary=256), 379 379 + SMConfig(sm_id=2, cell_count=512, tier_boundary=256), 380 380 + ], 381 381 + ) 382 382 + 383 383 + # Verify all SMs share same t0_store object 384 384 + assert sys.sms[0].t0_store is sys.sms[1].t0_store 385 385 + assert sys.sms[1].t0_store is sys.sms[2].t0_store 386 386 + 387 387 + def test_t0_write_by_sm0_visible_to_sm1(self): 388 388 + """Data written to T0 by SM0 is visible to SM1 READ.""" 389 389 + env = simpy.Environment() 390 390 + 391 391 + sys = build_topology( 392 392 + env, 393 393 + [], 394 394 + [ 395 395 + SMConfig(sm_id=0, cell_count=512, tier_boundary=256), 396 396 + SMConfig(sm_id=1, cell_count=512, tier_boundary=256), 397 397 + ], 398 398 + ) 399 399 + 400 400 + # Collector for SM1 results 401 401 + collector = simpy.Store(env) 402 402 + sys.sms[1].route_table[0] = collector 403 403 + 404 404 + def test_sequence(): 405 405 + # SM0 writes to T0 406 406 + write_token = SMToken(target=0, addr=256, op=MemOp.WRITE, flags=None, data=0x9999, ret=None) 407 407 + yield sys.sms[0].input_store.put(write_token) 408 408 + 409 409 + # SM1 reads from same T0 410 410 + yield env.timeout(10) 411 411 + ret_route = CMToken(target=0, offset=0, ctx=0, data=0) 412 412 + read_token = SMToken(target=1, addr=256, op=MemOp.READ, flags=None, data=None, ret=ret_route) 413 413 + yield sys.sms[1].input_store.put(read_token) 414 414 + 415 415 + env.process(test_sequence()) 416 416 + env.run(until=100) 417 417 + 418 418 + # Verify SM1 read returned SM0's written data 419 419 + assert len(collector.items) == 1 420 420 + assert collector.items[0].data == 0x9999 421 421 + 422 422 + 423 423 + class TestAC6_1SMCellIsWideField: 424 424 + """AC6.1: SMCell has is_wide field (default False).""" 425 425 + 426 426 + def test_is_wide_defaults_to_false(self): 427 427 + from sm_mod import Presence, SMCell 428 428 + cell = SMCell(Presence.EMPTY, None, None) 429 429 + assert cell.is_wide is False 430 430 + 431 431 + def test_is_wide_can_be_set_true(self): 432 432 + from sm_mod import Presence, SMCell 433 433 + cell = SMCell(Presence.EMPTY, None, None, is_wide=True) 434 434 + assert cell.is_wide is True 435 435 + 436 436 + 437 437 + class TestAC6_2IsWideMetadata: 438 438 + """AC6.2: Existing I-structure behaviour unchanged (is_wide=False path).""" 439 439 + 440 440 + def test_is_wide_false_preserves_existing_behavior(self): 441 441 + """DyadToken with is_wide=False uses existing I-structure semantics.""" 442 442 + env = simpy.Environment() 443 443 + pe_iram = {} # Will use token.offset as instruction pointer 444 444 + 445 445 + from emu.pe import ProcessingElement 446 446 + 447 447 + pe = ProcessingElement(env, 0, pe_iram) 448 448 + 449 449 + # Create DyadToken with is_wide=False (default) 450 450 + from tokens import DyadToken 451 451 + from cm_inst import Port 452 452 + 453 453 + token = DyadToken( 454 454 + target=0, 455 455 + offset=5, 456 456 + ctx=0, 457 457 + data=0x1234, 458 458 + port=Port.L, 459 459 + gen=0, 460 460 + wide=False, # Existing default 461 461 + ) 462 462 + 463 463 + # This should work exactly as before 464 464 + def inject(): 465 465 + yield pe.input_store.put(token) 466 466 + yield env.timeout(10) 467 467 + 468 468 + env.process(inject()) 469 469 + env.run(until=100) 470 470 + 471 471 + # Token should be stored in matching store at correct offset 472 472 + assert len(pe.matching_store[0]) > 5 473 473 + # The matching entry should have the token's data 474 474 + assert pe.matching_store[0][5].data == 0x1234

+8 -29

tokens.py

··· 1 1 from dataclasses import dataclass 2 2 from typing import Optional 3 3 4 4 - from cm_inst import ALUInst, CfgOp, MemOp, Port, SMInst 4 4 + from cm_inst import ALUInst, MemOp, Port, SMInst 5 5 6 6 7 7 @dataclass(frozen=True) ··· 29 29 30 30 31 31 @dataclass(frozen=True) 32 32 + class IRAMWriteToken(CMToken): 33 33 + instructions: tuple[ALUInst | SMInst, ...] 34 34 + 35 35 + 36 36 + @dataclass(frozen=True) 32 37 class SMToken(Token): 33 38 addr: int 34 39 op: MemOp 35 35 - flags: Optional[int] # TBD 40 40 + flags: Optional[int] 36 41 data: Optional[int] 37 37 - ret: Optional[CMToken] # return path 38 38 - 39 39 - 40 40 - @dataclass(frozen=True) 41 41 - class SysToken(Token): 42 42 - addr: Optional[int] 43 43 - 44 44 - 45 45 - @dataclass(frozen=True) 46 46 - class IOToken(SysToken): 47 47 - data: Optional[list[int]] 48 48 - 49 49 - 50 50 - @dataclass(frozen=True) 51 51 - class CfgToken(SysToken): 52 52 - op: CfgOp 53 53 - 54 54 - 55 55 - @dataclass(frozen=True) 56 56 - class LoadInstToken(CfgToken): 57 57 - instructions: tuple[ALUInst | SMInst, ...] 58 58 - 59 59 - 60 60 - @dataclass(frozen=True) 61 61 - class RouteSetToken(CfgToken): 62 62 - pe_routes: frozenset[int] 63 63 - sm_routes: frozenset[int] 42 42 + ret: Optional[CMToken]