Implementation Strategy

This document describes how the Turn compiler and runtime are built — from source text to bytecode execution. The implementation is written entirely in Rust and produces a single static binary with zero runtime dependencies.

Architecture Pipeline

The Turn toolchain follows a traditional compiler pipeline with domain-specific extensions for agentic primitives. Each stage transforms the program into a progressively lower-level representation.

Stage	Component	Output
Lexer	`src/lexer.rs`	A stream of tokens with position information for error reporting.
Parser	`src/parser.rs`	An abstract syntax tree (AST) representing the program's structure.
Semantic Analysis	`src/analysis.rs`	Validated AST with resolved identifiers and collected type information.
Compiler	`src/compiler.rs`	A flat sequence of bytecode instructions targeting the Turn VM.
VM	`src/vm.rs`	Stack-based bytecode execution with process scheduling and suspension.
Runtime	`src/runtime.rs`	Semantic memory, write-ahead log, persistence, and context management.
Tools	`src/tools.rs`	Tool registry mapping string names to native handler functions.
LLM Bridge	`src/llm_tools.rs`	OpenAI and Azure integration with JSON Schema generation from structs.
Server	`src/server.rs`	gRPC switchboard for distributed process communication across nodes.
LSP	`src/lsp.rs`	Language Server Protocol implementation for editor integration.

Bytecode Instructions

The compiler emits a flat instruction sequence that the VM executes on a stack machine. Each opcode corresponds to a well-defined semantic operation — there are no "magic" instructions whose behavior depends on runtime configuration.

Opcode	Meaning
`PushConst`	Push a literal value (number, string, boolean, null) onto the operand stack.
`Store` / `Load`	Write a value to or read a value from a named local variable in the current scope.
`Infer`	Trigger an LLM inference call with a struct schema. Suspends the VM until the result arrives.
`Remember` / `Recall`	Write to or read from the process's persistent key-value memory store.
`Spawn`	Create a new actor process from a closure. Returns a `Pid` to the parent.
`Send` / `Receive`	Enqueue a message into another process's mailbox, or block until a message arrives in the current process's mailbox.
`Link` / `Monitor`	Establish a bidirectional lifecycle link or a unidirectional monitor between two processes.
`Suspend`	Write the full VM state to durable storage and halt execution until an external event resumes it.
`CallTool`	Invoke an external tool by name. This is the effect boundary — the single point where non-determinism enters the system.

Concurrency Model

Turn processes run as lightweight async tasks on the Tokio work-stealing runtime. Each Turn process maps one-to-one with a tokio::spawn green thread, allowing thousands of concurrent agents on a single machine with minimal overhead.

NOTE

Inter-process communication uses tokio::sync::mpsc channels for zero-copy message passing. The mailbox abstraction is built on top of these channels, with persistence hooks that serialize queued messages to the write-ahead log on suspension.

The scheduler is non-preemptive within a turn — once a turn begins executing, it runs to completion or suspension without being interrupted. This eliminates a large class of concurrency bugs and makes turn execution deterministic for a given input sequence.

Persistence: Write-Ahead Log

The suspend opcode triggers a durable checkpoint. The runtime serializes the complete process state to a write-ahead log (WAL) before halting execution:

The serialized tuple contains the environment, context, memory, mailbox contents, and the current instruction pointer.
On restart, Vm::resume_from_disk reconstructs the exact execution state from the most recent checkpoint.
Mailbox messages are persisted alongside the process state, ensuring that no messages are lost across restarts.
The WAL uses append-only writes for crash safety — if the process crashes mid-write, the previous checkpoint remains valid.

This gives Turn programs orthogonal persistence: the developer never writes save/load code. The runtime handles durability transparently.

Why Rust

The choice of Rust is not a preference — it is a direct consequence of the design constraints:

Single static binary. The Turn compiler, VM, LSP server, and gRPC switchboard ship as one executable with zero runtime dependencies. No Python virtual environments, no Node.js version conflicts, no container images required.
True parallelism. Rust's async/await model with Tokio provides genuine concurrent execution without a Global Interpreter Lock. Every agent process can run on a separate CPU core.
Memory safety without garbage collection. Long-running agent processes cannot afford GC pauses or memory leaks. Rust's ownership model guarantees memory safety at compile time with zero runtime overhead.
Minimal overhead. Every CPU cycle and byte of memory that the interpreter does not consume is budget that can be spent on LLM tokens. A Rust VM has orders of magnitude less overhead than a Python or JavaScript runtime.