CREXX

REXX Language implementation

View the Project on GitHub adesutherland/CREXX

cRexx Architecture

crexx is a custom Rexx-to-bytecode toolchain that translates Classic Rexx semantics into an optimized bytecode format executed by a specialized VM. The core bytecode path happens through four main binaries:

  1. rxc - The Compiler
  2. rxas - The Assembler
  3. rxlink - The Linker
  4. rxvm - The Virtual Machine (Interpreter)

rxpp is a first-class preprocessor component for .rxpp sources. It runs before rxc when the wrapper or a build step asks for preprocessing, but rxc does not run RXPP internally.

The Compilation Pipeline

The pipeline of transforming Rexx source code into executable bytecode is structured as follows:

  1. RXPP Preprocessor (optional)
    • Lives in the root preprocessor/ component and builds the rxpp tool plus its native precomp helper.
    • Expands .rxpp macro/directive source into generated CREXX source.
    • Emits options ... srcmap and raw @ source-map markers by default so rxc diagnostics and source-step metadata can point back to the original .rxpp file. ##CFLAG nosrcmap is the explicit legacy escape hatch for plain generated CREXX.
  2. re2c (Lexical Analyzer)
    • Used to generate the scanner/lexer from .re rules (e.g.,
      compiler/rxcpbscn.re and assembler/rxasscan.re).
    • Converts the raw Rexx source text into a stream of discrete tokens (Token structs).
  3. Lemon (Parser Generator)
    • The token stream is parsed using a grammar defined in Lemon (e.g., compiler/rxcpbgmr.y and compiler/rxcpopgr.y).
    • Lemon applies the grammar rules to recognize the syntactic structures of the Rexx language and translates them into an Abstract Syntax Tree (AST).
    • DO ... END is overloaded: statement-leading DO remains the normal grouped/loop form, while expression-position DO ... END becomes BLOCK_EXPR. The grammar resolves the command-start ambiguity by routing top-level command expressions through a restricted command_expression spine while leaving the general expression grammar free to accept block expressions.
  4. AST (Abstract Syntax Tree) & Compiler Exits
    • The primary data structure bridging the parser and the code emitter.
    • Built using a hierarchical structure of ASTNode C structs that capture operations, scopes, typing, and tree associations.
    • Standard comparison operators (=, <>, >, <, >=, <=) retain loose Rexx comparison semantics for string-targeted operands. The compiler emits the r* opcode family (req, rne, rgt, rlt, rgte, rlte), and the VM compares numerically only when both runtime string forms parse as numbers; otherwise it performs blank-padded string comparison. This prevents dynamic nonnumeric text such as "a" > 1 from raising CONVERSION_ERROR.
    • String comparison operators (==, >>, <<, >>=, <<=) are carried as a distinct OP_COMPARE_S_* family. During type validation both operands are retargeted to TP_STRING, but the optimiser still preserves intrinsic numeric constant types long enough to stringify from value rather than source spelling. That keeps folded behaviour aligned with runtime cases like 01 == 1.
    • Inline cloning must preserve the callee scope’s numeric context when it builds replacement scopes. Without that, folded float/decimal string comparisons can silently drift from runtime behaviour under local NUMERIC DIGITS / FORM / CASE settings.
    • Expression-level control flow is supported through BLOCK_EXPR (DO ... END used as an expression) and LEAVE_WITH (LEAVE WITH expr). The association pointer links each LEAVE_WITH back to its owning BLOCK_EXPR, similar to how loop LEAVE / ITERATE link to DO.
    • Contains the Exit Bridge Framework (rxcp_exit.c), which intercepts unrecognized IMPLICIT_CMD nodes, invokes user-provided rxplugin macros to generate replacement source code, parses the interpolated strings (preserving literal quotes), and surgically grafts the resulting AST back into the main tree without violating return-type constraints. Compiler-exit dispatch is keyed by the first source token in the instruction, not by the first marshalled AST node; this matters for command-position member calls such as x.add(...), whose AST node token is the member name but whose command head is x. A non-implicit exit that returns REJECT falls through to the certified implicit ADDRESS exit; ERROR remains a compiler error.
    • Implicit Main Argument Bridge: when the compiler synthesizes the file-level main() wrapper, that procedure is marked is_implicit_main. Later typing and emission use that marker to interpret arg[] / arg[n] access against the hidden command-line .string[] that the VM already passes to main. Ordinary procedures still use normal vararg semantics, and explicit zero-argument main() does not gain accidental source-level visibility of the hidden VM argv payload.
    • Exit Fragment Scope Lifecycle: replacement fragments from exits are parsed and structurally normalized before grafting, but any new lexical block scopes created by structured replacements are finalized later during symbol structuring/build. Nested DO / IF / INSTRUCTIONS emitted by exits are therefore a supported shape, and debug validation is staged after that scope rebuild so the validator sees the stabilized tree rather than the transient pre-scope fragment form.
    • Expose and Initializer Mechanics: Implements automatic scope resolution that allows namespace ... expose global variables to implicitly bind into local PROCEDURE scopes. Procedure-level name: procedure [= .type] expose var ... remains the local form for selected private module state shared by specific procedures. name: initialiser [expose var ...] creates a private zero-argument .void lifecycle procedure. Multiple initializers are retained in declaration order; the compiler rejects calls to them, namespace exposure, arguments, and returned values, then emits dedicated .initializer metadata.
    • Automatic Register Allocation: Within rxcp_val_sym.c (Step 3 - Pass 3), the compiler walks the AST (build_symbols_walker) to identify explicit NODE_REGISTER allocations via the with register.N[.view] clause on class attributes. register.0 is the source-level convention for a typed view of the containing value itself; register.1 and above are one-based child attribute slots. register.0 and duplicate typed views of the same physical register.N slot are complex attributes: ordinary emitted reads copy the linked physical payload view into a local register before expression manipulation, and writes copy the selected payload view back through the physical slot. A bounded packed or encoded binary-memory operation is the narrow exception: exact register.0.binary storage is passed directly as the operation’s receiver, while a child binary attribute may be linked only until that one operation consumes it. This compiler-managed borrow cannot escape, and ordinary binary-value reads still detach with bcopy. The compiler does not copy status flags as hidden cache maintenance for these typed views; runtime classes such as RexxValue own their library/user flag protocol explicitly. Register flag views are direct masked status-word views, with VM/compiler/readable partitions validated as read-only source views and writable source partitions lowered through settpmask. The compiler automatically maps any remaining unmapped attributes of a class to unused VM registers (r1, r2, etc.) by synthesizing implicit NODE_REGISTER AST nodes. For normal classes, prefer this implicit allocation and keep callers on factories/methods; explicit with register... mappings should be reserved for genuine physical interop where a fixed layout is required.
  5. Emitter (IR -> Assembly)
    • AST walkers (e.g., rxcp_ast_walk.c, rxcp_emit_*.c) traverse the tree.
    • Outputs intermediate string fragments representing the rxas Assembly instructions.
    • The optimiser performs opportunity-based AST inlining before emission. A callable may be structurally available for inlining, but each call site is still validated against the supported rewrite shapes. The release slice uses a 300 AST-node body cutoff as a profitability and metadata-size policy, not as a semantic safety boundary. Unsupported semantic gates still fail closed, including procedure-level expose, assembler aliasing, dynamic-index varargs, receiver/copyback shapes without proof, and interface/member dispatch that cannot be proved monomorphic. When a right-hand eager-operator call is inlineable but the left operand is already a computed value, the inliner rewrites the whole operator to a BLOCK_EXPR: first capture the left operand in a compiler temp, then LEAVE WITH temp <op> call(...). The next fixed-point pass can inline the call with a stable left operand, avoiding BLOCK_EXPR result-register aliasing without giving up the inline opportunity.
    • Inlining is implemented as AST tree surgery, not textual substitution. The cloned body must preserve callee-local scopes, remap symbols, bind arguments exactly as a real call would, preserve source/debug metadata, and leave the downstream emitter with an ordinary validated tree. This is why new inline cases are opened by specific parent/operand shape instead of by globally deciding that a callable is “always inlineable”.
    • A real method call binds the receiver value directly as a1. The inliner may therefore place a proved direct-object receiver in that same storage. For a nested call on the enclosing method’s direct §this, the current summary proof admits arbitrary internal branches and calls, plus fallthrough, when there is no more than one explicit return. Already-proved receiver aliases are remapped or retained through later clone passes. Multiple explicit returns keep the materialise/copyback path because the summary does not yet prove receiver-owned attribute-link balance on every rewritten exit. Computed receivers, class attributes, reference arguments and flow-substituted receivers also retain their evaluate-once/copyback paths. This is a per-site ownership proof, not a general §this shortcut. Exact scalar accessors add two bounded optimizations to that rule. A single direct final packed load may donate its register to the surrounding BLOCK_EXPR and fall through to the next block-end label. The ordinary method-entry initialization check may be omitted only for a direct factory receiver or a non-aliased local with one dominating factory binding; arguments, replacement writes, aliases, labels and dynamic SIGNAL retain assertinitialized.
    • Cross-file inlining uses compiler-owned META_INLINE payloads alongside normal callable metadata. The current I6 payload begins with a versioned callable summary containing formal read/write/escape and exact-shape facts, result/control/context facts, and structural cost. The reader reconstructs those facts from the body and checks the result shape against the separately parsed callable declaration. Only exact agreement opens the imported template and summary-backed binding; older, missing, malformed, or mismatched summaries retain the ordinary call. This is an evidence gate, not a permanent exclusion: when a currently closed transformation gains a complete mathematical proof and regression coverage, open that case narrowly rather than leaving the conservative fallback in place. Import attachment also distinguishes the explicit callable from any compiler-created implicit main; class-factory evidence is attached to the synthesized FACTORY contract node rather than its semantically different generic registry procedure. Libraries preserve this metadata for downstream rxc optimisation; final linked images strip it by default.
    • Suitable SELECT and equality ladders are lowered through a dedicated dispatch AST to RXAS packed jump tables. Eligibility, semantic gates, profitability thresholds, and regression invariants are documented in RXC_DISPATCH_OPTIMIZATION.md.
  6. Assembler (rxas)
    • Parses the generated rxas Assembly instructions.
    • Translates human-readable IR assembly into packed binary format (rxbin bytecode).
    • Validates .initializer metadata against its local .void, zero-argument bytecode procedure and sets the RXBIN 007 initializer feature bit.
    • Generates the final executable bytecode file.
  7. Linker (rxlink, optional)
    • Combines one or more .rxbin modules into a single linked image with one shared constant-pool record and one shared-backed module record per selected module.
    • Resolves imports and interface-provider relationships up front, while preserving the module boundaries needed by the VM loader.
    • Remaps and preserves ordered initializer metadata without turning the private lifecycle procedures into exports.
    • Can strip source-level debug metadata (META_SOURCE_STEP and META_TRACE_EVENT) for smaller deployable artifacts without removing runtime contract metadata.
  8. Interpreter (rxvm)
    • rxvm reads and executes the rxbin bytecode.
    • Modules are loaded via rxldmod.
    • Provider resolution and linking precede execution-image preparation; rxvm_initialize() then advances each mutable module overlay through its once-only initializer state before main or a public call can enter it.
    • The execution loop happens inside the rxvm_run function (e.g., in rxvmmain.c / rxvmintp.c).

Text, UTF-8, and Binary Data

In normal UTF builds, .string register data is stored as UTF-8 bytes while the VM exposes character operations as codepoint operations. A VM value tracks string_length as the byte length and, in UTF builds, also tracks string_chars. The UTF VM may privately cache one byte/codepoint lookup pair (string_cache_byte_pos and string_cache_char_pos), but that cache is not RXAS-visible value state and is never copied, traced, or modelled by flow analysis. Instructions such as strlen, indexed strchar, explicit substring destination,source,start,length, strpos, appendchar, fndblnk, and fndnblnk operate in codepoint space and use helpers such as string_cache_seek_char(), string_slice_at(), and string_concat_char() to walk or synthesize the underlying UTF-8 bytes. Some scan paths have ASCII fast paths when byte length and codepoint count match. These string instructions assume valid UTF-8 in the register payload. NUTF8 builds collapse this model back to byte positions and byte lengths without private UTF cache fields.

Numeric-to-string promotion opcodes such as itos, btos, ftos, and dtos may mutate the target VM value to materialize its string representation. That is valid even when the target register is linked to caller-visible storage, such as a class attribute, because this is representation materialization rather than a source-level assignment. The compiler must therefore emit normal type promotion for linked expression values too. The VM does not currently set or consume a “string representation is valid” cache flag, and the compiler does not reuse a previously materialized string form across multiple uses; this remains a potential performance improvement rather than current behaviour.

Hex and binary suffixed source strings now split by decoded byte content. ast_fstr() validates the hex or binary digit syntax and decodes the bytes. ast_fstr_chain() handles adjacent string literal tokens separated by a physical line break, joining the raw chunk bodies before normal decoding; only the final chunk may carry an x or b suffix, and the line break contributes no byte or character. When the decoded span is valid UTF-8, the literal remains a STRING AST node and follows the normal escaped RXAS string path. When the decoded span is not valid UTF-8, the literal becomes a BINARY AST node with canonical 0x... RXAS text. That keeps arbitrary bytes out of string-only character opcodes: using such a byte literal in a text context is rejected as CANNOT_CAST_BINARY, while assigning it to .binary, or writing literal as .binary, emits an RXAS binary load. A first untyped assignment such as x = 'ffff'x is deliberately treated as a text context, so invalid UTF-8 byte literals do not silently infer .binary. Ordinary strings can still be converted to .binary; the conversion stores the exact current UTF-8 byte sequence with no normalization. Constant strings in an explicitly binary target may be folded into a binary load, and runtime string expressions use the VM stobin instruction. The reverse conversion is explicit-only: .binary as .string validates the bytes as UTF-8. Valid constant binary values may be folded back into string literals, while invalid constant binary-to-string casts are rejected as CANNOT_CAST_BINARY and invalid runtime binary values raise UNICODE_ERROR through bintos.

.binary is present in the Level B surface and compiler metadata as TP_BINARY. The VM value has separate binary_value, binary_length, and binary_buffer_length slots. Socket and file byte operations use them (socksendb, sockrecvb, freadb, fwriteb), and native payloads reuse it with rxvm_native_payload_ops. The VM has shared binary buffer helpers for reserve/set/append/concat/slice operations. GETBYTE reads a zero-based byte index and returns -1 when the requested byte is outside the current binary length. BSLICE destination,source,start,length uses an explicit zero-based byte start and does not mutate its source. FREADB and socket binary receive reuse the binary buffer growth machinery instead of reallocating to exact byte counts.

At the Level B source surface, || is byte concatenation when either operand is .binary; the compiler targets both operands as .binary and emits bconcat. This means a string operand in a binary concat is converted to its exact UTF-8 bytes with stobin, while binary bytes are never routed through .string or bintos. Blank concatenation (OP_SCONCAT) remains a text-only operation for now and binary use is a type mismatch.

The public byte-helper BIFs live in lib/rxfnsb/rexx/binary.crexx: binlength, binbyte, binsetbyte, binsubstr, binconcat, binoverlay, bininsert, bindelstr, binpos, bincompare, bin2x, and x2bin. They are thin Level B wrappers over the RXAS byte-buffer instructions plus small loops for search and hex conversion.

At the RXAS level, BINARY_CONST and OP_BINARY support exist and 0x... operands are constant-pool binary records. load rN,0x... lowers to LOAD_REG_BINARY and populates the register’s binary slot rather than the string slot. Binary literals are byte-paired hex (0x00ff is two bytes); 0x/0X is the empty binary literal, and disassembly canonicalizes as lowercase 0x.... RXAS also exposes byte-buffer instructions for length, single-byte update, concat, append, explicit-offset slice, fixed-size overlay update, stobin string-byte conversion, and bintos binary-to-string conversion. Binary-buffer instructions never validate UTF-8 and clear VM-private UTF cache flags on the destination. bintos is the exception: it validates the source bytes and raises UNICODE_ERROR in UTF builds when they are not valid text. Most character and string opcodes still take string operands and assume valid UTF-8 in UTF builds.

Level C text and binary behavior should be treated as design space, not as settled current compiler behavior. Classic Rexx is byte-oriented and commonly stores binary data in the same text values used for strings, while current Level B separates the intended surfaces as .string and .binary. Any Level C compatibility mode therefore has to choose where Classic byte-text semantics map: to UTF-8 .string semantics, to .binary, or to an explicit option such as bytetext. Classic Rexx BIFs will need to be audited against that decision. Level G and library work use the explicit rxunicode extension path above the core codepoint-level VM string contract. The current Unicode 17.0.0 baseline provides normalization, full default case mapping, case folding, default extended grapheme clusters, and typed byte/text codecs. Word/sentence boundaries, properties, locale services, and collation remain separate future contracts.

Locked Direction

The architecture direction is:

Trust boundaries for .string validation are compiler/assembler string constants, RXVML string setters, CREXXSAA ADDRESS variable setters, RXPA native function returns and updated argument trees, text file and socket reads, ADDRESS callbacks, and any explicit byte-to-text conversion API. Internal operations that preserve validity, such as copying, concatenating two already-valid strings, slicing on codepoint boundaries, and appending a valid Unicode scalar, should propagate cached validity/count state rather than rescanning.

The VM now maintains the first two VM-private status bits as a UTF-8 cache: RXFLAG_VM_UTF8_VALID and RXFLAG_VM_UTF8_COUNT_VALID. Trusted string constants set both bits, bounded native setters validate and count before setting them, and operations that preserve validity copy or propagate the bits. Internal raw setters can still clear the cache when they are used to materialize bytes, but Level B user-facing text boundaries reject invalid UTF-8 instead of letting those bytes become .string values. RXAS string constants, source literals in text context, RXVML setters, CREXXSAA variable setters, RXPA native return/argument trees, command-line arguments, ADDRESS callback text/result copying, freadline, freadcdpt, socket text receive, and explicit .binary as .string all validate. RXPA validation is recursive over child attributes and is enabled by default; developers can temporarily opt out with CREXX_RXPA_DISABLE_UTF8_CHECKS=1 while migrating plugins. Invalid byte input must stay on the binary path (.binary, freadb, fwriteb, sockrecvb, and socksendb) until it is decoded explicitly.

The VM register/value status word is a uint32_t field partitioned in
binutils/include/rxflags.h instead of adding a second flag field:

SETTP, SETORTP, and LOADSETTP mask external writes so VM-private bits are preserved or cleared only by VM internals. Trusted RXAS may explicitly assert language certificates, but compiler ABI writes preserve that separate band and SETTP reg,0 does not clear it. Level B exposes .flags.language as read-only. Non-zero writes replace only the requested public bands. This lets compiler call-ABI setup update REGTP_* without destroying protected language facts or runtime/library flags stored on the same value. GETTP, GETANDTP, and explicit BRTPANDT masks may observe readable VM-private and language bits; unmasked BRTPT excludes both bands so cached facts do not change old branch semantics.

The four normalization bits are positive certificates about the current whole .string byte span: absence means unknown, not false. Exact whole-string copy and owner-local move preserve them. Logical string mutation clears all four; empty and known-ASCII production sets all four. A successful normalization predicate certifies its source, and normalization certifies its result. NFKD also certifies NFD, while NFKC also certifies NFC. Native post-call validation conservatively drops non-ASCII certificates because an RXPA plugin may have mutated bytes directly. Cross-worker non-ASCII materialization starts unknown; the channel format does not carry certificates.

RXAS/RXBIN integer operands remain rxinteger. The canonical definition is platform/rxinteger.h, and Release 1 fixes it to signed 64-bit across the compiler, assembler, VM, and RXPA ABI. Host pointer width is not the language integer width. Status instructions cast masks to the 32-bit flag word before applying the partition. Level B flag-view assignments use SETTPMASK, a masked replacement operation restricted to the source-writable library/user bands, so .flags.compiler and .flags.language remain read-only to source code while generated call setup and trusted language algorithms maintain their respective bands.

Regression coverage for the partition and UTF cache contract lives in

Compiler and runtime regression coverage for .string/.binary coexistence lives in

and the generated negative tests in compiler/tests/CMakeLists.txt.

Boundary regressions include direct RXAS invalid string constants,

The completed UTF baseline is:

  1. Bounded UTF-8 validate-and-count support exists through utf8nvalid_count() and is used by VM string cache refresh paths.
  2. Compiler source hex/binary literals are decoded as bytes, stay .string only when valid UTF-8, and otherwise require explicit .binary.
  3. The register status word is partitioned in rxflags.h; VM-private UTF cache bits are preserved from external writes.
  4. .binary has growable buffer helpers plus RXAS/VM literal load, length, byte get/set, append, concat, explicit-offset slice, and overlay instructions.
  5. Explicit .string/.binary coexistence is first class in Level B: as .binary stores exact UTF-8 bytes through stobin, as .string validates bytes through bintos, and valid constant casts fold in both directions. Invalid constant byte-to-text casts fail as CANNOT_CAST_BINARY; invalid runtime byte-to-text conversions raise UNICODE_ERROR.
  6. Direct RXAS string constants are part of the same contract: hand-written RXAS STRING_CONST / OP_STRING creation validates and rejects invalid UTF-8 with guidance to use binary literals.
  7. External Level B text ingress validates. The public rxvml_set_str() returns non-zero for invalid UTF-8, RXVML run arguments and ADDRESS native callback text are checked, CREXXSAA rejects invalid variable setter values immediately, RXPA recursively validates returned values and updated arguments after native calls, freadline/freadcdpt raise UNICODE_ERROR, and socket text receive reports an invalid text status. Character-walking opcodes require valid UTF cache state before using codepoint iterators.

The remaining work has moved to its owning levels:

Compiler Import Discovery

rxc does not treat every import location as both source and binary space anymore. Import discovery is now split into two root classes:

The primary source root is the directory containing the source file being compiled. Additional source roots come from -s. Binary roots come from any -i paths and the compiler executable directory. Repeated -i and -s options are accumulated in order. Search order keeps project source files ahead of deployed binary artifacts.

An extensionless initial rxc input falls back to .crexx. Headerless .crexx, .crx, and arbitrary-extension sources default to Level G; headerless .rexx defaults to Level C. .rxpp is reserved for the preprocessor and is not scanned as an import source extension.

For source discovery the compiler now performs a lightweight header scan before any full parse. That scan reads the leading options, namespace, and import clauses so namespace-invisible files can be rejected before rexbpars() and rxcp_val() are invoked. Full source parsing is still used once a source file is actually selected as an import candidate.

Within a binary root, same-stem artifacts are collapsed to the freshest candidate. If timestamps tie, .rxbin is preferred over .rxas. Directory entries are sorted by name, and discovery prefers two consecutive scans with the same entry set. If a root stays active, discovery merges a bounded number of scans and drops entries that no longer exist. This prevents native filesystem enumeration order or concurrent generated-library publication from silently changing the imported contract set without letting continuous unrelated activity block compilation.

rxc --import-resolution-report <path> writes an observe-only JSON record of those discovery decisions. The v1 report records ordered logical roots, candidate kinds and mtimes, admission/rejection/replacement reasons, content digests, executable-directory visibility, and the post-collapse candidate set. Logical identifiers (@primary-source/0, @source/N, @binary/N, and @executable/0) keep reports independent of the checkout path; the build manifest maps them to physical roots. Publication uses a temporary sibling and an atomic replacement. This initial report deliberately leaves provider_bindings empty: candidate admission is not proof that a namespace or symbol was ultimately supplied by that file. It is evidence of current selection behaviour only and does not enforce an expected provider or alter the timestamp/tie-break policy.

Compiler-generated consumer .rxas treats imported declaration blocks as a runtime dependency snapshot, not as a copy of the provider’s full public surface. The provider artifact’s exports and metadata remain definitive. When rxc emits a consumer .rxas, it re-emits only imported callable declarations that the generated instruction stream still needs for linking or runtime lookup. If optimization or inlining removes every runtime reference to an imported file, the imported declaration block is suppressed entirely.

For each retained callable that was actually reconstructed from a packaged .rxbin, rxc also emits an .autoload metadata hint containing that package’s filename stem. It deliberately emits no hint for source or RXAS imports: those inputs do not prove a distributable runtime filename. The hint is enabled by default and can be suppressed with --no-autoload. It is a deployment convenience, not a build dependency mechanism and not an alternate symbol contract; the callable name and signature remain authoritative.

Level B Classes and Interfaces

Level B interface support is now implemented across the compiler, assembler metadata path, and VM.

Compiler model

The source surface includes:

Interface methods with bodies are emitted as final/default methods. The class must still implement abstract members, but it may not override a final/default interface member.

Qualified references use namespace..symbol; the left side must be an imported namespace, not a class or interface name. namespace::symbol remains accepted as a compatibility alias.

Metadata and import

The contract model is carried through normal .rxbin metadata with:

That metadata is sufficient for import reconstruction of class/interface headers without parsing procedure bodies. Imported stubs are not re-exported as new local contracts, and richer imported stubs replace poorer duplicates.

Dynamic RXPA discovery stages initializer callbacks before reconstructing any declaration. The initializer runs under the platform loader mutex, so ADDPROC and class/interface callbacks only collect their metadata at that point. After the initializer returns and releases the mutex, rxc imports class/interface metadata first and then parses the procedure declarations while the provider remains open. This ordering permits a native procedure to return a namespaced Rexx class such as .rxstats..linearfit without recursively reopening the provider or deadlocking the loader.

Runtime dispatch

Created objects carry their concrete class identity. The VM then resolves contract calls through load/link-time registries:

srcmethodsel resolves the effective method for an interface/class receiver. The registry prefers a concrete class method and otherwise falls back to a final interface default method. The selector is a callable descriptor (rxsig1|name|return_type|args), and the resolved procedure metadata must match it.

srcfprocsel resolves interface factories from the same descriptor form. Every candidate provider is evaluated through its effective match; omitted match behaves as score 1, scores <= 0 reject, highest positive score wins, and tied scores are broken alphabetically by concrete class name.

Source Tree and Parser Mode

cRexx now has an explicit split between the user-facing source model and the mutable compiler tree.

Parser mode (rxc --syntaxhighlight) uses the same parser and early source preparation, but it routes through compiler/rxcp_highlight_controller.c and serializes DSLSH from source_tree, not from the later rewritten work tree. The controller also keeps retained parser-mode cache state for imports and exit discovery across requests.

For the compiler-side build order and tree-split details, see Parsing Pipeline Anatomy.

For the DSLSH/editor mapping and parser-mode contract, see cREXX DSLSH Integration.

For RXPP build shape, wrapper role, and source-map marker rules, see RXPP Preprocessor.

Core Data Structures

ASTNode (from compiler/rxcp_ast.h)

The ASTNode forms the backbone of the compilation process, maintaining context, tree structure (parent/child/sibling relations), value/target typing details, code generation fragments, and parser token details.

struct ASTNode {
    Context *context;
    int node_number;
    NodeType node_type;
    char* file_name;               
    ValueType value_type;    /* Value type */
    size_t value_dims;       /* Value dimensions */
    int *value_dim_base;     
    int *value_dim_elements; 
    char* value_class;       /* Value class name */
    int *target_dim_base;    
    int *target_dim_elements;
    ValueType target_type;   /* Target type */
    size_t target_dims;      /* Target dimensions */
    char* target_class;      /* Target class name */
    int high_ordinal; /* Order of node after validation but before optimisations */
    int low_ordinal;  /* lowest in this tree root */
    int register_num;
    char register_type;
    int additional_registers; 
    int num_additional_registers;
    char is_ref_arg;
    char is_opt_arg;
    char is_const_arg;
    char is_varg;
    ASTNode *free_list;
    ASTNode *parent, *child, *sibling;
    ASTNode *association; /* E.g. for LEAVE / ITERATE relevant DO node or LEAVE_WITH relevant BLOCK_EXPR */
    Token *token;
    Scope *scope;
    char *node_string;
    size_t node_string_length;
    char free_node_string;
    rxinteger int_value;
    int bool_value;
    double float_value;
    char* decimal_value; /* Decimal value as a string */
    int exit_obj_reg; /* VM register index of the attached Exit object */
    /* These are only valid after the set_source_location walker has run */
    Token *token_start, *token_end;
    char *source_start, *source_end;
    int line, column;
    SymbolNode *symbolNode;
    /* These are used by the code emitters */
    OutputFragment *output;          /* Primary node output or loop assign / init instruction */
    OutputFragment *cleanup;         /* Clean up logic */
    OutputFragment *loopstartchecks; /* Begin Loop exit checks */
    OutputFragment *loopinc;         /* Loop increments */
    OutputFragment *loopendchecks;   /* End Loop exit checks */
};

Abstract Syntax Tree: Scope & Block Construction

Mapping DO loops from Lemon Parser to AST

In the compiler, block scopes such as DO loops are strictly translated from Lemon grammar tokens into a parent-child AST topology.

Example from compiler/rxcpbgmr.y:

tk_doloop(D)  ::= TK_DO(T).
                  { D = ast_f(context, DO, T); }
do(G)         ::= tk_doloop(T) dorep(R) TK_EOC instruction_list(I) TK_END.
                  { G = T; add_ast(G,R); add_ast(G,I); }
  1. Root Creation: When a TK_DO token is identified, a new ASTNode of type DO is instantiated (via ast_f).
  2. Repetition and Conditions: Sub-rules process the dorep (like TO, BY, FOR attributes) into a REPEAT AST node or docond (like WHILE / UNTIL).
  3. Block Body: The instruction_list(I) contains all expressions and assignments defined within the loop body.
  4. Tree Assembly: The REPEAT clause, WHILE/UNTIL conditions, and the actual loop body instructions are iteratively appended as child nodes to the parent DO node using the add_ast(parent, child) and add_sbtr(older_sibling, younger_sibling) C functions.
  5. Association: The AST node also uses the association pointer to link commands like LEAVE or ITERATE directly back to the target enclosing DO loop node.

Execution and the VM

Once compilation via rxc and rxas is complete, rxvm handles the execution. Modules are ingested into memory mapping via functions like rxldmod. The VM spins up its contexts, resolves dynamically or statically linked providers, links and prepares the modules, runs each declared initializer once for that context’s mutable module overlay, and only then invokes the requested program entry through rxvm_run/rxvm_call.