Data runtime in 0.2.0

Streaming failure model and invariants

• Cancellation is checked before source initialization, before and during every pull, and before each transform or renderer write. Cancellation after a partial write remains an error; already-written bytes are not reported as a successful value.
• CLI execution also arms the plan's one monotonic absolute deadline before data initialization. Expiry sets the same engine-observed cancellation flag, covers parsing, adapters, transforms, pulls, materialization, and interactive writes, and is normalized back to the plan deadline error only after readers and external processes have cleaned up. The external-process policy may choose its earlier two-second security deadline but cannot extend the plan.
• A lazy source owns its reader until the stream is dropped. Adapter failure after earlier rows does not manufacture a successful partial result, and dropping the stream after success, cancellation, or error closes the reader.
• Values remain DataValue from source ingress through transforms and shared execution outcomes. JSON, YAML, TOML, and byte strings are named conversion boundaries; none is the evaluator's internal representation.
• Every conversion validates bytes, rows, record width, envelope/value nesting depth, value nodes, retained text, and materialized bytes. A bridge fails with ResourceLimit before retaining the first value beyond its configured bound, and reports the configured limit and observed use when safe.
• sort/sort-by, last, reverse, uniq, group-by, columns, math, table rendering, DataOutput::into_envelope, and the collected convenience APIs are the only materializing stream operations, each bounded by the row limit. Lazy lines, CSV, tar, where, get, select, reject, take, first <count>, and skip retain an O(one-row) window.
• Type mismatches and lossy conversions are operating errors. No adapter, transform, or renderer panics on user input, recurses over untrusted value structure, or performs a hidden JSON round trip.
• first and last without a count are the focused grammar's optional operations: an empty stream becomes Option::None and a present row becomes Option::Some. With a count they return a list of at most that many rows. Operating failures remain ShellError unless a caller explicitly requests a result envelope. Task remains bounded declarative state; this runtime does not claim a scheduler or asynchronous execution.
• A successful shared execution outcome has one zero status and one typed value representation. Cancellation, adapter failure, bridge overflow, and transform errors remain Err(ShellError) after owned readers are released; they never become a successful status with a partial value.

quirl data evaluates native structured values and renders them explicitly:

quirl data 'open users.csv | where enabled == "true" | select name' --format table
quirl data 'open config.toml | get service.port' --format plain
quirl data 'open users.json' --format json
quirl data '^external printf "{\"ok\":true}" | from json'

Interactive Data mode renders records as tables. Record streams, including inline JSON arrays, render in batches of at most 256 rows; a batch older than 100 ms renders with the next pulled record, so slow sources show rows early while memory stays bounded. Rows pulled before a failure remain visible. Scalar rows such as lines output stay line-oriented. quirl data ... --format table materializes one complete bounded table.

Table output measures rendered Unicode cell widths and presents records in a compact rounded frame with padded cells and a header divider. Numeric, size, and duration values are right-aligned. It never relies on terminal tab stops, so alignment is stable in the rich viewport and redirected text. A single record renders vertically as one key │ value line per field. Nested values inside cells are summarized as {record 2 fields}, [table 3 rows], or [list 4 items]; use get or --format json to see them in full.

Tables fit the terminal. In the rich surface, and for quirl data when standard output is a terminal, the widest columns shrink and end in …, never below eight cells or their heading width (up to twelve). Trailing columns that still do not fit collapse into one … column. Redirected output keeps every cell at its natural width.

Filesystem tables use a focused display projection (#, name, kind, size, and modified): headings and values are the record's own fields, so where kind == "directory" filters exactly what the table shows. Sizes use decimal units and Unix modification times are shown relative to one time captured for the complete render. The underlying records retain path, hidden, readonly, and target for transforms and machine formats; exceptional values are added to the table only when present. The rich surface keeps every outline glyph in one subdued theme color and applies the accent only to heading text; redirected table output remains free of ANSI escapes.

Transforms

The transform vocabulary follows Nushell's names where the behavior matches:

where compares sizes with size literals written with a unit and no quotes: ls | where size > 10kB. Decimal units (kB, MB, GB, TB, PB) count in thousands and binary units (KiB, MiB, GiB, TiB, PiB) in 1024s, case-insensitively. reject and group-by report a missing field instead of ignoring it, so a typo cannot silently keep data.

math keeps numbers exact: integers and decimals add as decimal digits, never through binary floating point, and a total beyond 38 significant digits fails rather than rounding. An average is exact when the division terminates and otherwise rounds to 12 decimal places. Sizes and durations reduce only with their own kind and keep their unit; min and max also compare text and dates.

Syntax ownership and analysis

quirl-data owns the focused data lexer, AST, inert diagnostics, formatter, and data-token highlighting because those contracts name data sources, byte/value bridges, transforms, predicates, and data-domain types. quirl-syntax remains the independent command-grammar foundation. This keeps the accepted ADR 0016 dependency graph unchanged: the CLI composition root maps data diagnostics into ShellError for execution and check/format boundaries, and injects a side-effect-free native analyzer callback into quirl-lsp rather than adding an LSP-to-data dependency.

Parsing never opens a path, reads the current directory, invokes an adapter, or calls an external process. The iterative lexer and literal parser preserve half-open UTF-8 byte spans and reject input before retaining more than 256 KiB of source, 32,768 tokens, 64 levels of nesting, 100,000 AST nodes, 256 fields per record/field list, or 64 KiB per decoded literal by default. Library callers can replace every syntax budget through DataSyntaxLimits.

The AST describes the currently implemented pwd, files/ls, open, JSON literal, and ^external sources; lines, from json, and to json bridges; and get, where, select, reject, sort/sort-by, take, first, last, skip, reverse, uniq, group-by, columns, math, and length transforms. Its DataType surface names existing scalar/domain DataValue forms plus List, Record, Table, Option, Result, Task, Stream, and Command. Those type nodes remain focused semantic vocabulary rather than general type inference. The evaluator now preserves DataValue through live streams and implements first as Option; Task still does not claim scheduling.

An empty expression is now a syntax diagnostic instead of the earlier implementation accident that produced null; write null explicitly. Bare paths containing whitespace also receive a migration diagnostic directing the author to quote the complete path.

Numeric where predicates and sort compare retained decimal text exactly, including comparisons with signed and unsigned integers. Distinct values above 2^53 remain distinct, trailing decimal zeros do not change equality, and a tiny nonzero decimal does not compare equal to zero through binary-float underflow. Decimal exponents must fit a signed 64-bit integer; a larger exponent produces ResourceLimit. Comparisons scan the bounded input digits with constant extra storage and never materialize the zeros implied by an exponent. This applies to the values retained by the runtime; comparison cannot recover precision already lost at a JSON, YAML, or TOML conversion boundary.

get, where, and sort accept dotted cell paths that index into lists with a non-negative integer segment as well as into records by field name, for example get items.0.name. A path accepted by get or where may end with a trailing ? to make the whole path optional: get b? returns Nothing instead of failing when a field or index is absent, and where b? == 1 treats an absent b as a non-match instead of erroring. Without the ? marker, get and where fail closed on a field a row does not have, so a typo in a path is a diagnostic rather than a silently empty result.

select requires every requested field in each consumed record. A missing field reports the transform's source range and a bounded list of available fields instead of silently omitting the column. Deferred stream failures keep that original location even when a later transform pulls the row. Empty inputs remain empty; projections preserve the requested field order.

Record is an insertion-ordered mapping, not an alphabetized one: select emits fields in the order requested rather than sorted order, and table rendering unions columns across heterogeneous rows in first-seen order. A record's own field order otherwise follows however it was constructed (for example, the order YAML or CSV headers declared it); native JSON literals and from json still normalize to serde_json's parse order, which is alphabetical, since that conversion boundary does not yet preserve source order end to end.

--format json emits a tagged Value or Stream envelope so a script does not need to infer whether an array is an ordinary value or a pipeline stream. Values themselves retain an ABI tag (int, decimal, path, size, and so on) rather than making domain values look like strings. The current native parser emits generic scalar/list/record values. Filesystem rows preserve Path, Size, and optional DateTime fields; tar rows preserve Path and Size; TOML datetimes remain DateTime. Option, Result (ok or error), and Task (pending, complete, cancelled, or failed) remain explicit in the same ABI. first produces Option::Some or Option::None. DataRuntime::eval_result_envelope is the intentional boundary that captures an operating error; other evaluator entry points return ShellError. Task states are validated declarative data and never start asynchronous work.

Supported adapters are JSON, YAML, TOML, CSV, uncompressed POSIX tar archive inspection, and filesystem rows (files [path], with ls retained as an alias). CSV and tar entries are pull-based: a row is parsed when the consumer asks for it, and cancellation is checked before each pull. The public CLI writes plain and JSON rows directly to stdout as it pulls them, keeping those paths O(window) rather than constructing a complete output string. It keeps that laziness through where, get, select, and take; sort, table rendering, DataOutput::into_envelope, and the collected eval/render convenience APIs are deliberate bounded collection boundaries. render_to is the streaming output boundary. eval_typed is the collected typed API; the older eval method is retained as a named JSON-compatibility bridge for script and watch consumers and therefore applies the same documented loss as to json. JSON, YAML, TOML, and directory entries are validated then materialized because their current underlying parsers expose whole-document APIs. YAML anchors and aliases are rejected by a syntax-aware preflight before the whole-document parser can expand them. Quotes and block-style punctuation inside plain scalars do not hide later references; genuine quoted and block scalar text remains supported. Leading UTF-8 BOMs are consumed as text encoding markers, while malformed UTF-8 is a data encoding diagnostic with conversion guidance. CSV enforces record width while constructing both the header and each lazily pulled row, before retaining the first excess field.

Every adapter enforces the default 8 MiB file size, 100,000 row/node, 256 field, 64 nesting-depth, 8 MiB retained-text, 16 MiB materialization, and 256 KiB expression limits. Library callers can set DataLimits explicitly; evaluator entry points derive matching syntax byte, depth, and field limits before parsing. Resource-limit diagnostics report the configured limit and observed use when safe. CSV requires a single unique header row and does not support multiline quoted fields. CSV columns remain strings, including boolean-looking and numeric-looking cells; compare text explicitly, for example where enabled == "true". The adapter does not infer or coerce column types. Tar inspection lists headers only: it never extracts entries and intentionally supports only uncompressed POSIX .tar archives (not zip, gzip, bzip2, xz, or PAX/GNU extended-name semantics); each header checksum is verified before the entry is reported.

Byte/value crossings are explicit. lines turns one string byte value into a lazy stream of newline-delimited strings; from json parses a string byte value (or each string stream item); and to json serializes a value or each stream item. JSON never creates domain tags. At to json, paths, datetimes, and patterns become strings; sizes and durations become strings suffixed with B and ns; non-finite decimal text is rejected rather than silently changed. YAML requires string mapping keys and rejects application-specific tags. TOML converts directly without passing through JSON. ^external <command> is the only external byte producer. A standalone DataRuntime has no ambient process capability and rejects it. The CLI injects the sandboxed process host only when the plan declares process authority, with a two-second engine-local deadline and a 1 MiB combined retained-output limit. The local deadline is intersected with the plan through the same shareable cancellation token. Non-zero exits remain ShellError failures with the bounded stderr context, rather than being silently converted into values.

This release intentionally does not implement SQLite, zip/compressed archive inspection, HTTP, background task scheduling, or a fully streaming JSON/YAML/TOML parser. HTTP is not implied by open: a future HTTP adapter must expose explicit request, timeout, redirect, byte, and capability limits. Those remain design targets rather than silently available 0.2.0 behavior.

The rich interactive data branch also uses a shared ExecutionRequest with an inherited output target. It writes terminal-safe plain values one row at a time, flushes each row, checks the shared SIGINT/SIGTERM cancellation identity before every pull and write, observes the same absolute plan deadline, and yields after at most 16 pulls. A failed, cancelled, or expired stream may already have produced scrollback but never commits its partial rows to the picker cache. Only successful typed rows enter the 128-item, 512 KiB session cache used by Alt-D; opening that picker performs no evaluation. Nested Option wrappers are flattened iteratively and rejected before output at a depth above 64, so renderer stack use is independent of source nesting.