Columnar batch transfer contract

@honua/sdk-js/query-planner includes the first bounded data-plane slice for large query results. It defines a dependency-free Honua batch envelope, an ownership transfer primitive, and a lazy bounded worker-session protocol. Arrow/GeoArrow adapters may populate its buffers and metadata, but the envelope does not define a standalone Arrow layout and is not independently interoperable without the originating adapter's layout contract. It does not decode Arrow or ship built-in filter/reprojection/aggregation operators. Applications register those operations in their worker module.

The entrypoint is experimental while the broader planner, streaming, renderer, and realtime work in issue #394 is completed.

Create a batch without copying payload bytes

import { createColumnarBatch, leaseColumnarBatch } from "@honua/sdk-js/query-planner";

const coordinates = new Float64Array([21.31, -157.86, 21.44, -157.77]);
const batch = createColumnarBatch({
  id: "places:0",
  sequence: 0,
  rowCount: 2,
  schema: {
    id: "places-schema-v1",
    fields: [
      {
        name: "geometry",
        type: { name: "geoarrow.point", parameters: { dimensions: 2 } },
        nullable: false,
        metadata: { "ARROW:extension:name": "geoarrow.point" },
      },
    ],
    metadata: { crs: "EPSG:4326" },
  },
  buffers: [
    {
      id: "geometry.values",
      field: "geometry",
      role: "geometry",
      data: coordinates.buffer,
      byteOffset: coordinates.byteOffset,
      byteLength: coordinates.byteLength,
    },
  ],
});

const lease = leaseColumnarBatch(batch);
const receipt = await lease.transfer((message, transfer) => {
  worker.postMessage(message, { transfer: [...transfer] });
});

console.log(receipt.metrics);
// { rows: 2, logicalBytes: 32, backingBytes: 32,
//   transferBytes: 32, copiedBytes: 0, ... }

Payload bytes are never copied. Schema and metadata descriptors are normalized and frozen, while batch creation retains each caller-provided ArrayBuffer. Multiple views over one buffer produce one transfer-list entry. Zero-byte backing buffers are valid; detached buffers are rejected using an attachment check that distinguishes the two cases.

Memory ceilings

Creation and transfer default to at most 1,000,000 rows and 64 MiB of unique backing allocations per batch. Descriptor normalization also defaults to at most 4,096 total schema fields, 8,192 metadata/type-parameter entries, 16,384 buffer views, and 1 MiB of UTF-8 descriptor identifiers, keys, and string values. The corresponding maxRows, maxBackingBytes, maxSchemaNodes, maxMetadataEntries, maxBufferViews, and maxStringBytes limits may be lowered or explicitly raised; there is no unbounded mode.

Array widths are checked from one captured length before element access, and metadata keys are accumulated only to the configured bound. Normalization therefore fails before copying an oversized schema or descriptor list. Empty views and many views sharing one small backing allocation still count against maxBufferViews; they cannot bypass the CPU/heap ceiling by keeping backingBytes low.

Limits supplied when a lease is created remain its transfer defaults, so a deliberately raised ceiling is not accidentally replaced by the global default. A transfer may tighten either ceiling; a pre-handoff limit failure leaves the lease owned and invokes no target.

backingBytes sums ArrayBuffer.byteLength for every unique backing allocation. It does not claim operating-system resident or physical memory usage. This intentionally rejects a tiny view backed by an unexpectedly large buffer. logicalBytes is the sum of described view lengths and can differ when views overlap or share memory. copiedBytes is always zero for this API.

Ownership, cancellation, and acknowledgement

A ColumnarBatchLease starts in owned and can be transferred once. Live leases reserve their unique backing buffers, so the same batch—or another batch sharing one buffer—cannot be leased concurrently. Disposal releases the reservation.

The SDK checks an AbortSignal, then performs a structured-clone ownership transfer itself before invoking the consumer. The original buffers are detached, the lease becomes transferred, and the consumer receives the SDK-owned clone plus its exact transfer list for an optional subsequent worker/port handoff. The optional promise returned by the consumer is an acknowledgement and backpressure boundary.

Cancellation only applies before ownership handoff. If the consumer throws or acknowledgement fails, the error is transport-failed, but the lease stays transferred: the original buffers are already detached and retrying them would be unsafe. A limit or structured-clone failure before handoff leaves the lease owned.

dispose() is idempotent for an owned or transferred lease and releases the lease's references. It cannot revoke other references held by the caller.

Lazy worker execution

createColumnarWorkerSession() supplies the lifecycle missing from a raw postMessage call: lazy worker creation, a bounded serial queue, exact request correlation, monotonic progress, cross-thread cancellation, returned-batch validation, typed failures, and deterministic teardown. The SDK does not import or construct a browser or Node worker. The application injects a small ColumnarWorkerTransport, so its worker URL, CSP policy, credentials, module type, and bundler remain explicit.

import { createColumnarWorkerSession } from "@honua/sdk-js/query-planner";

const session = createColumnarWorkerSession({
  maxPendingRequests: 8,
  createWorker: () => {
    const worker = new Worker(new URL("./columnar.worker.js", import.meta.url), {
      type: "module",
    });
    return {
      postMessage: (message, transfer) => worker.postMessage(message, [...transfer]),
      addEventListener: worker.addEventListener.bind(worker),
      removeEventListener: worker.removeEventListener.bind(worker),
      dispose: () => worker.terminate(),
    };
  },
});

const result = await session.execute("filter-active", batch, {
  signal: abortController.signal,
  onProgress: ({ fraction, stage }) => updateProgress(fraction, stage),
});

// result.batch now owns the buffers returned by the worker.
session.dispose();

The worker module registers application-owned operations against its transport:

import { startColumnarWorkerHost } from "@honua/sdk-js/query-planner";

startColumnarWorkerHost({
  transport: wrapDedicatedWorkerGlobal(self),
  operations: {
    async "filter-active"(input, { signal, reportProgress }) {
      signal.throwIfAborted();
      reportProgress(0.25, "filter");
      const output = await filterActiveRows(input, { signal });
      reportProgress(1, "complete");
      return output;
    },
  },
});

Only one request is transferred to a session worker at a time. Queued batches remain owned by the caller until dispatch, and maxPendingRequests (16 by default) includes the active request. There is no unbounded mode. An acknowledged result is validated against the same batch ceilings before the next request starts.

Cancellation before dispatch removes the request without transferring its buffers. Cancellation after dispatch sends the versioned cancel message and retires the worker transport; queued work resumes on a newly created worker. This makes cancellation deterministic even when an application operator fails to observe its AbortSignal. Worker operators should still poll the signal so worker-local resources are released promptly. Late or duplicate messages from a retired worker cannot settle another request.

The main session and worker host both fail closed on protocol-version drift, unknown operations, batch/metric disagreement, invalid or decreasing progress, transport faults, and malformed results. Progress callbacks are observational: an exception thrown by a callback cannot corrupt ownership or settlement. Worker factories and hosts snapshot transport methods and batch ceilings before the first asynchronous boundary; later mutation of the caller-owned options object cannot redirect transferred buffers. Abort signals are accessed through a failure-contained listener/read seam, so a throwing foreign signal settles the request instead of losing it. A closed host transport can prevent a response from being delivered, but that delivery failure is contained and the host still releases its active-request slot without an unhandled rejection.

Typed errors

HonuaColumnarTransferError.code is one of:

HonuaColumnarWorkerError.code is one of:

Deliberate remaining scope

This slice does not claim the full #394 workstream. Arrow IPC decoding, built-in filter/projection/reprojection/aggregation operators, multi-batch streaming across more than one in-flight worker, planner integration, renderer consumption, batch cache identity, realtime patches, application-specific CSP worker URL policy, and bounded conversion back to feature objects remain separate work.