@aweftjs/store

Keeping a document, and keeping it as it changes.

Open a document and mutate it. Every commit it produces is written when it is produced, at the granularity of the slots that changed. There is no save interval, no flush, and no reload.

import { atomic, createObject } from '@aweftjs/core';
import { createStore, memoryDriver } from '@aweftjs/store';

const store = createStore({ driver: memoryDriver() });

const board = await store.open('board:42');
const root = board.root as Record<string, unknown>;

atomic(() => {
	root.title = 'a board';
	root.owner = 'u_7';
	root.meta = createObject({ authorId: 'u_7' });     // a slot holds one of these, not a plain object
});

await store.settled(board);        // wait for the writes already in flight

Reopen it anywhere over the same driver and it is what you left.

A slot holds a primitive, bytes, or an observable, and an observable is one core made: createObject, createArray or createMap. Assigning a plain object or array is refused (inline-container), because a document is a graph of observables and a bare object has no identity to give a row.

Reach for atomic from the start. Every assignment outside it is its own commit, and its own entry in the history. Two bare assignments are two commits, so a watcher sees the document half-changed and a replica receives the halves separately. atomic makes the whole block one commit that applies whole or not at all.

Finishing. close(handle) lets go of one document. stop() ends the whole store and the driver with it; nothing can be opened afterwards.

What it holds

Three things, one job each.

Rows. One per observable: its id, its kind, where it is attached, and its slots. A commit writes only the slots its deltas name, so the cost of a write follows the change rather than the document. Measured against writing the document whole, in an embedded Postgres outside this repo (design 047; no script here reproduces it): 0.054 ms and 3.9 KB of write-ahead log at a 620 KB document, against 34.98 ms and 571 KB. The rows are the source of truth, and a document opens by reading them.

A commit tail. Every commit, in order, with a per-document sequence: the document's history, which an application reads through since. It is derived, so losing it costs a resynchronization rather than a document, and truncate bounds it.

A projection. One record per document, one value per declared path, written in the same transaction as the commit that changed it. This is what a query reads.

Asking about documents

Declare the paths you will filter on, and query them:

const store = createStore({
	driver: memoryDriver(),
	declare: {
		ownerId: ['ownerId'],
		status: ['status'],
		author: ['meta', 'authorId'],     // a path may cross observables
	},
});

for (const { doc, fields } of await store.find({
	where: [
		{ field: 'ownerId', op: 'eq', value: 'u_7' },
		{ field: 'status', op: 'eq', value: 'open' },
	],
	sort: { field: 'ownerId' },
	limit: 20,
})) console.log(doc, fields.status);

A hit carries the declared fields the index already held, so listing what you found does not mean reopening every document. Page with after, which takes the cursor of the last hit. A cursor names a position in the order you asked for, not a document, so the next page carries on even when that document has since been removed or re-ranked. It belongs to the sort it came from: handing it to a query with a different sort is refused (reason: 'cursor').

op is eq, gt, gte, lt or lte. There is no offset, because an offset re-reads what you already saw.

The first condition is the one an index answers, and it does the pruning. The rest narrow what it returned. So put the most selective condition first, and that is the whole of the tuning advice.

There is no query for every document ordered by a field. A condition is required and it is what selects; reading everything is scan, which orders by name.

An undeclared path is refused. Not scanned:

store: a condition names title, which is not declared. Declare it, or use scan

That is deliberate. A scan is not slow in the same way on two drivers: 32x on Postgres and growing with the collection, and every record into JavaScript on IndexedDB. A query whose cost depends on where it runs is a cliff wearing a portable API. When you do want the un-indexed read, say so and give it a limit:

for (const doc of await store.scan(100, lastSeen)) await migrate(doc);

A declaration holds literal steps only, and may not cross an array. ['tasks', ANY, 'status'] names many paths inside one document, and a projection has one value per path per document, so there is nowhere for them to go. Writing the position out literally does not rescue it: an array slot is a byte string the runtime chooses and nothing keeps stable, so ['tasks', '0', 'title'] is refused at the first write rather than indexing nothing quietly. "Which documents have an urgent task" wants one row per match, which is a different index shape and is not built. See design 049.

Keep a scalar on the document instead: an application that needs "the top priority in this project" maintains that number as a slot and declares it.

Declaring is a schema decision. IndexedDB may only create an index during a version change, so a path declared later is a migration rather than a lazy index build. And indexes are not free: twenty of them came to 3.3 MB against 2.2 MB of rows at 20,000 documents. Declaring everything is the whole-document index by another route.

Reading the history

const missed = await store.since('board:42', session.seq);
for (const { seq, commit } of missed) send(seq, commit);

A sequence the tail no longer reaches back to throws truncated rather than answering with what is left, because a short answer reads exactly like a complete one. Catch it and take the document whole:

try {
	for (const { seq, commit } of await store.since('board:42', session.seq)) send(seq, commit);
} catch (e) {
	if ((e as { reason?: string }).reason !== 'truncated') throw e;
	await sendWholeDocument('board:42');
}

truncate(doc, keep) drops everything but the most recent keep commits. How much history to keep is yours; nothing in the stack reads the tail on its own.

Commits from elsewhere

import { decodeCommit } from '@aweftjs/store';

await store.receive(board, decodeCommit(bytes));

A received commit joins the same tail as a local one, under the next sequence. Apply it through the store rather than around it and a commit the document refuses is refused before anything is written.

Detaching is not deleting

An observable that loses its attach edge keeps its row and its slots. Nothing collects it until you say so:

if (store.orphans(board).length > 10_000) await store.sweep(board);

sweep frees the rows for good, in storage and not just in this handle. A swept observable cannot be re-attached afterwards, which is why nothing sweeps on its own.

This is the application's call, not the store's. The growth is visible and bounded while an automatic sweep is data leaving at a moment nothing announces.

One limit to know about. A reopened document holds what is reachable, because that is all fromSnapshot will build. So a commit that re-attaches an observable a reopened document does not hold is refused with detached-elsewhere rather than applied against an empty observable. In one process, popping an item off one list and pushing it onto another in two commits works and stays one delta. Across a restart it is refused with detached-elsewhere, even though the commit that re-attaches an observable carries everything it holds (design 084): the refusal is stricter than the rule requires, loses nothing, and stays until there is a reason to relax it.

On Postgres

import { Pool } from 'pg';
import { createStore } from '@aweftjs/store';
import { postgresDriver } from '@aweftjs/store/postgres';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const store = createStore({ driver: postgresDriver(pool), declare: { owner: ['ownerId'] } });

You hand in the pool. Anything whose connect() answers a client with query(text, values) and release() will do, which is what pg's Pool is, so nothing is installed by this package and nothing is imported by the subpath. How many connections it has, how it authenticates and when it ends are yours. close() finishes the driver and leaves the pool alone.

The driver makes its own tables (aweft_documents, aweft_rows, aweft_tail, aweft_projection, aweft_declared and aweft_version), in whatever schema the pool's search_path names, on the first call, which is declare. Two applications on one database keep apart by schema, which the pool already decides, so there is no option for a prefix. The role the pool connects as needs the right to create them.

A version row guards the tables. A shape this driver does not know is refused before anything is read:

unknown-version: these tables are version 9 and this driver writes version 1.
Point the pool at a schema this driver made, or run the driver whose version matches the tables.

Declaring a new path reads every stored document once, at startup, inside declare, before the store answers anything. That is what fills in the projection for documents written before the path existed, and it happens again when a field keeps its name and changes its path, which aweft_declared is what makes visible. It is proportional to the collection, so a large one pays it at boot, and only when the declaration actually changed. Finish one deployment before starting the next: a process still writing under the old declaration while a new one declares can have a write projected from the rows just before it.

The declaration is what the index holds. A field you stop declaring leaves the projection, so two stores over one schema declare the same paths, or the one that declares fewer drops the rest and the other pays the walk to put them back.

Nothing is told that a document changed. No notification, no LISTEN. Live updates go through sync, and one end decides a document's commits.

Two things Postgres will not hold. A string containing a zero byte (\u0000) is one, in a document name, a slot name, a slot value or a projected value alike. The driver refuses it itself, before the transaction opens, with zero-byte and the place it found it, rather than letting a bare database error out of a half written commit. And string ordering is by code point (the projection's text column is collated C), which matches every other driver for text inside the basic plane.

Tested against Postgres 18, started per run by embedded-postgres; both it and pg are development dependencies of this package and neither ships. Measured by bench/store-postgres.ts on one machine, editing one field of one record over and over, two runs back to back:

Flat in the size of the document, which is the whole point of design 047: the two document sizes differ by four times and the numbers do not separate. A declared read against 20,000 documents is 0.29 to 0.36 ms through the index, and the same script prints the EXPLAIN showing the index-only scan it takes.

Writing a driver

A driver is twelve methods: take the declaration, claim a name, write a commit's slots and its tail entry and its projection together, read the rows back, read a range of the tail, say where the head is, answer one indexed condition, read documents for scan, truncate the tail, forget swept rows, forget a whole document, and close. Nothing else. The interface stays this narrow on purpose, because widening it until the weakest target fits is how the weakest target ends up deciding what the strongest may offer.

Three obligations are easy to miss, and each one is a check in the suite:

declare carries the paths, and a driver that already holds documents brings their projection into line with it inside it, before it resolves (design 162). A document lacks a field when it has no value for it and equally when it holds one computed from a path that has since changed, so record the paths you projected under and compute a changed field again for every document; a field the declaration no longer names leaves the projection, so Found.fields carries the declared fields and nothing else. Read those documents' rows and compute each projection with projectionOf:

async declare(declaration) {
	for (const doc of await documentsMissingAField(declaration)) {
		const held = await myRead(doc);
		await writeMissingFields(doc, projectionOf(held.rows, held.root, declaration));
	}
}

It is the same function store runs on every write, so a backfill and a write cannot disagree about what a path names.

A slot may hold bytes, and they have to come back as bytes. A driver that keeps slots as JSON writes a Uint8Array as an object keyed by index and reads back that object, silently. So store a byte value as { bytes: <base64> } and turn it back into a Uint8Array on read (design 163). No reference carries that key and no primitive is an object, so the shape is unambiguous.

An absent edge on a patch means unchanged, never detached.

Prove it rather than claim it:

import { driverChecks } from '@aweftjs/testing';

for (const check of driverChecks()) {
	test(check.name, () => check.run(() => myDriver()));
}

The check that matters most runs four writers concurrently against slots that do not overlap and asserts every one of them survives. Two more exist for the same mistake from other angles: a row nothing attaches keeps its slots, and an unset slot stays unset. A driver that writes rows whole rather than slot by slot fails all three, which is what they are there to catch. Thirty-three checks in all, a count the proof program pins, and the example driver below passes every one there.

recipes/store/driver-file.ts is a complete driver written outside the package, and the proof program runs the real thing: it writes a document, sends the writing process a SIGKILL, and reopens what survived.

One thing to know about aliases

An observable has one home, the slot it was assigned into; alias(observable) from core puts a second name for it in another slot without giving it a second home, and delete on the home slot takes it out of the document while the alias still names it. A snapshot holds what a document holds, so an observable nothing attaches is not in one. If a slot still names that observable through alias, opening the document would mean rebuilding a snapshot that names what it does not contain, which fromSnapshot refuses. So store drops that slot when it opens the document rather than failing to open it at all, and the handle says which slots went: handle.droppedSlots is a list of <id>.<slot>, empty when nothing was dropped, so an application can log the loss where it happened. See design 050.

What this does not do

• Answer a question about many things inside one document. See the declaration note above.

• Talk to a network. sync moves commits between documents; this one keeps them.

• Decide who may write. That is the application's rule, run before receive.

• Sweep on its own. See above.

Known limits

A declared path that crosses an alias indexes what a reopened document no longer holds. Every row is kept whether or not something attaches it, so the projection follows an alias into a detached row and indexes the value there, while opening the document drops that slot instead (design 050). A query can therefore answer with a document that has no such field once it is open; settling it means the projection and open agreeing about an aliased row.

The design notes

A design NNN above is the note of that number in docs/design/, which says what was decided, why, what it costs, and what would reverse it.

API

Every export of @aweftjs/store, its signature as the compiler resolves it, and its block comment.

@aweftjs/store

Commit

interface Commit { readonly deltas: readonly Delta[]; readonly tag?: Tag; } (from @aweftjs/codec)

The unit that crosses every boundary.

A commit applies whole or not at all, and it carries at least one delta. Its deltas are a set, written in the canonical order of section 6.9, and a receiver replaying them one at a time would pass through states the sender never had.

tag is 4 to 32 bytes and is not a checksum of these bytes. It is a digest over the prior values of the slots this commit addresses, computed by the sender against its own state before the commit, and it answers whether the commit landed on the state the sender expected. A receiver computing a different tag treats the replicas as diverged and resynchronizes. The algorithm that fills it is still open, so nothing here computes one.

Declaration

type Declaration = Readonly<Record<string, readonly string[]>>;

The paths a store indexes, by the name a query calls each one.

Driver

interface Driver { declare(declaration: Declaration): Promise<void>; find(lookup: Lookup): Promise<Found[]>; scan(limit: number, after?: string): Promise<Found[]>; write(write: Write): Promise<number>; create(doc: string, root: string, rootKind: ObservableKind): Promise<boolean>; read(doc: string): Promise<{ root: string; rootKind: ObservableKind; rows: Row[]; } | null>; since(doc: string, seq: number): Promise<Entry[]>; head(doc: string): Promise<number>; truncate(doc: string, seq: number): Promise<void>; forget(doc: string, ids: readonly string[]): Promise<void>; remove(doc: string): Promise<void>; close(): Promise<void>; }

No block comment on this export.

Entry

interface Entry { readonly seq: number; readonly body: Uint8Array; readonly project?: Readonly<Record<string, Indexable>>; }

One commit as it was persisted, with the sequence its document gave it.

Found

interface Found { readonly doc: string; readonly fields: Readonly<Record<string, Indexable>>; readonly cursor: string; }

One document as a query answers it: its name, the declared fields it holds, and where it sat in the order it was found in.

cursor is minted by the driver and opaque to everyone else. It carries the sort value and the name, so that seeking past it works whether or not the document is still there or still ranks where it did (design 060). It means something only to the driver that minted it, and only under the sort it was minted for.

Handle

interface Handle { readonly doc: string; readonly root: object; readonly seq: number; readonly droppedSlots: readonly string[]; }

One open document.

root is the live observable. Mutate it and the change is persisted; nothing else is needed. seq is the sequence of the last commit written, which is what a resuming session presents. droppedSlots names the slots left out when the document was built from its rows, as <id>.<slot>: an alias to an observable nothing attaches cannot be rebuilt, so the slot is dropped and reported here rather than making the document unopenable (design 050). Empty when nothing was dropped, and the same list on every handle of one open document.

Held

interface Held { readonly seq: number; readonly commit: Commit; }

One commit of a document's history, as store hands it back.

Indexable

type Indexable = string | number | boolean | null;

What a declared path can hold. A reference is indexed as the id it names.

Lookup

interface Lookup { readonly where: Where; readonly sort?: { readonly field: string; readonly direction: 'asc' | 'desc'; }; readonly limit?: number; readonly after?: string; }

What a driver is asked for: one indexed condition, and how to order and page what it finds.

Exactly one condition, because that is the one an index answers. store narrows the result with the rest of the query, so every driver does the same amount of the work.

after is the cursor of the last hit of the previous page. A driver seeks past the position the cursor names, by the value inside it, and never by looking the document up again. A cursor minted under a different sort is refused with reason: 'cursor'.

Patch

interface Patch { readonly id: string; readonly kind: ObservableKind; readonly edge?: { readonly parent: string; readonly slot: string; } | null; readonly set: Readonly<Record<string, SnapshotValue>>; readonly unset: readonly string[]; }

What one commit did to one observable.

Slots, not the whole row. A commit changes the slots it names and nothing else, so writing the row whole would make two writers touching different slots of one observable overwrite each other: the same lost update a whole-document write causes, one level down. A driver merges this into what it holds.

edge is present only when the commit changed where the observable is attached, and null inside it means it was detached and keeps its slots.

Query

interface Query { readonly where: readonly Where[]; readonly sort?: { readonly field: string; readonly direction?: 'asc' | 'desc'; }; readonly limit?: number; readonly after?: string; }

A question about documents.

The first condition is answered by its index and does the pruning; the rest narrow what it returned. That is why a query costs what its first condition selects, and why putting the most selective one first is the whole of the tuning advice.

Row

interface Row { readonly id: string; readonly kind: ObservableKind; readonly parent: string | null; readonly slot: string | null; readonly slots: Readonly<Record<string, SnapshotValue>>; }

One observable, as a store holds it.

parent and slot are null together, and mean the observable has no attach edge. Its slots are kept anyway: detaching is not deleting (design 048), and what collects an orphan is a sweep the application runs rather than the act of writing.

A slot holds a primitive, bytes, or a reference to another observable. A driver that keeps slots as JSON writes a byte value as { bytes: <base64> } and turns that object back into a Uint8Array on read, because JSON has no other way to carry one and the obvious way loses it in silence (design 163). No reference carries that key and no primitive is an object.

Store

interface Store { open(doc: string, kind?: ObservableKind): Promise<Handle>; receive(handle: Handle, commit: Commit): Promise<number>; settled(handle: Handle): Promise<void>; find(query: Query): Promise<Found[]>; scan(limit: number, after?: string): Promise<Found[]>; since(doc: string, seq: number): Promise<Held[]>; truncate(doc: string, keep: number): Promise<void>; orphans(handle: Handle): string[]; sweep(handle: Handle): Promise<number>; close(handle: Handle): Promise<void>; remove(doc: string): Promise<void>; stop(): Promise<void>; head(doc: string): Promise<number>; }

A place that keeps documents.

Made by createStore. Every document it opens is cached by name, so opening one twice hands back the same live observable, reference counted.

Where

interface Where { readonly field: string; readonly op: 'eq' | 'gt' | 'gte' | 'lt' | 'lte'; readonly value: Indexable; }

One condition. field names a declared path.

Write

interface Write { readonly doc: string; readonly root: string; readonly rootKind: ObservableKind; readonly rows: readonly Patch[]; readonly dropped: readonly string[]; readonly body: Uint8Array; readonly project?: Readonly<Record<string, Indexable>>; }

What one commit changes in a document, handed to a driver as a single unit of work.

rows are merged slot by slot, dropped are the ids that lost their last edge, and body is the commit itself. A driver applies all of it or none of it: rows that disagree with the tail beside them is the one corruption this design can produce, and a transaction is what rules it out.

compare

compare: (a: Indexable, b: Indexable) => number

Order two values the way every driver has to order them.

createStore

createStore: ({ driver, declare }: { driver: Driver; declare?: Readonly<Record<string, readonly string[]>> | undefined; }) => Store

A place that keeps documents.

Params

• driver: where the documents go

• declare: the paths to index, by the name a query calls each one

Returns a store. Every document it opens is cached by name, so opening one twice hands back the same live observable and the same handle, reference counted.

Throws empty-path when a declared field names no steps, and wildcard-path when one of its steps is not a literal. Both are checked here, before anything is built from them.

Example

const store = createStore({ driver: memoryDriver() });
const board = await store.open('board:42');
(board.root as Record<string, unknown>).title = 'a board';
await store.settled(board);

decodeCommit

decodeCommit: (bytes: Uint8Array<ArrayBufferLike>) => Commit (from @aweftjs/codec)

Decode a commit, rejecting anything the encoder would not have produced.

Params

• bytes: one commit, with nothing before or after it

Returns the commit, its deltas in canonical order.

Throws a CodecError naming the rule broken. Deltas out of canonical order are rejected rather than sorted, because accepting them would mean two byte strings decode to one commit and re-encoding could not reproduce the input.

encodeCommit

encodeCommit: (commit: Commit) => Uint8Array<ArrayBufferLike> (from @aweftjs/codec)

Encode a commit.

Params

• commit: its deltas in any order, and an optional integrity tag

Returns the canonical bytes. Deltas are sorted here, so the same commit given in any order produces the same bytes.

Throws a CodecError naming the rule broken, for an empty commit, a duplicated slot, a malformed id, position or value, or a remove carrying a value.

holds

holds: (where: Where, value: Indexable) => boolean

Whether one record satisfies one condition.

memoryDriver

memoryDriver: () => Driver

A driver that keeps documents in memory.

Returns a Driver. Nothing is shared between two calls, so two stores over one of these are two separate places. The driver throws driver-closed after close, undeclared when a lookup names a field nothing indexed, cursor when a page is asked for with a cursor from another sort, and root-conflict when a write names a root the stored document does not have.

Example

const store = createStore({ driver: memoryDriver() });

projectionOf

projectionOf: (rows: readonly Row[], root: string, declaration: Readonly<Record<string, readonly string[]>>) => Readonly<Record<string, Indexable>>

Every declared field of one document, and what it holds.

Params

• rows: the document's rows, as read hands them back

• root: the id of the document's root

• declaration: the paths to read, by the name a query calls each one

Returns one entry per declared field. A field whose path is missing, or whose path runs into a primitive or into bytes part way down, holds null rather than being left out: a document absent from the index answers nothing, not even field eq null.

Throws array-in-path when a declared path crosses an array, which is a mistake rather than an empty result.

Example

const held = await driver.read(doc);
const fields = projectionOf(held.rows, held.root, { title: ['title'] });

@aweftjs/store/postgres

Pool

interface Pool { connect(): Promise<PoolClient>; }

Where the driver gets its connections.

pg's Pool is one. The application makes it, decides how many connections it has, how it authenticates and which schema it lands in, and ends it; the driver only borrows.

PoolClient

interface PoolClient { query(text: string, values?: readonly unknown[]): Promise<{ rows: unknown[]; rowCount: number | null; }>; release(): void; }

One connection, borrowed from a pool.

pg's PoolClient is one. Stated structurally so this file imports nothing.

postgresDriver

postgresDriver: (pool: Pool) => Driver

A driver that keeps documents in Postgres.

Params

• pool: where connections come from. pg's Pool is one. The driver borrows and returns them and never ends the pool, because the application made it

Returns a Driver. It makes its tables on declare, in the schema the pool's search_path names, and brings the projection into line with the declaration before declare resolves: a field nothing projected, or one projected under a path that has since changed, is computed again from every document's rows, and a field the declaration no longer names is dropped. The driver throws unknown-version when the tables in that schema were made by a driver of another shape, driver-closed after close, undeclared when a lookup names a field nothing indexed, cursor when a page is asked for with a cursor from another sort, and root-conflict when a write names a root the stored document does not have. Nothing is told that a document changed. Live updates go through sync.

Example

import { Pool } from 'pg';
import { createStore } from '@aweftjs/store';
import { postgresDriver } from '@aweftjs/store/postgres';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const store = createStore({ driver: postgresDriver(pool), declare: { title: ['title'] } });

Refusals

Every error this package raises carries a reason to switch on and a fix that says what to do. These are its reasons, from errors.txt.

Recipes

The programs in the stack's gate that use this package, each a job someone would have.

• recipes/full-stack: A page and a server in one directory: the page reaching the server in development through the dev server's proxy, and a sign-in that sets the cookie on that one origin

• recipes/health: A deploy's verification: the health endpoint polled until the shipped build is the one answering, and the two states a poll must not mistake for health

• recipes/logs: A page recorded end to end in a browser and the visit read back: an error, a rejection, a console line, a failed call on both sides, a commit's shape with a private slot absent, a typed character never stored, sign-in mid-visit

• recipes/uploads: A page uploads pictures under the gate and they paint from /files/<id>; each refusal reaches the page with its reason, a module makes a file of its own, and the static battery behind it never sees a file

• recipes/notify: Two pages of one user hear a send live and mark it read for each other, a device registered from the page, email and push against two fakes, a failed mail kept, a forged write refused, a restart, and a server with no store sending a contact form's mail

• recipes/documents-on-postgres: Documents kept in Postgres: a writer killed mid-edit, eight processes opening one name at once, and a path declared after the data was there

• recipes/posts-to-pages: Pages written while the application runs: a post published over a socket becomes a page, and a scheduled full write refreshes the sitemap

• recipes/backend: The boot pattern to copy: a twelve-line boot file and a folder of modules, one holding a document, one the rules, one a scheduler, one the gate, and one configuring a battery module

• recipes/store: the package's own recipe