Rolling upgrades: builds N-1 and N side by side
Every Loom process is a full replica behind one URL, and an upgrade replaces them one at a time. For the length of a rollout, and again during a rollback, the previous build (N-1) and the new one (N) serve the same clients against the same ClickHouse, on N’s schema. These rules keep that safe. The operator procedure is in upgrades.
An insertion fence cannot coordinate a writer that does not take it. The memory writer fence cutover drains older serving and background writers and settles their bounded sends before the source-head guarantee applies.
Retirement of an N-1 participant also requires its authority-store work to settle.
Under Postgres, a client without participant session tags is recertified only
after all generic loom authority backends are absent, including idle pools.
Stop and drain those legacy clients before the final retirement census; tagged
participants can remain live. ClickHouse-authority recovery uses the verified
deployment’s ClickHouse checks. Neither path treats an unknown authority as empty.
See the participant lifecycle baseline.
The rules
-
Migrations are additive and run at boot under the migration lock. N-1 keeps reading and writing every table it knew (
tests/unit/test_migrations_additive.py). N-1 booting on N’s schema logs one WARNING naming the migrations it does not carry (server/boot.py::warn_newer_schema) and serves. -
A version row keeps columns its build does not know.
loom.memory,loom.shared,loom.memory_embeddings,loom.session_summaries,loom.entity_summariesandloom.documentsareReplacingMergeTree(version): the newest row replaces the whole previous one. Before each insert into them,ch.insertfills every stored column outside its build’s row builder from the newest stored version of the row’s key (server/core/rmt_carry.py), so N-1’s touch, revision or tombstone keeps the values N wrote to N’s new columns. This costs a cached lookup while every build agrees with the schema, and one extra keyed read per insert while a newer column exists. A moved document’s row is a new key (its new namespace). A move N-1 makes after N’s migration added aloom.documentscolumn fills that column from the new key’s newest row (a document deleted there, or the column’s default), not from the source’s: the copy lacks that value for good, and a restore of the source, whose deleted row keeps it, brings it back there. A move N makes builds the row with every column; its chunks and table rows are copied withSELECT *and keep every column.- Each process checks
loom._migrationsand the carried tables’ metadata time every 2 s in the background and then re-reads the column lists. - A write that took the lists before the
ALTERwould still drop the column. So each process acknowledges the lists it carries by inloom.carry_acks(a fingerprint per carried table), once no write that took older lists is in flight, and a booting process waits, after its migrations and before it serves, until every other live process acknowledges the schema it now has (await_acks, 120 s, orLOOM_CH_TIMEOUT_WRITE_S+ 60 s when that is longer, then the boot fails naming them). A process killed without withdrawing can leave a carried INSERT running after its acknowledgement expires, so each carried query id names its process, and the boot also waits whilesystem.processeson any replica shows a carried insert of a process that no longer acknowledges (when it cannot read them, while such a process acknowledged within the longest an insert can run). A process withdraws at shutdown (acknowledges no table) only once no carried write of it is in flight or pinned, and the boot also waits for a withdrawn process whose carried insert still runs. N therefore writes a new column only once no writer that would drop it is left. The acknowledgement gates the writes: a carried write runs only while this process’s last published acknowledgement is younger than 15 s (half the 30 s a boot counts it for). Otherwise (never enrolled, a failed publish, a pause past expiry) the process publishes, re-reads the lists and publishes again before writing, and refuses the write (503, retryable) when any step fails, as it does when it cannot read the lists. The watcher re-publishes every 7.5 s, so a write rarely renews it, and concurrent writes that find it stale wait for one renewal. ClickHouse fences each carried batch on arrival: it is sent asINSERT … SELECT … FROM input(…)whoseWHERE throwIf(…)refuses it when, as the query starts, the table has a stored column outside the lists the rows were built against: a request held in transit (a proxy, a retry) past an ALTER lands no row, and the write answers 503 and is retried with fresh lists. A query that started earlier is bounded by itsmax_execution_time(60 s when the write class has no limit); the guard reads the receiving replica’s metadata. Each batch also carries a start deadline on the server’s clock (send time + 15 s, the clock measured from this process): one that starts later is refused, so a request still in transit can start only until then. A write that ends in an error or a cancellation after a send stays in flight, no newer lists acknowledged, until the server’s clock passes its deadline plus itsmax_execution_time(when no query of it can run or start); once the deadline has passed,KILL QUERY … SYNCand a count of 0 insystem.processeson every replica release it earlier. A held write whose queries cannot be counted shows on/v1/healthascarry_write_pinned. The acknowledgements keep old writers from meeting that refusal during a rollout; the guard makes a late one harmless. It costs a carried insert about 5 ms at p50. This fails closed: whileloom.carry_ackscannot be written, every carried write (memory.setamong them) answers 503. - A new key’s row leaves the column out, so ClickHouse computes its
DEFAULT. - Only state columns may be added to these tables. A column derived from the value
goes stale when N-1 revises the value, so one that must exist records what it was
computed from (for example the
content_sha256it describes), and N can tell it is stale. - A hand-written
INSERT … SELECTwith a column list is not carried. Only operator commands (loom reembed) write those; run them when one build serves.
- Each process checks
-
New enum or status values: N ships readers that accept them, and only N+1 writes them. N-1’s Pydantic
Literalwould answer 500 on a value it does not know. A projection failure’s name is a plain string that N-1 already tolerates: its repair worker leaves a name it does not know active asunsupported_projection. So N records a triples repair that failed after its insert asentity_triples_repair, notentity_triples, which N-1 would replay by inserting the triples again. A replay of anentity_triplesfailure that inserts them and then fails its repair keeps the failure: one mutation records that they landed and moves itsentriesandresultsto keys N-1 does not read, so N-1 refuses it (no entries) and leaves it active. N writes a failure’sabandonedstatus because N-1 reads that status as a plain string and lists onlyactiveones; thestatus="abandoned"list filter works only on N replicas. The conflict resolver’s owed follow istriples_follow_owed, nottriples_follow_rows, for the same reason: N-1’s follow does not check that the loser is still retired. While N-1 leads, its repair re-reads every failure whose name it does not know on every pass (oldest first, each answeredunsupported_projectionand left active), so they take places in its batch (LOOM_PROJECTION_REPAIR_MAX_FAILURES_PER_PASS); past that many, N-1 replays nothing else until an N replica leads. -
Wire formats (cursors, session and run ids, worker resume cursors, JSON stored in rows): N reads the old and the new format, and only N+1 writes the new one. Exception: a cursor for a request N-1 answers for
namespacealone (aknowledge.listnamingnamespaces) is new in N, and each build refuses the one the other wrote (rule 12). -
Signing keys: rotate with
--activate-inof at least twice the slowest keyring refresh interval (the command refuses less), so no process signs with a kid an N-1 replica cannot verify yet. -
Workers: one leader runs them, elected by a ClickHouse lease. Workers stay idempotent under one overlapping in-flight statement, since there are no fencing tokens. An older build leaves the lease to a newer one only when their schema versions differ (each build’s last migration number): a code-only release never hands over, and the old leader keeps the workers until it stops. Either way the order replicas are replaced in does not matter.
-
Settings: a removed setting warns at boot rather than refusing to start. A new setting’s default accepts what N-1 is configured with.
-
Retiring a column or table: stop using it in N, drop it in N+1 or later.
-
Replica-wide state lives in the deployment’s stores, never in a process. Anything a replica must agree on (leases, runs, the conflict resolver’s queue, the similarity calibration, the frontend bundle of every live build) is in a ClickHouse table, so a request can land on any replica of either build. Under the Postgres memory authority (storage authority) Postgres holds the memory heads, the revision journal, the deliveries, the scope generations and the requests’
Idempotency-Keyrecords, and N and later read and write them there (rule 33); the ClickHouse rule holds for everything else. The one exception is the bytes of document originals withLOOM_DOCUMENT_OBJECT_STORE=s3: they live in an S3-compatible bucket that every replica reads the same way, and the catalog row naming each object is in ClickHouse. A build that does not read that row’s location answers such a download 410 and cannot erase such an object, so the store is switched on after the rollout, and no rollback goes past this build while rows name objects. -
A
task.createkey’s exclusion holds only among N’s replicas.task.createwith an idempotency key holds a lease on the poster and its key (_one_create_per_key) and inserts under the key’s dedup token (insert_created_task); the coverage watch’s ready-event delivery holds one on the event’s task id (insert_ready_event). One whose lease ran out before its insert can still race the next, so both insert under one dedup token (named by the same poster and key, or the event’s task id) and storage drops the later row. An erased row’s token can outlive it and drop a new row too: with nothing stored, the row goes in again under a fresh token. The build before takes the delivery’s lease and writes its token too, so deliveries exclude across builds. Creates do not: the build before names the create’s lease and token without the poster, so neither build’s create of a key excludes the other’s, and racing they can store two tasks. A build that looks a key up without the poster answers another poster’s task to a create with the same key until it stops serving. Each task is whole (its event and its lease row) and claimable: a create of N cut off after its task row is finished by its retry, which under the key’s lease writes the created event the first one left out, and by the lease sweep, which writes its lease row after its grace. The created event takes a fresh seq, so cursor readers deliver it, unless the task has events since: then it goes just before the first of them and is history only. A first create that was only stalled past its lease, or cancelled while its insert ran on, can still land its own event. Both inserts carry one token, the task’s (insert_event), so storage keeps whichever lands first. Past the dedup window both can be stored: the backfill is markedbackfilledin its patch, and N’s readers drop it when the task has an earlier created event, so history shows one creation and a cursor delivers at most one. A reader that passed the first one’s seq before it landed and reads the backfill after misses the creation, as it misses any event landing behind it. A retry that reads an N-1 create’s task between its row and its event logs such a backfill too. N-1’s readers do not drop it; that ends once N-1 stops serving. -
A link N adds is not yet complete. N links each value to the triples and edges it stated (
entity_triple_rows,entity_edge_rows); N-1 links only keys, just after it writes the row. So a value with no row link still holds by its key link when that link is no older than the value and no row of the key links the fact, and a close N runs keeps what an N-1 row states. A key that rows of both builds link to one fact reads as N’s: there an N-1 row can lose that fact until it is stated again. -
A request field N adds is ignored by N-1 until N-1 stops serving: it answers the call as if the field were not sent, naming it in
ignored_args. Somemory.restore’s preconditionsif_superseded_byandonly_if_superseded(the dashboard’s Undo sends them) do not hold on an N-1 replica, which restores a deleted row or one retired behind another memory, and therelationsofmemory.set,memory.updateandmemory.update_by_idare dropped there, so that write graphs none of the facts it names.memory.get_by_id’somit_high_valuedoes not hold there either: N-1 answers ahighrow with its value and hash, and audits that read as a sensitive one. N’s conflict resolver does not read thememory.unsupersederecord of an Undo made on an N-1 replica, or before the upgrade: it carries neither the restored hash nor the restore version. The restored row then reads as a statement made at the Undo, newer than the memory it was retired behind, so a confident update verdict can retire that memory although the person kept both. Pinning it prevents that. N-1’sknowledge.listrows carry nonamespace, on any list; one namingnamespaceslistsnamespacealone there, and each build refuses the cursor the other wrote for such a list, so a client paging it starts again.memory.set’sif_match_sha256='absent'uses an existing, nonempty precondition field. A build without this meaning refuses it as a hash mismatch, including on a missing key, instead of storing an unconditional write. First creation may therefore answer 409 while such a build serves; clients must not retry without the condition. Ordinary content hashes still work across builds, and rollback needs no data change. -
A name N spells differently is a different node. N keeps combining marks and NFKC-normalises an entity’s name (
दिल्ली,İstanbulasistanbul); N-1 split them into fragments (द ल ल,i stanbul). ASCII names spell the same in both, and so do Latin ones but for compatibility characters, which N folds and N-1 keeps (the ligaturesfiandij, full-width letters, superscripts, fractions,ª,º,ſ). For the other scripts the two builds write separate nodes until N-1 stops serving, and a node N-1 wrote keeps its fragment name until a write states that entity again. -
An erasure N-1 runs misses the tables N adds. N-1’s saga and its census do not know
entity_triple_rowsandentity_edge_rows, so an erasure it runs during the rollout leaves their rows in its scope and still records the sagacompleted. Once N-1 stops serving, re-verify the erasures completed since the rollout began:loom erasure-re-verify --completed-since <rollout start> --resumedeletes, in each one’s scope, the rows of those two tables whose memory has no live row and no row written since the saga’s census (orphan row links). It runs no saga step again and touches nothing else: data written in the scope after the erasure finished, its links included, stays, tombstoned bymemory.deleteor not. -
A marker N writes in a column N-1 reads is not one N-1 reads as its own. The conflict resolver’s queue lives on
memory_embeddings.conflict_checked_at, where an N-1 with the per-write judge on reads the epoch itself as a judgement it owes and re-runs it. N writes00:00:01(owed) and00:00:02(judged), which N-1’s statements read as settled (they match the epoch exactly); N treats N-1’s epoch marks as owed once N-1’s own settle had a minute. N-1’s Python is wider: itsis_marktakes any time before 2000 as its pending mark, so where N-1 runs its per-write judge (LOOM_MEMORY_CONFLICT_JUDGE, off by default) its settle of a row it wrote can stamp its settle time over N’s mark on that row; N then reads an older build’s verdict, owed again only while the row isconflicted, and that one row may go unjudged. While N-1 holds the worker lease (a code-only release keeps it until it stops) N’s marks wait, and the first N leader judges them. A pin is the other way round: N’smemory.pinholds the key’s write lock, N-1’s does not, so during the rollout an N-1 pin racing an N retirement can leave the pinned row’s vector superseded until the row’s next write (orloom reconcile-status). -
A triple row N-1 writes arrives at its whole second. N-1 does not set
entity_triples.seen_at(migration 067), so its rows take the column’s default, theirlast_seensecond (.000000). Two values of a single-valued relation starting the same day and arriving in the same second, one from each build, then order the N-1 row first, whichever arrived later; the next write of either fact orders them again. -
N-1 does not know N’s retirement intents. A user erasure N-1 runs deletes no projection failure (N-1 erases them by namespace only), so a point-write retirement intent naming that user in
retired[]stays until N erases the user again, or the intent is resolved and its TTL ends it. A projection repair on N-1 leaves every projection kind N adds (rule 3),triples_follow_owed,entity_edges_drop_undeclared,entity_edges_restoreandsensitivity_facts_owedamong them, active asunsupported_projection; N’s repair replays them once N-1 stops serving. -
A new revocation key (such as SCIM’s
SCIM_PREFIX): N-1 does not read it, so a token it revokes still works on an N-1 replica. The feature that writes it is turned on only once no N-1 replica serves: SCIM provisioning (LOOM_SCIM_TOKEN) after the whole rollout. -
A new access rule (such as collections, sharing): N-1 does not know that a
col-namespace opens only through grants, so an org admin’s wildcard token reads one on an N-1 replica, and a personal collection is one too. An end-user token is refused there (N-1 has no key for itskid), but on N its first@mewrite creates a personal collection. N-1 also rewrites a document (a metadata edit, a delete) without the provenance columns, so a user erasure would miss that version; it registers acol-name that N refuses (the boot warning lists the first 20); and it resumes a user purge without purging (it reads nopurgecolumn). Collections are created (collection.create, which N-1 does not serve) and end-user issuers registered (LOOM_JWT_END_USER_ISSUERS_JSON) only once no N-1 replica serves. -
Saved conversation messages: older builds can compact concurrently saved turns and rebuild embeddings without carrying their saved state. Clients use
conversation.save_messageonly once every serving replica runs the build that preserves saved messages. A rollback pauses saved-message writes and compaction until every replica supports that state again. -
An extraction policy (
extraction_policy.set): N-1 does not readloom.extraction_policiesand extracts without it. Set a policy only once no N-1 replica serves. A rollback leaves the rows in place but no replica applies them until the build that reads them is back. -
A new side effect of a write (such as memory webhook events): N-1 does not perform it. A memory write served by an N-1 replica leaves no
loom.memory_webhook_outboxrow, so it sends no event, and an erasure saga run by N-1 has no step for the outbox and its census does not count it. Webhooks are registered (webhook.register) only once no N-1 replica serves, so no outbox row exists while an N-1 replica can complete an erasure. A rollback to N-1 after webhooks exist leaves the outbox rows written before it: N-1 never reads or erases them, and they leave the disk within about 4 days (the outbox is read for 3 days and drops a day’s part once its last row is 3 days old). Rolling forward again, N’s worker sends none of an erased user’s events, because it skips an event whose key has no history left inloom.memory_versions, which every build’s erasure deletes. -
Assistant-turn learnings (
memory_extract_agent_learnings): N-1 does not eraseloom.learnings_supporters, where a user’s quotes behind a bot’s rule live, and onlearnings.reviewit promotes anassistant_turncandidate as the approver’s own memory. Turn the setting on, and reviewassistant_turncandidates, only once no N-1 replica serves. Before a rollback, turn it off and delete the rows ofloom.learnings_supportersand theassistant_turnrows ofloom.learnings_queue(extraction). N-1 has no per-agent filter on knowledge reads, so after a rollback it serves a promoted bot rule to every agent in the namespace. To keep a rule to its bot, delete the documents taggedsource-kind:assistant_turnwithknowledge.deletebefore the rollback; otherwise accept the wider serving. -
A namespace guide (
namespace.guide_set): N-1 does not serve the guide ops, shows no guide innamespace.browse,namespace.getor the session-start pack, and a namespace erasure it runs has no step forloom.namespace_guides. Sonamespace.guide_setanswers 503OVERLOADEDwhile any live replica’s schema version (loom.instances) is below 72, or no heartbeat has been read: no guide is written while an N-1 replica could erase its namespace. A rollback to N-1 after guides exist leaves them in place, and a namespace erasure N-1 runs then leaves them too; run that erasure again once N serves.tests/unit/test_namespace_guide.pychecks the refusal; the mixed-version gate does not. The gate counts the deployment’s processes (loom.instancesof its kind): an older process outside it on the same ClickHouse, such as a branch server, is not counted. -
Platform-admin bots (
loom.machine_identities.platform_admin, migration076_machine_platform_admin.sql): the column’sUInt8 DEFAULT 0reads 0 for every bot made before it. N-1 refuses an admin bot’s exchange but accepts a flat admin JWT issued by N, and its SSO account routes lack the human-only check. Both OAuth admin exchanges andloom mint-token --platform-admintherefore refuse until a fresh, nonempty fleet view shows every replica at migration 76 or later, including processes outside the deployment on the same ClickHouse. Neither path uses the kind-filtered replica cache. OAuth answers 503temporarily_unavailable; the CLI refreshes the fleet before minting and prints no token when it is unreadable or contains an older replica. Register and use such bots only after the rollout. Before starting any N-1 replica for a rollback, disable admin bots and revoke every standalone CLI-minted admin subject or token. N-1 writes the column back to 0 when it disables or replaces a bot. -
A bot’s longest token (
loom.machine_identities.max_access_token_s, migration077_machine_access_token_ttl.sql): the column’sUInt32 DEFAULT 3600reads an hour for every bot made before it. N-1 ignores a request’sexpires_inand issues an hour, its/v1/admin/bots.createrefuses the field (422), and when it saves a bot (a disable, a replace, a namespace release that drops one of its roles) it writes 3600 back. This can lengthen a token requested or capped below an hour, or shorten one allowed longer. Socreate_bot(the admin route andloom bot create) refuses a maximum other than 3600 until a fresh fleet view shows every replica, in every deployment, at migration 77 or later; an unread or empty view (no server registered yet) refuses too. A request’sexpires_inneeds no gate: N-1 answers it with an hour. After a rollback, reapply each maximum once N serves again (loom bot create --replacewith the bot’s key or trust and roles as registered, which also ends its tokens). The mixed-version gate does not check this rule. -
An erasure saga’s audit actor (
loom.erasure_requests.actor, migration081_erasure_request_actor.sql): N-1 inserts ledger rows without the column, so a saga it checkpoints reads''afterwards, and a later resume by N names the resuming process (erasure-recovery-workeror the CLI command) on its failed and completed rows instead of the principal that asked. N-1 also still writes the body’srequesteras theactorof theerasure.*rows of a request or export it serves, or of a saga it resumes. No gate holds the rollout: the cost is the actor named on those audit rows, nothing is lost or refused, and it ends once no N-1 replica serves. The mixed-version gate does not check this rule. -
A new document-source kind (
recordingornotioninloom.knowledge_sources.kind): N-1 refuses to add one (BAD_KIND) and lists and inspects one like any source. Itscheckandrotateof one fail with a server error, since it has no probe for the kind. An N-1 sync worker that takes such a source’s turn reads it as a shared folder: the credential is not a service account key, so the turn endscredential_invalidbefore any request and retires nothing, and the next turn by N writes the real status. An N-1 mining step keys a document by its revision alone, while N keys anotiondocument by its revision and text hash, so a hand-off between the builds mines anotiondocument again. No gate holds the rollout: the cost is that status, shown until N’s turn, and those extra mining calls, and it ends once no N-1 replica serves. -
Typed datasets (
loom.datasets,loom.dataset_rows, migration083_knowledge_datasets.sql): N-1 ingests a file without its datasets and serves no dataset op. A dataset is served only while the source’s live document row has the version it was published with, so a write or a delete N-1 makes of the source stops serving N’s datasets as N’s own would; N-1’s delete or ingest of the source leaves their rows, which the source’s next ingest or delete by N removes (a delete of a source already deleted included). A metadata edit N-1 makes of an ingested source carries nothing, unlike N’s: its datasets stop being served, and the next ingest infers their types again, an earlier approval dropped; ingest the file again after the rollout to serve them, and approve again. An erasure N-1 runs does not know the two tables, so the rows N wrote in its scope stay and the saga still recordscompleted: once N-1 stops serving, request the erasures completed since the rollout began again, which N’s saga runs through every table. No gate holds the rollout. -
Extracted decisions in the model’s words (
loom decisions extract, the document-source sync’s mining step): N records a decision’s reasons in the model’s words and retires a row only when its support check finds it gone twice; N-1 retires the extractor’s row whose reasoning is not a span of the document. So an N-1 run on a document N mined (an operator’s N-1 command, an N-1 replica’s sync turn on a changed document, every changed document after a rollback) deletes the extractor’s unpinned rows N wrote and tags its pinned, attested or acted-on onessource-changed, and writes its own. The rows are the extractor’s own and the next run by N writes them again from the document; asource-changedtag stays until a run by N emits the key and finds it stated. No gate holds the rollout: the cost is that churn while N-1 serves, and after a rollback until the document is mined by N. -
A new shared-folder file kind (
docxin agdrivesource’sinclude): N accepts the kind and writes it only when a caller names it; a config withoutinclude, and the dashboard form’s default, take the four kinds N-1 knows. N-1 has nodocxin its file-kind table, so it refuses to add a config that names it (BAD_CONFIG), itscheckandrotateof such a source answer a server error, and its sync worker fails such a source’s turn before any request: the source’s state shows statusfailed, errorKeyError, nothing is retired, and the next turn by N syncs it. No gate holds the rollout: the cost falls only on sources whose config namesdocx, and it ends once no N-1 replica serves. After a rollback it stays for those sources, every file kind of them unsynced, until each is removed; a second source that names onlydocxbeside one that names the other kinds (the way document sources add a kind) confines that to the.docxfiles. N+1 may takedocxby default. This departs from rule 3 the way rule 28 does for a new source kind: holdingaddback a release would hold back every folder of.docxfiles, while the cost falls on the sources a maintainer opts in, ends with the rollout, and loses nothing. The mixed-version gate does not check this rule. -
A native document’s pictures out of its export (the
gdrivesync’sdocsconversion): N drops each pasted picture’s base64 reference definition from a native document’s Markdown export; N-1 stores the export as it comes. A document N-1 writes is found by N on its next full listing, whose stored-revision read checks each stored text for such a definition, and converted again at the same revision. N writes every native document with atext_sha256in its metadata, so the mining step of either build, which keys a document carrying one by its revision and that hash, mines the converted text once; a document N-1 writes carries none and is keyed by its revision alone, which is new whenever N-1 writes; N-1 rewrites the document only when its file changes. No gate holds the rollout: the cost is the base64 a document N-1 writes holds until N’s next full listing, and the check, which reads each document’s stored text on every full listing, stays while a build that stores raw exports may serve. The mixed-version gate does not check this rule. -
The memory authority (
loom.storage_authority, migration085_storage_authority.sql;schemas/pg_migrations/): N records the deployment’s authority at the first boot that runs its migrations (LOOM_BOOT_MIGRATE) and refuses to serve against another. A build before N does not read the record and writesloom.memorydirectly, so once a Postgres deployment is recorded every replica must run N or later, and it never rolls back past N. Postgres migrations follow rule 1: additive, applied at boot under an advisory lock, and a build at or after N serves on a later build’s Postgres schema with a warning naming the files it does not carry (Postgres migrations). The mixed-version gate does not check this rule. -
A user purge deletes the memory rows last (
memory.retire): N-1 has nomemory.retire, and itsloom erasure-re-verify --resumeruns a user saga as a tombstone, so a user saga N started and N-1 resumes can keep the answers that quoted a summary, and a purge can end with the user’s rows or the summaries built from them only tombstoned; resume user sagas N started from N. A saga’s mutation tag names its step on N, so a step resumed across builds submits its mutation again; the statements are idempotent. No gate holds the rollout, and the mixed-version gate does not check this rule. -
One token mint (
service_accounts.create,caller_home.minted_home,token_saves_to,me_service_accounts.dev_caller,admin_routes.admin_token_mint).- Where a token saves. N sends a delegated token’s unaddressed calls to the
namespace its claim names before
*when it may write there; N-1 sends them to the owner’s saved pick, as for a bare*token. A token of that shape (one fromservice_accounts.createwith a namespace before*, or anadmin/token.minttoken whosenamespacesstart with one) writes to either namespace while both builds serve, depending on the replica; both are namespaces it may write, and the split ends with the rollout. After a rollback such a token follows the pick for the rest of its life. - Where a list token saves (
saves_toclaim andloom.service_accounts.saves_to, migration093_service_account_saves_to.sql). N records the list token’s target in the token and its row, and sends its unaddressed calls there, then down its own list, never to the owner’s saved pick. N-1 ignores the claim: on an N-1 replica the token follows the saved pick, as every list token did before, until the rollout ends. A list mint N-1 serves dropssaves_to(both mint routes ignore an unknown field) and mints a token without the claim, which follows the saved pick for its life;service_accounts.createanswers where it saves under that rule, andadmin/token.mintanswers nosaves_to. A*token’s home stays the name before*, and one that may no longer write there still falls back through the saved pick. On N a delegated token, with the claim or without, never saves to a collection unasked (one it may only read can still be its default for reads,caller_home._pick_default): a list token whose only writable entry is a collection saves there by default on N-1 and nowhere by default on N, so its calls name where they save. N-1 revoking a row writessaves_toempty: a revoked token saves nowhere. - Mints refused by the other build, before anything is saved. N-1 refuses
*in a list,namespaces: "*"or none at all (create_token(),loom token createwithout--namespace) and a mint without a name (the dashboard’s form with its Name left empty,loom token createwithout--name). N has no/v1/me/setup/mcp_config, which N-1’s dashboard calls. N refusessaves_tobeside*and a collection before*, which N-1 mints (droppingsaves_to). A retry once the rollout ends works. - A mint N-1 accepts. A named list mint is saved by N-1, which answers without
saves_to,mcp_url,reads_everything,rolesandclients; N’s dashboard fills those from the page (its own/mcp, the claim) and shows the token. A list token N-1 mints gets no bound tracker record. N-1’s dashboard posting a list mint to N sends nosaves_to, so N records the first writable memory entry, kept for the token’s life; N’s answer names it, and the created dialog shows it. - The list. N-1 matches a token’s maker by the exact subject, so a person signed in
under another spelling of theirs neither sees nor revokes there a token made under
the first, and N-1’s dashboard lists a
*token N minted by the names in its claim (it reads every namespace all the same). - Who owns an auth-off token. With auth off and no bearer, N saves a token with
the single user
youas its owner and N-1 withdev. Each build’s Tokens page shows the tokens of its own owner (and, as for any owner, those on a namespace it administers); every token stays valid. On N, a token N-1 saved is listed and revoked by naming its owner:subject=devand the token’sorg(defaultfor one made on the Tokens page) onGET /v1/me/service_accountsandPOST /v1/me/service_accounts.revoke.
- Where a token saves. N sends a delegated token’s unaddressed calls to the
namespace its claim names before
-
The memory a cached
code.askanswer quoted (loom.code_answers.memory_namespaceandmemory_keys, migration086_code_answer_memory_scope.sql): N-1 writes an answer without them, and a tombstone N-1 writes of an entry N stamped (a stale entry, a negativecode.outcome) drops them, so its text stays where no erasure finds it. N-1’s saga, census andmemory.redactdo not know the table, so an erasure or redact N-1 runs leaves the answers N stamped in its scope and still recordscompleted. A saga N-1 leftin_progressrunscode_answers.deletewhen N resumes it, since the step is not in itssteps_completed; one N-1 completed is not run again, andloom erasure-re-verifydoes not reach this table. So an erasure N runs before the cache is cleared can leave text in the cache. N also changes the answer policy in the cache key, and each build reads an entry of another prompt version as dead: it serves it to nobody and tombstones it on sight. While both serve, a question asked of each in turn is synthesized again at every switch, and N-1’s tombstones of N’s entries, which carry no stamp, are the main way the stamp is lost. Where a memory namespace is bound, clear the cache once N serves alone (upgrade procedure, step 9); after a rollback, clear it once N-1 serves alone and again once N serves again. No gate holds the rollout. -
What a summary of the user’s memories fed (its history, chunks, entity vectors, graph rows and shared copies; erasure saga): N-1’s user saga and census reach only the user’s own, so a user erasure N-1 runs leaves the summary’s rows and records
completed. Outside a purge, a saga N-1 left in progress that N resumes fails N’s census when N-1 ran those rows’ steps, and the recovery worker’s next resume runs them again; the shared copies, which the user-scope census does not count, stay when N-1 passedmemory.retire. In a purge, N’smemory.retireruns them again, except for the search rows of shared copies of the user’s own memories when N-1 ran itsshared.tombstone, which deleted the rows they are found by. After a purge that passedmemory.retire, including one N-1 completed, the user’s keys are gone and nothing finds the summaries’ rows. Every build before N leaves them, and the ledger does not name the build that ran a saga: once N serves alone, request again each user erasure withoutpurgethat was requested before then (theuser_idrows ofloom.erasure_requests), and after a rollback, those requested since the rollback once N serves again (upgrade procedure, step 10, which names what the new saga also erases); no gate holds the rollout. -
Knowledge document heads (
loom.knowledge_heads, Postgres migration002_knowledge_documents.sql): under the Postgres authority N commits each document write as its source’s head; N-1 writesloom.documentsalone. N reads both and serves the newer version, and a commit over a head older than the row its writer read replaces it, so N serves N-1’s writes and a rollback loses none. Two writes of one source are refused one against the other only when both run N. N-1’s erasure has nopg_knowledge.deletestep and leaves the scope’s heads. Its tombstone stamps the rows at their version, and at a tie N serves ClickHouse’s copy, so the document reads as deleted; after a purge the first N read finds no ClickHouse row at the head’s version and nothing owed, and deletes the head instead of serving it. A delivery still owed or parked when N-1 erases its scope stays, and N’s delivery worker publishes it again; an erased document’s text stays in its head. Once N-1 stops serving, run again with N every erasure that completed while an N-1 replica served since N first served, a rollback’s included, before an operator retries a parked delivery (upgrade procedure, step 11). N refuses to boot withLOOM_RETENTION_DOCUMENTS_DAYSset under this authority (unset it first, step 2) and puts no namespace retention TTL onloom.documentsorloom.chunks; while an N-1 replica leads theretention_reconcilepass it applies a namespace’sretention_daysto both again and N’s next pass removes it, so a document whose rows age out in that window is gone from ClickHouse, and the first N read of it drops its head as a purged one. N-1’s originals sweep and user erasure readloom.documentsalone, so while N-1 runs them they may delete an original only an unpublished N commit names (any owed or parked one for a user erasure; for the sweep, one whose original is older thanLOOM_DOCUMENT_BLOB_MIN_AGE_SECONDS): that document’s download then answers 410 once its row lands, until the file is uploaded again. N-1’s restore does not wait for a delivery still owed, or for one the worker settled whose replaced request may still be sending: one that restores a source whose N tombstone landed in ClickHouse before its delivery finished (an insert that timed out and landed, a failed completion) has its chunks retired again when N’s worker publishes that delivery or the replaced request’s retirement lands, and the document serves no search hits until it is written again. No gate holds the rollout, and the mixed-version gate does not check this rule. -
The chunks a knowledge head serves (
chunks_revision,chunks_versiononloom.knowledge_heads, Postgres migration004_knowledge_chunk_heads.sql): under the Postgres authority N’s search, outline, synthesis andmemory.stackknowledge reads serve only a head’s chunks (a source with no head serves what ClickHouse holds) (knowledge search); N-1 serves every chunk ClickHouse reads as live, as before, so while it serves, a search it answers can show an older version still being retired or a write not committed. N-1’s commits leave both columns as they were, so after one that numbers a new revisionchunks_revisionis no longer the head’s and N judges that source’s chunks by revision alone, which also serves a refused writer’s chunks of that revision until the source’s next N write that inserts chunks (a delete, a metadata edit or a restore keeps the columns as they are). A head N-1 created holds none. N-1’sknowledge.addandbulk_addagain keep the earlier version’s chunks, which N does not serve and retires at the source’s next N write that inserts chunks. No gate holds the rollout, and the mixed-version gate does not check this rule. -
Knowledge ingestion jobs (
loom.knowledge_jobs, Postgres migration005_knowledge_jobs.sql): under the Postgres authority N keeps each job in Postgres and projects each committed version intoloom.knowledge_ingestions, best effort, so N-1 may read a job a version behind until N’s next save, or until N’s worker next takes it, and may run again a job N ended whose last projection did not land (a published one N-1 finds published); N-1’s originals sweep and user erasure read open jobs from ClickHouse alone, so while N-1 runs them they may delete the original of an N job whose projection has not landed and whose upload has expired, and that job then failsoriginal_gone; N-1 keeps its jobs in ClickHouse alone. N’s status and cancel read the newer version of the two stores, and its worker’s queue interleaves its own due jobs with ClickHouse’s, the two leading in turn from one pass to the next. A job N-1 queued runs in ClickHouse alone, as N-1 runs it; a run of one Postgres holds first takes the version N-1 saved when it is the newer, and waits while an earlier open job of its source is held in Postgres or only in ClickHouse, so a source’s jobs still run one at a time, oldest first. N thus runs, reports and cancels the jobs N-1 queued or ran. A cancel N accepts is also written toloom.knowledge_ingest_cancels, which N-1 reads, and N reads the cancels N-1 writes there. The deployment-wide job lease alone keeps a run of each build from overlapping the other’s: N-1 does not raise a job’slease_token, so the token does not fence an N run whose lease lapsed against an N-1 taker. N-1’s erasure has no step forloom.knowledge_jobsand leaves the scope’s jobs there, and N’s worker projects an open one back intoloom.knowledge_ingestionsand runs it; once N-1 stops serving, run such an erasure again with N, as for rule 38 (upgrade procedure, step 12). N breaks a tie between two jobs of a source created in one millisecond by the id’s text, N-1 by its UUID, so the two builds may each run one of such a pair at once. No gate holds the rollout, and the mixed-version gate does not check this rule. -
The erasure receipt (
loom.erasure_requests.census, migration091_erasure_census.sql): N writes every count of a complete census with the verify columns of a saga’s terminal and failed rows, and the recovery worker’s abandonment checkpoint carries it forward; a resume writes in-progress rows without it until its own census lands. N-1 neither writes nor reads it, so a row N-1 checkpointed reads'', which N’serasure.statusanswers as no receipt (census: null). N also writes the purge a saga applies to thepurgecolumn (false for a single namespace), which N-1 reads as before, since N-1 never purged a single namespace either. No gate holds the rollout, and the mixed-version gate does not check this rule. -
The knowledge dataset catalog (
loom.knowledge_datasets, Postgres migration007_knowledge_datasets.sql): under the Postgres authority N commits each catalog row there and projects it intoloom.datasets, best effort, so N-1 may read a catalog row a version behind until N’s next write of it; N-1 writesloom.datasetsalone. Every read of the catalog by N (datasets_repo.at_version:knowledge.dataset_get,knowledge.query, an approval’s reads, a publish’s and a metadata edit’s carry, and a publish’s check that no approval landed) takes Postgres’s rows and those only ClickHouse holds, the newer version of a dataset winning, so N serves and queries what N-1 published or approved. N’s approval commits in the transaction that holds the source’s head, over the version of the catalog row it read: Postgres’s, or a newer one N-1 wrote to ClickHouse, which the approval then commits to Postgres. N-1’s approval writes ClickHouse alone, and N reads it as the newer version once the replica it reads has caught up: until then N serves Postgres’s older row, whose replaced snapshot N-1 may already have deleted, so the dataset reads empty, and an approval by N in that window outranks N-1’s. Both builds take the source’s claim for an approval and a publish, so an approval and a publish of one source never overlap, whichever builds run them. N-1’s deletes (knowledge.delete, a later publish’s retirement) and its erasure removeloom.datasetsrows alone and leave N’s catalog rows in Postgres until that source is next published or deleted by N: ask such a delete again with N, which removes them; once N-1 stops serving, run such an erasure again with N (upgrade procedure, step 12). No gate holds the rollout, and the mixed-version gate does not check this rule. -
A revision’s delivery time (
delivered_atonloom.memory_revisions, Postgres migration010_revision_delivered_at.sql): under the Postgres authority N stamps a revision it delivers more than 15 minutes after its commit, and the payload clearing pass takes an unstamped one as delivered within 15 minutes ofcommitted_at(payload clearing). N-1 completes a late delivery without the stamp, so a revision it delivered late counts as delivered earlier than it was, by up to that delivery’s lateness, and stays so. TurnLOOM_PG_PAYLOAD_CLEAR_ENABLEDon no sooner than one clearing horizon after N serves alone: every revision N-1 delivered is then past the horizon from its real delivery. A rollback to N-1 starts that wait again: turn the switch off before it, and on again no sooner than one clearing horizon after N serves alone after the roll forward. The switch is off by default, and N-1 runs no clearing pass. N-1’s delivery cannot send a revision whose payload was cleared. Once a clearing pass has run,loom reconcileruns only with a build that carries 010, and that build is the one the deployment starts on after it; a rollback waits until N owes no claimable delivery for a cleared revision, and while N-1 serves a retry of parked deliveries leaves the cleared revisions out (rollback). No gate holds the rollout, and the mixed-version gate does not check this rule. -
Document sources on any namespace (
knowledge.source.*without the trunk check): N-1 still refusesadd,check,rotateandsync_nowon a namespace that is not a repository’s trunk (NOT_A_CODE_NAMESPACE), so while it serves, the dashboard’s Add (the list the Library’s Sources button opens offers it on every shared namespace), Check, Sync now and Rotate of such a source fail on the replicas it answers from, and its sync worker skips such a source until N leads the workers. N-1’snamespace.delete,collection.deleteand erasure do not retire a namespace’s sources. A source whose namespace N-1 deleted stays active with its credential, and the sync skips it (its namespace is inactive); one whose namespace N-1 erased keeps syncing, and its next full listing writes the namespace’s documents back. An erasure N-1 started and N finished runsknowledge_sources.retirelate, and its final census reopens the steps for what the source wrote back meanwhile;loom erasure-re-verify(rule 14) runs no step. N-1’sadddoes not read the ledger, so a source it adds during an erasure N runs, after N’sknowledge_sources.retire, stays active and syncs after the erasure completes. Once N serves alone, the upgrade procedure, step 13, lists what N-1 left, with its queries and cleanup. No gate holds the rollout: the cost is those refusals and that cleanup, and it ends once no N-1 replica serves. The mixed-version gate does not check this rule. -
A namespace’s promoted rows (
shared.tombstonein the namespace saga): N-1’s namespace erasure has no step forloom.sharedand its census does not count the table, so anamespaceornamespace_listerasure N-1 runs completes with the namespace’s promoted rows live, and so did every one an older build completed. Their search rows are gone (memory_embeddings.tombstone), butmemory.lineageand the shared reads still return them. Once N serves alone, the upgrade procedure, step 14, lists each namespace an erasure in the ledger (kept 90 days) reached and runs over it what N’s step runs: rows promoted before the erasure are tombstoned, or deleted after a purge. An older erasure’s rows cannot be found and stay. No gate holds the rollout, and the mixed-version gate does not check this rule. -
An
/assistanthistory entry (the dashboard’s expanded assistant window has no address): an N-1 tab pushed/assistantentries, and after the ReloadBar’s reload on N, Back or Forward onto one reaches N in the same document. N answers it: the address becomes the landing page’s (landingTargetfor the tab’s namespace; the view the tab shows when that is Setup), the page arrives there, and the assistant opens over it (expanded from 1024 px; nothing opens on Setup, on a namespace the tab cannot read, or at phone widths). The server keeps serving theassistantpage segment for bookmarks, which N answers the same way at boot (except that on a phone the sheet opens over the landing page), so this answer stays as long as the segment is served. The mixed-version gate does not check this rule;frontend/e2e/assistant-launch.spec.ts(“an entry an earlier build pushed opens the assistant on Back after a reload”) does. -
A cell’s org triggers and
loom.tenancyguard keep their names and shape (cells). The session markerloom.tenancy_checked, the functionsloom.tenancy_refuse_unheld()andloom.tenancy_refuse_unmarked(), the trigger namestenancy_insert,tenancy_updateandtenancy_record, and their shape (AFTER, row level, on their events) do not change, since builds N-1 and N create and read them on one cell. Each function’s body changes only through a numbered migration (CREATE OR REPLACE FUNCTION), andmissingaccepts the body and settings any shipped file writes. A changed body ships in two releases: release K adds the new body tomissingas an accepted value beside the shipped files’, and the migration that writes it ships no earlier than K+1, so an N-1loom doctoror attach on a migrated cell names no defect. TheWHENtext lives in the code that creates the triggers (server/core/cell_triggers.py), so a changedWHENships in three releases, in the body’s order: release K adds the newWHENtomissing(whichloom doctorand attach read) as an accepted value beside the one it creates; K+1 creates the newWHENand keeps the old one accepted, so a K pod that creates a trigger on a skipped table, and a Kmissingreading one K+1 created, both name no defect; and the migration that drops the triggers, which the same boot then creates with the newWHEN, ships no earlier than K+2, so the N-1 that could create a skipped table’s trigger already carries the new text. The gate on processes withLOOM_CELLunset, the marker and the triggers ship in one release: a release that carried the triggers without the marker would refuse Loom’s own writes on a cell, and one with the gate alone would leave its unchecked writers unguarded. It breaks side-by-side serving in one place: on a later cell already prepared, the release’s first migrating boot creates the triggers, and an older pod still serving there has its inserts of a tenant org’s rows refused, so such a cell is rolled out in one step (upgrades, step 2). The mixed-version gate does not check this rule. -
Knowledge gaps (
loom.search_missesandloom.knowledge_gaps, migration 95): N-1’s erasure saga has no step for either table. So while any live replica’s schema version is below 95, or no heartbeat has been read, no miss is recorded, theknowledge_gapsworker writes nothing andknowledge.gaps.listanswers 503, asnamespace.guide_setdoes under rule 24. The operator turnsLOOM_KNOWLEDGE_GAPS_ENABLEDon only once every replica runs N. Before a rollback, turn it off and delete both tables’ rows, then delete again after the first N-1 replica serves (knowledge gaps).tests/unit/test_knowledge_gaps.pychecks the refusals; the mixed-version gate does not. -
A document source’s items (
loom.knowledge_sources.source_kind,items_json,secret_enc, migration096_knowledge_source_items.sql): N writes a source whose items the id-listconfigcan hold (only non-recursive folders, at most 20 of anything) as N-1 writes it,kind,config_jsonand the credential incredential_enc, withitems_jsonbeside them, so N-1 lists, checks, rotates and syncs it as before. Any other source is stored withkindempty, its kind insource_kind, its credential insecret_enc(purposeknowledge_sources.secret),credential_encempty andconfig_jsonholding no item. N-1 lists such a source with kind''; itscheckanswerscredential_invalid(itscredential_encis empty) and asks for a rotate, and itsrotatefails with a server error before writing, since it has no probe for kind''; and its sync turn reads it as a shared folder with no credential, so the turn endscredential_invalidbefore any request and retires nothing. N-1’sremoveof it, and its retirement when its namespace goes, write an inactive row whose new columns read empty. Soknowledge.source.addandknowledge.source.updateanswer 503OVERLOADED(retry_after_s30) for a source they would store withkindempty while any live process of any deployment on the ClickHouse service (loom.instances,read_live(all_deployments=True)) reports a schema version below 96, or none has been read. N readssource_kindfirst andkindwhen it is empty, anditems_jsonfirst andconfig_jsonwhen it holds no item, so a row N-1 rewrote reads as it wrote it. N-1’ssync_nowwithoutfullwritesfull: falseover an update’s waiting full request, so while it serves, an added item’s files older than the stamp wait for the periodic full listing. N-1’sloom secrets resealdoes not resealsecret_enc, and its--checkreports clean, so run the reseal from build N before droppingLOOM_JWT_SECRET_PREVIOUS: asecret_encleft under the old key reads as no credential and its source’s turns endcredential_invaliduntil rotated. A later release, once no build below 96 runs, writeskindfor every source and keeps one credential column.tests/unit/test_knowledge_sources.pychecks the refusal, andtests/unit/test_knowledge_source_sync.pythat a folder turn over such a row ends before any request; N-1’scheckandrotatefailing is read from its code, and the mixed-version gate does not check this rule. -
A source’s document bound (
LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS): N reads a source’s stored copies in pages and holds back new files past the setting, for every kind. N-1 reads at most 50,001 of them: itsgdriveturn holds back new files past 50,000 and endsfailedwithtoo_many_documentsover it, retiring nothing; itslinear,notionandrecordingturns check no bound, so past the cut a stored copy reads as new and is fetched again each turn, and the cut read retires less. Both rewrite and retire under the same source lease. While N-1 serves, a setting at or below 50,000 is safe; raise it only once every replica runs N. The bound tests intests/unit/test_knowledge_source_sync.py,test_knowledge_source_sync_linear.py,test_notion_source_sync.pyandtest_knowledge_source_recordings.pycheck N’s side; N-1’s is read from its code, and the mixed-version gate does not check this rule. -
Mirror names are reserved (migration
097_mirror_names_reserved.sql): N’s gate refuses every request that names a namespace of the formacl-and 24 hex digits (only a platform admin’serasure.requestmay name one), with auth on or off.namespace.registerandnamespace.restorerefuse the form, and N’s org-wide scopes, thesql_v1_views and the analytics export leave it out. N-1 admits the form as any other name. The migration refuses to run while any namespace of the form is registered or holds live rows in ClickHouse (the registry, memory, documents, conversations, threads, sources and code repositories), naming up to ten asorg/namespacewith their tables and counting the rest. Since N refuses every request naming such a namespace, the operator clears it with N-1: an erasure of what it holds,code.repo.removefor a repository andnamespace.delete. The build that writes mirrors comes after this one, so the build beside it already refuses the form; the one gap is a name an N-1 process registers or writes after the migration ran. The mirror writer draws a new name again while the one it drew is in use (server/knowledge/app/mirrors.py, rule 54), so no such namespace is ever taken for a mirror.tests/unit/test_mirror_names_refused.pychecks N’s side, and the mixed-version gate does not check this rule. -
The mirror registry (migration
098_knowledge_mirrors.sql): N addsloom.knowledge_mirrorsand theaccesscolumn ofloom.knowledge_sources, whose default (namespace) is what every row N-1 writes means. N stores a source whoseaccessismirroredwithkindempty, so a build without items finds no probe or credential for it. N serves none either: its sync pass skips one (rule 54’s build syncs a mirroredgdriveone into the mirror), its retire pass leaves its delete request alone, andcheck,rotate,update,sync_nowand anaddof a removed one’s name refuse withSOURCE_NOT_SERVED. So beside the build that writes mirrors, N never syncs a mirrored source into the named namespace or orphans its documents.On N, each namespace erasure step reads the registry, which keeps every mirror the named namespaces have had, so every run of the step, a resume’s or a replay’s included, covers them: a scope that holds a mirror is erased as a purge, in one pass over the named namespaces and their mirrors, and the census counts it the same way. The last Loom step,
knowledge_mirrors.forget, writes the mirrorsdeleted, and a live one keeps the name in use. N-1 runs no mirror pass and counts no registry row; N writes no mirror, so N-1 meets an empty registry, and the build that writes mirrors (rule 54) runs beside N, which reads them.tests/unit/test_knowledge_mirrors.pyand the mirrored cases intests/unit/test_knowledge_sources.pyandtests/unit/test_knowledge_source_sync.pycheck N’s side. -
A document’s permission list (migration
099_document_access_lists.sql): N addsloom.documents.access_json, the permission list of a mirrored document (document sources); under the Postgres authority it rides in the head’srow, with no Postgres migration. No writer in N passes a list, and the row builder carries none forward: every row N builds, a tombstone included, has the column empty, so it readsunreadand admits nobody; an erasure purges a mirror’s rows outright (rule 52). N-1 never meets a list: N writes none, and the first build that writes one runs beside N, never beside N-1, which matters because N-1 would carry a list through its rewrites (rule 2). No op’s input or output has the field.tests/unit/test_access_list.pyandtests/unit/test_document_row_builder.pycheck N’s side. -
Mirrored folder sources sync (migration
100_knowledge_source_listings.sql): N’s sync worker runs a mirroredgdrivesource, while its ops still refuse one (SOURCE_NOT_SERVED, rule 52). Each turn lists every file with its permissions, readingpermissions.listfor a file the listing leaves them out of, and writes into the namespace’s mirror (made under the namespace’s sources lease when it has none) at<address>#<source id>, taggedsource-id:<id>. Once the turn starts, it brings every stored copy’s list to the store’s (only narrowing a copy behind its file’s revision), or tounreadfor a copy a whole listing does not have at its address (bounded as rule 55 says); then it fetches and writes each new or changed file with its list, and at the full interval retires a copy whose file is gone, with no audit row, since the audit log outlives an erasure. The listing’s start goes toloom.knowledge_source_listingsonce every stored copy’s list is written. No request can add a mirrored source yet, so N meets none it did not write, and N runs every mirroredgdrivesource as one shared service account: the build that adds sources of another credential mode stores them in a formmirror_syncedrejects while N may run. By rule 52, N-1 serves no mirrored source, and it reads no listing row; an N-1 namespace erasure has no listings step, so it leaves the mirror’s listing rows (source names and times, no memory content), which the next erasure of the namespace on N removes, since its scope holds every mirror the namespace has had. N counts a document as searchable when its chunks are those of its version or of its revision, which a row-only rewrite (a new list, a metadata edit) keeps, so such a rewrite stays until the file changes; N-1 checks the version alone, so after a metadata edit it fetches and writes the file again, as it does today.tests/unit/test_knowledge_source_sync_mirror.pyandtests/unit/test_knowledge_source_revisions_sql.pycheck N’s side. -
The mirrored sources’ share of a pass: N runs the mirrored sources first, until a third of the time the pass starts work in has passed (a quarter of
knowledge_source_sync_max_pass_seconds) and under their own document bound (LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS), the one synced longest ago first, a sync request included; the namespace sources then have the rest of the pass under their own bound. A mirrored source’s turn keeps its own state under its mirror and source id, leaving only its request, status and time in the namespace row. Its model calls are attributed to the mirror. It reads the lists on from a cursor in that state, in at most half of what is left of the mirrors’ share when its turn reaches them, so its new and changed files still get time, and a turn whose list reads stop there is still complete, so it retires gone files and keeps the full interval; a cycle’s listing row carries the start of the turn that began it. N-1 runs a mirrored source under the whole pass’s bounds and keeps its whole state in the namespace row; N reads neither N-1’s fields there nor N-1 the mirror row, so a mixed fleet reads some lists again and retries some waits, and the next N turn drops N-1’s fields from the namespace row. No request can add a mirrored source yet.tests/unit/test_knowledge_source_mirror_pass.pychecks N’s side. -
Mirror erasure receipts (migrations
101_erasure_mirror_mark.sqland Postgres013_erasure_mirror_mark.sql): N adds a Booleanmirror_marked, false for ordinary receipts, to both erasure ledgers. Before a namespace step or its failure census reaches mirrors, N persists true. Request replies and namespace audit rows then carry codes only; status and list expose counts only to a freshly authorized flat platform admin. Recovery, reverify resume and restore replay preserve the mark; cross-store copies promote it without replacing terminal state. A replay promotion changes no receipt or pending fence. Marked versions reserve bit 63 of the UInt64 version, so even a later unmarked checkpoint cannot replace one; timestamps order marked writes. N-1 ignores the column and cannot advance a marked saga. Mirrored sources remain refused until every process implements this protection and the later read and session protections. Before N-1 may serve after enablement, the enablement and rollback procedure must settle outstanding marked sagas, remove mirrored content and scrub marked receipts and audit counts while preserving their mark and version priority; a schema downgrade alone provides no privacy protection. -
Stopped mirrored sources retain their retirement requests: N writes a
knowledge_source_retire:<source id>key under the mirror incode_index_statebefore inactivating the registration. Its always-onknowledge_source_retireworker deletes that source id’s owned copies under its own cursor, deadline and document bound, even after the name is reused. N-1 ignores those queue keys and cannot serve the mirrored registration; its ordinary cleanup skips it. Queue rows remain until N finishes, and namespace erasure removes them with the mirror. Mirrored registration remains unavailable.tests/unit/test_knowledge_source_retirements.pychecks name replacement, bounded progress and cursor persistence. -
Mirrored work keeps diagnostics inside its namespace: N suppresses shared logs, spans, metrics and work figures while syncing or retiring mirrored content. Its worker summaries carry fixed codes and no elapsed time or document figures; billing rows stay in the mirror namespace. N-1 has no equivalent boundary, so mirrored source creation remains refused until every process implements the read, session and enablement protections. Before N-1 can serve after enablement, stop writers, settle outstanding inserts and sagas, scrub marked receipts and audit figures, and erase every mirror store, including
agent_events,agent_events_privateandagent_cost_rollup_daily, before clearing the enablement latch. Stopping a client after its timeout does not prove a sent insert has completed. -
Personal-agent reader claims and verified directory evidence (migration
103_sso_domain_evidence.sql): N adds empty-defaulthosted_domainandissuerString columns tosso_links. Direct verified sign-in writes them only for a fresh active passwordless account under its ownership lock, after the SCIM check. N-1 ignores them and its ordinary link writes clear them to defaults; missing evidence denies domain access. No backfill infers a domain from an email. N accepts the optional Booleanreads_as_creatorclaim, absent meaning false. Only the primary first-party sign-in routePOST /v1/me/personal_tokens.createand the assistant’s one-answer bridge set it; ordinary delegation omits it. Saved personal tokens use the existing Stringkindcolumn with valuepersonal_agent, which N-1 lists and preserves during revocation. N-1 ignores the claim and has no mirrored document leg, so it grants no mirrored access. Mirrored sources remain unavailable until every process implements the document and session protections.tests/unit/test_mirror_reader_identity.pychecks the claim and delegation boundaries. -
Checked mirror reads use source-scoped group evidence (migration
104_knowledge_group_members.sql): N reads each membership snapshot by org, mirror, source id and group, and admits only a nondeleted snapshot withinLOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS. An empty table admits no group. N-1 ignores the table; namespace erasure removes its rows with the mirror. Mirrored source creation remains refused until every process implements the read, marked request, session and enablement protections. -
Coarse permission keys and private lexical statistics (migration
105_mirror_access_candidates.sql):documents.access_keysis materialized from the existing permission JSON; N-1 writers need no new field. N still checks the strict current list, and Postgres heads do not depend on published keys. Each chunk-frequency view changes its query in place, retaining the input, target and output columns. Both views exclude mirror namespaces for either build. Mirror creation remains refused until the read, request, session and deployment protections are complete. -
User cost caps and private budget warnings (migration
107_user_cost_budgets.sql): activate the global and namespace user caps and alert inboxes only after every serving build supports the migration; keep them disabled and do not rely on user caps while any build is older or unknown. N-1 cannot enforce user caps, filters no private warning events, and has no user-erasure step for those events. Before the first N-1 process starts, the operator stops all warning producers for maintenance, settles scheduled and uncertain writes and their writer leases, deletes onlybudget.warningtask events, and verifies completed deletion and zero warning rows on every storage replica. There is no interval in which N-1 serves with private warnings stored. Save policy before rollback: N-1 namespace writes omit the user-cap and alert-inbox columns, so their defaults replace the saved values. Restore and verify the global default and namespace policies only after every serving build supports the migration again. The activation and rollback procedure defines the maintenance boundary; the warning fleet check alone does not make a downgrade safe.
What checks them
scripts/gates/mixed_version.py (make test-mixed-version, the mixed-version CI job,
once with a Keeper and once without) boots the base build on an empty ClickHouse, then
this build, which applies its migrations, then the base again on the newer schema, and a
third process of this build with a changed dashboard bundle and one no-op migration more
(N+1). It checks:
- each build answers ready after the cross-boot;
- memories, a run begun on one build and completed on the other, and a conflict verdict through the other build’s version write read back whole across builds;
- a column added as N+1’s migration would survives a version write by each build, made as soon as every process has acknowledged the column (rule 2);
- a build on a ledger newer than itself logs the WARNING;
- every asset each dashboard page names loads from a replica of the other build (rule 9);
- with every process on
LOOM_BACKGROUND_WORKERS=auto, exactly one process runs the workers, and it is N+1: N, which took the lease first, hands it over to the higher schema version (rule 6).
It does not check new enum values (rule 3), wire formats (rule 4), signing keys other
than default (rule 5), new revocation keys (rule 18), an /assistant history entry
(rule 46), or a process killed mid-migration. It skips with exit 0 where docker is
unreachable; CI passes --require, which fails there instead.
Analyst event storage cutover
The CHECK proof requires ClickHouse with system.constraints, as supplied by the
pinned deployment release. Upgrade the database before provisioning the event catalog;
a server without that metadata refuses the cutover.
agent_events retains ordinary event rows, its columns and the analyst’s signed,
namespace-scoped grant. Both builds can insert and query ordinary events while the
entity catalog advances to version 5. The ordinary_events_only CHECK rejects marked
rows and reserved mirror namespaces. Ordinary namespaces matching the reserved
mirror spelling are refused too; they cannot be used as an alternate private sink.
Marked requests and mirror workers write agent_events_private, which has no analyst
grant. The typed, signed analysis_agent_events view unions both stores and masks
private fields before predicates run. Trusted cost, budget and history-policy reads
also cover both stores. A row policy on a shared physical table cannot replace this
split: table metadata could disclose private rows even when a SELECT returns none.
Before granting access, provisioning verifies the exact CHECK and both table UUIDs on every configured writer, freezes the identities of all running INSERTs, waits for that finite set to finish, and counts forbidden ordinary rows. Internal async flushes participate even after their submitting client disconnects. Queued requests that have not prepared a pipeline encounter the installed CHECK when they execute. An ALTER alone does not settle old pipelines. Missing writers, changed topology, unknown metadata, a deadline or any surviving forbidden row refuses provisioning; existing rows remain untouched. Writers continue accepting valid ordinary inserts.
Negative observations use a checked administrative HTTP session on each writer.
The configured service origin is included; LOOM_CH_ANALYSIS_OBSERVER_URLS adds
explicit trusted HTTP origins using the same existing service credentials. Every
writer UUID needs exactly one route. Missing routes, aliases to the same backend,
redirects, backend changes and incomplete replies refuse. Routes must preserve
non-replaying POST semantics, including any intermediary. HTTPS cannot downgrade.
The proof never discovers another credential or derives HTTP ports from native
cluster addresses.
Each writer supplies its complete native SHOW CREATE ROW POLICIES catalog. A
SELECT from system.row_policies cannot establish visibility because policies can
filter that catalog too. Policies on process, topology or CHECK metadata sources,
database-wide policies and unknown event policies refuse. Generated analyst event
policies require their matching unrestricted policy for other users: an analyst-only
policy can implicitly hide every row from another user. The observer cannot be the
analyst user or hold its enabled role. Catalog, backend, principal and table identities
remain fixed through the drain and census. The complete proof has a ten-second
deadline; exact owned sessions close concurrently with one bounded cleanup interval.
Run loom ch-setup --apply-analyst-sql with explicitly configured existing
access-management credentials to apply only the analyst boundary. It uses the same
live proof and lease without rebuilding the reader user. --print-analyst-sql refuses
with a nonzero result and prints no grants or secrets: an unchecked SQL file cannot
enforce the native policy-catalog observation. General --print-sql still prints
database and reader preparation with a password placeholder.
Mirrored creation remains disabled until the fleet, read and session protections
and enablement procedure are complete. Before an older build serves after enablement,
the rollback procedure must settle writes and erase every mirror store, including
agent_events_private; restoring a grant or downgrading the schema is insufficient.
Analyst probes across catalog versions
The analyst probe checks its signed key canary before reading the stored entity version. When a newer build owns the entities, the older probe leaves that build’s catalog and grant checks to it. It does not query retired tables or overwrite the newer entities. A failed key canary still refuses analysis, regardless of the version stamp.
Deploy this probe ordering across the fleet before a later catalog change revokes a raw
relation. Builds with the earlier probe order can refuse insights.query after that
cutover because they probe the retired relation before checking the entity version.
Statements naming a removed relation still fail its grant check; trusting the newer
catalog does not restore a retired grant.
Connected-account registrations
The account registry is additive and is read only by builds that understand connected
sources. Such a source stores an empty legacy kind and credential, with its provider
kind in source_kind; an older worker cannot interpret it as a shared credential. Its
owner_subject and connection_id bind every credential checkout and publication to
the canonical person’s current account generation. The source id also distinguishes
its mirror copies from another registration using the same item address.
Keep connected and mirrored creation disabled until the fleet’s mirror read and session
protections are complete. A rollback or restore must settle source writers and perform
the mirror cleanup described in document sources
before an older build serves. Account rows restored without their matching Keeper
fence refuse credential checkout. That refusal does not prove physical erasure: restored
provider identities, sealed credentials and source history still need canonical-owner
cleanup and its census. Namespace or end-user erasure cannot substitute for that exact
owner key. loom reconcile --apply ends connection authority before publication for
either restored store. On a ClickHouse-only deployment, stop all serving processes,
restore the configured stores, then run python dev/restore_connected_accounts.py first
without flags and then with --apply. It refuses recent worker activity and never
remaps a scratch restore target into the deployment client. Both paths fence all
historical connection keys and verify physical owner-history deletion. Complete the
separate mirror-content cleanup before serving; this account procedure does not replace it.