How burl stores a notebook

Reference for what a .burl file holds under its notes: the exact bytes burl hashes, the named roots that keep earlier states of the notebook alive, how long each kind of root lives, and what archiving removes. It is for a reader who wants to check burl's hashes with sqlite3, or understand why a file is the size it is. The merge itself, in general, is explained on its own page; this page covers only what is specific to burl.

The sample notebook

The sample below is three notes: Groceries with a child note Dairy, and Guided, which carries two lines of guidance (rules attached to a note for AI assistants). Its stable ids and sort keys are the ones burl generated when the sample was created, so the hashes on this page are the hashes burl stored for it (what was compared). Edit any field, and every table below recomputes. Change a single character of a line's text and watch which hashes move.

Reset to the real file's values

1. How burl hashes

Take the first line of Groceries. burl stores it as a block object: a fixed sequence of netstrings (each field written as its byte length, a colon, the bytes and a comma; spelled out below the table), and the object's hash is the SHA-256 of exactly those bytes.

A netstring is <byte length>:<bytes>,, with the length counted in UTF-8 bytes. A field that may be absent is written as -, or as + followed by a netstring when present. The stable id is inside the hash: two lines with the same text and different ids are different blocks, which is what lets a merge tell a retyped line from a moved one. The first field, burl-block-v1, is the object's tag: its kind and layout version, so no reader mistakes one kind for another. The block's hash is .

Symbols: Merkle.serialize_block, Merkle.digest, Merkle::BLOCK_TAG; the private Merkle.net and Merkle.opt write the netstrings and markers.

A note: a body, then edges

A note's lines are not inside the note's bytes. The note object stores its scalar fields and two counts, and the hashes of what it contains are kept as edges: the first edges are the guidance and remarks objects if present, then one edge per line, then one per child note. Choose a note to see its fields.

Symbols: Merkle.serialize_note, Merkle::NOTE_TAG (burl-note-v7), Merkle::ANNOTATED_NOTE_TAG (burl-note-v8, used only when the note has remarks), Merkle.note_edges. No row id and no timestamp is in any note, so two devices holding the same content compute the same hash.

The hash is taken over the body followed by each edge's hash, each written as a netstring of its 64 hex characters. Merkle.preimage builds that string. For this note it is:

and its SHA-256 is . A block has no edges, so a block's preimage is its body and nothing more. A note's hash changes when any line, any scalar field or any child changes, and that change carries up through every ancestor to the root.

The root

burl's notes form a forest, since many have no parent. The root object stands over the top-level notes, in sort-key order. Its body is the tag burl-tree-v3, a marker saying whether the file has pinned settings (an object that would be the first edge), and the number of top-level notes.

Root hash: . This is the value in merkle_roots under the name live. Symbols: Merkle.serialize_root, Merkle::ROOT_TAG.

The other object kinds

Every row of objects has a kind. Three are above (block, note and root); the rest hang off notes and the root.

Symbols: Merkle.serialize_guidance, Merkle.serialize_settings, Merkle.serialize_annotations, Merkle.serialize_annotation. Guidance lines are ordinary block objects, which is why the merge treats them like body lines.

Every object the sample produces

This is the objects table for the sample, which a real file holds as well. Hashes are shown in hex; the file stores them as 32-byte BLOBs, so queries use hex(hash) or X'…'.

2. Checking a hash yourself

Start from any hash in your own file. Get its kind and edges from the file, and the page recomputes the rest. This recipe works for every kind, with or without edges:

f=notebook.burl
h=<64 hex characters from objects.hash, via lower(hex(hash))>
{ sqlite3 -readonly "$f" "SELECT body FROM objects WHERE hash = X'$h'" | head -c -1
  sqlite3 -readonly "$f" "SELECT lower(hex(child_hash)) FROM object_edges
                          WHERE parent_hash = X'$h' ORDER BY seq" \
    | while read -r c; do printf '64:%s,' "$c"; done
} | sha256sum

The first command prints the body (head -c -1 removes the newline sqlite3 adds). The second prints each edge in order, and the loop writes each as a 64-byte netstring. The output must equal $h. To list candidates:

sqlite3 -readonly notebook.burl "SELECT lower(hex(hash)), kind, size FROM objects LIMIT 10"
sqlite3 -readonly notebook.burl "SELECT name, kind, lower(hex(hash)) FROM merkle_roots ORDER BY created_at"

Or paste a body and its edges here. The page hashes them the same way:

SHA-256 of body and edges:

What was compared against a real file

The 11 objects reachable from the live root of that real file (6 blocks, 3 notes, 1 guidance object, 1 root) are the same 11 hashes this page computes from the sample as it loads: these match a real file. Not compared with a file: remarks (the annotation objects and the burl-note-v8 tag), pinned settings and attachment lists, which the real file did not contain. Their layouts follow the source, and the page marks no hash for them as verified.

3. Root kinds and their lifecycles

Objects are content-addressed and never edited in place; a change writes new objects, and garbage collection removes the ones nothing reaches. What keeps an old state of the notebook readable is a name in the table merkle_roots(name, hash, kind, reason, created_at). Garbage collection keeps an object only if some row of that table reaches it, or it is the base, mine or theirs side (the shared earlier state, your version and the imported copy's version) of an unresolved merge conflict. The four kinds differ in when the row is written and when it goes.

A pin has three families. A fork point, fork:<copy id>, is written when a copy is published. A merge base and the copy's tree, import:<id>:base and import:<id>:theirs, are written when an import leaves conflicts queued. A record, import:<id>:consumed, is written for each discovered source an import drains. A discovered source is a copy that was not published from this notebook, so burl finds the shared earlier state by comparing the two files.

The operation that uses a pin removes it. A fork pin advances with each import from its copy. The base and theirs pins go when the last conflict of that import is resolved. A consumed record is removed after 365 days (CONSUMED_KEEP_DAYS).

Reverting to a checkpoint exists in the code as Checkout.revert! in backend/src/burl/checkout.cr, which retains a history root for the tree it replaces. The app's windows, the burl-backend command line and the MCP server do not call it; only the specs and a playground script do. A user of burl cannot revert to a checkpoint today, and Ctrl/Cmd+Z undo is a separate mechanism.

What thinning keeps, and the disk it saves

burl writes a checkpoint on every save that changes something. The note you are typing in autosaves 1.5 seconds after you stop, or every 15 seconds while you keep going, so a busy day is thousands of checkpoints. Kept forever, they would grow the file without limit. Thinning keeps every checkpoint for 48 hours, then the last one of each day for a year, then the last one of each month for ten years, and none after that.

Saves a day: (5,760 is one every 15 seconds, all day)   Days of use:

Each checkpoint is counted at about 3.9 KB. That is what one save that rewrites a line of a 40-line note added to a vacuumed file, measured with burl 1.64.2: the new line, the new note and the new root, plus the note's 41 edge rows. A longer note costs more per save, because the note lists every line. A kept checkpoint can also hold every line that changed since the one kept before it, so read the thinned figure as a floor. The notebook's current text is not counted.

4. Archiving

Archive a note and burl moves it under the Archive section, sets its archived field to 1, and then calls Merkle#squash_note! (from Archive.archive!). Because archived is a field of the note object, the note's hash changes, and so does every ancestor's. Tick archived on a note above to watch it.

squash_note! deletes the note's earlier note objects, and only those that no named root and no unresolved conflict reaches. It deletes nothing else. Kept:

• Every earlier version that a checkpoint, a history row or a pin still reaches. A fork pin holds the merge base for a published copy, so archiving a note never makes a later import refuse for want of a base.
• Every block and guidance object. They are removed only by garbage collection, which runs in Database#compact! once the roots that reach them have thinned away.

The space an archived note held therefore comes back as its checkpoints age out, at the next compaction (Database#compact!, which also runs garbage collection) after that.

5. What a merge compares beyond the general explanation

The general explanation follows lines of a body. burl's import compares three more things.

• Scalar fields. SCALAR_FIELDS is parent, sort key, title, format, extension, icon, rich spans and archived; when both sides changed one differently, mine (your notebook's value) stays and theirs (the imported copy's) is queued.
• Guidance lines. merge_guidance merges them as a second set of lines with the same three tiers as the body.
• Remarks. plan_annotations and the code beside it merge remarks as a union by id, so a remark from either side lands and none is queued.

References

The sample notebook

A file made with the master build of burl-mcp (version 1.64.2); the comparison is described above.

Hash serializers

backend/src/burl/merkle.cr: Merkle.serialize_block, .serialize_note, .serialize_root, .serialize_guidance, .serialize_settings, .serialize_annotations, .serialize_annotation, .preimage, .note_edges.

Schema

backend/db/schema.sql: tables objects, object_edges, note_hashes, merkle_roots, merge_conflicts.

Root lifecycles

backend/src/burl/merkle.cr: Merkle#publish_root!, #checkpoint!, #pin!, #retain!, #prune_history!, #prune_checkpoints!, #squash_note!; backend/src/burl/database.cr: the HISTORY_KEEP_*, CHECKPOINT_* and CONSUMED_KEEP_DAYS constants, #compact!; backend/src/burl/archive.cr: Archive.archive!.

Merge extras

backend/src/burl/import.cr: SCALAR_FIELDS, merge_guidance, plan_annotations.

General merge explanation

Merging copies without conflicts, as a working model