hacking.postgres.tv
./menu

postgres@hacking:~$ cat projects/btree-page-merge

B-tree bloat removal (page merge)

Salma El-Sayed’s implementation, mentored by Kirk Wolak, merges sparse siblings and reclaims merged-away tombstone pages without rebuilding the entire index.

active[current -hackers thread ↗][working branch ↗][tombstone cleanup commit ↗][GSoC idea ↗]

progress

46%

blocker: The active prototype still needs concurrency, backward-scan, VACUUM, WAL/recovery, and crash-safety review before CommitFest.

next

  • Review invariants and locking with the current prototype
  • Expand correctness, concurrency, and recovery tests
  • Shape the branch into a reviewable patch series

sessions

7
· 77:12 · youtube ↗

Agent hacking Postgres

Agent-assisted patch work around the wait-event timing submission.

The group reviewed Kirk Wolak's pg_dump patch for keeping loadable placeholders when table data is excluded, grounding the discussion in his large-table restore workflow. They debated a positive interface, filter-file actions, arbitrary dump queries, PII transformation, and referential-integrity risks, then posted supportive review feedback to pgsql-hackers. The final segment examined Salma El-Sayed's B-tree leaf-page merge design, including merged-page markers, scan correctness, vacuum cleanup, and review plans.

[timecodes · 8]
  1. Postgres 19 development opens and patches await review
  2. Kirk introduces the pg_dump placeholder patch
  3. Large-table dump and one-command restore use case
  4. Reviewing the option name and user interface
  5. Using AI to review design and mailing-list history
  6. Parallel chunks for restoring one large table
  7. Custom dump queries, filtering, and PII transformation
  8. B-tree leaf-page merge design and scan safety
#wait events#B-tree#AI-assisted hacking#hacking process@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
-hackersCF #6984
active

correlated mapping
· 73:44 · youtube ↗

Vibe Hacking Postgres

Prototype merging sparse B-tree leaf pages and audit its correctness failures.

The workshop focused on a prototype for reclaiming B-tree bloat by merging adjacent sparse leaf pages. Kirk proposed an incremental, access-exclusive-lock design that first merges only one pair, prioritizing correctness of forward and backward scans. The pgindex_reclaim extension identified merge candidates, but execution crashed. Inspection showed incomplete parent-page handling and incorrect WAL/critical-section logic. The group chose to stop trusting the generated merge code, add a reproducible regression test, study related upstream page-deletion work, and restart from the smallest verifiable merge.

[timecodes · 8]
  1. Why merge sparse B-tree pages
  2. Forward and backward scan constraints
  3. Smallest verifiable merge design
  4. How the approach differs from pg_squeeze
  5. Building and running pgindex_reclaim
  6. Candidate pages found, execution crashes
  7. Critical sections, WAL, and parent updates
  8. Regression test and restart plan
#B-tree#AI-assisted hacking@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
no public artifact mapped
exploration

direct mapping
· 70:24 · youtube ↗

Vibe Hacking Postgres

Review B-tree merge blockers and prototype commit-LSN logging for DROP TABLE.

After reviewing UUIDv7 and amcheck support for GIN in Postgres 18, the group discussed responsible AI-assisted hacking and briefly parked the B-tree merge work over unresolved scan concurrency. Most of the session revisited recovery from accidental DROP TABLE: they asked an agent to extend transaction callbacks with the commit LSN and build a logging extension. The prototype found table names through object-access hooks, but cleanup timing and an invalid commit LSN prevented a working result, leaving the design unfinished.

[timecodes · 8]
  1. Postgres 18 outcomes: UUIDv7 and GIN checks
  2. Responsible AI-assisted Postgres hacking
  3. Why the B-tree merge prototype was paused
  4. Revisiting a recovery landmark for DROP TABLE
  5. Prompting a callback and extension prototype
  6. Building and loading the DROP TABLE extension
  7. Combining object-access and commit callbacks
  8. Recovery-event catalogs and the invalid-LSN blocker
#recovery#B-tree#hacking process@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
no public artifact mapped
exploration

direct mapping
· 51:42 · youtube ↗

Vibe Hacking Postgres

Test B-tree page merge against concurrent scans and expose correctness failures.

The group assessed the B-tree page-merge prototype after community feedback. They agreed benchmarks should emphasize extra buffer I/O, cache pollution, WAL, and sparse-page workloads rather than lookup latency alone. A new injection-point test then demonstrated the critical flaw: a concurrent forward scan returned duplicates while a backward scan missed tuples after pages were merged. Several recovery schemes were examined, but repeated vacuums and concurrent inserts defeated the simple designs. The session ended with correctness and concurrency still unresolved.

[timecodes · 8]
  1. Community feedback and the benchmark requirement
  2. Why sparse pages hurt buffers, WAL, and vacuum
  3. Scoping useful page-merge benchmarks
  4. Injection-point test exposes incorrect scans
  5. Visualizing the old-tree and new-tree race
  6. Concurrent inserts and possible query aborts
  7. When scans can recover from merged pages
  8. Why limited merges and repeated vacuums still fail
#B-tree#hacking process@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
no public artifact mapped
exploration

direct mapping
· 97:08 · youtube ↗

Index bloat and B-tree page merge, part 3

Continue page-merge implementation and correctness work.

The group refined a minimal B-tree merge: move tuples from a very sparse leaf page into its right sibling, then reuse existing page-deletion machinery. They inspected page layouts, asked an agent to implement the change inside vacuum, and built a sparse index. After removing an assertion, one page was successfully emptied without a crash. However, regression and foreign-key tests failed, and concurrent scans were not tested, so the result remained a potentially corrupt prototype rather than a valid patch.

[timecodes · 8]
  1. Recap: turning sparse leaf pages into deletable pages
  2. Comparing candidate leaf-page merge layouts
  3. Applying Lehman-Yao page-merge ideas to Postgres
  4. Minimal plan: move tuples, then reuse page deletion
  5. Inspecting leaf pages before and after vacuum
  6. Prompting the first vacuum-integrated implementation
  7. Building a sparse-index test for the prototype
  8. One merge succeeds, but regression tests fail
#B-tree@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
current -hackers threadworking branchtombstone cleanup
active

correlated mapping
· 75:39 · youtube ↗

Index bloat and B-tree page merge, part 2

Continue B-tree page-merge design and prototype.

The session established why sparse B-tree pages matter: they increase buffer churn, WAL full-page images, replica storage, and the cost of scans. The group then traced vacuum's existing half-dead page deletion and studied whether a merge could resemble a split. Moving tuples directly exposed a duplicate-read race for concurrent scans, implying a two-phase protocol that removes the parent downlink before relocation. By the end they identified split/finish-split as the code to reverse, but deferred implementation pending a sound scan protocol.

[timecodes · 8]
  1. Operational costs of sparse B-tree pages
  2. Why Postgres has reindexing but no page merge
  3. Can merge be shaped like an existing page split?
  4. Tracing vacuum and half-dead page deletion
  5. Moving tuples into a sibling before deletion
  6. Concurrent scans expose duplicate reads
  7. A two-phase merge and scan-awareness requirement
  8. Reversing finish-split becomes the next design step
#B-tree@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
-hackersGSoC idea
exploration

correlated mapping
· 84:02 · youtube ↗

Index bloat and B-tree page merge, part 1

Merge sparse sibling B-tree pages to reduce bloat.

This B-tree page-merge session moved from automated REINDEX toward fixing the cause of persistent bloat. The group defined the target as partially empty leaf pages that vacuum cannot reclaim, then had an agent build a logical leaf-page scan that reports adjacent pages fitting into one block. On a randomized 90%-deleted workload it found widespread merge opportunities. They did not implement merging: backward-scan visibility, transaction horizons, WAL replay, lock coupling, and undetectable lightweight-lock deadlocks were identified as the core design problems.

[timecodes · 8]
  1. From automated REINDEX to B-tree page merge
  2. Defining the persistent bloat worth fixing
  3. Planning a logical scan for merge candidates
  4. Agent-assisted prototype of the merge checker
  5. Correcting physical scan to logical leaf order
  6. The logical adjacency scanner works
  7. Randomized deletion reveals widespread candidates
  8. Backward scans, WAL, and lock-coupling hazards
#B-tree@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
-hackers WIPlater patch
exploration

direct mapping