hacking.postgres.tv
./menu

postgres=# select * from projects where slug = 'wal-compression';

Whole-WAL-record compression

Compress or batch larger WAL records, originating in B-tree build experiments.

blocked[-hackers ↗][CF #5621 ↗]

current state

blocked

blocker: Closed CommitFest entry is waiting on the author.

next

  • Confirm author intent
  • Rebase the series
  • Produce current benchmarks

sessions

3
· 55:46 · youtube ↗

Continue with WAL compression

Continue the B-tree/WAL compression prototype.

After noting Kirk Wolak's committed psql ECHO_HIDDEN improvement and comparing Meson with Autotools, the session resumed a WAL full-page-image compression prototype. Earlier work grouped up to 32 images in one WAL record; Andrey began replacing per-page compression with a contiguous buffer, disabling hole stripping and restructuring record-data links. The code became tangled around rdata ownership and was left uncompilable. They agreed to finish offline, then benchmark batch sizes 1, 4, 16, and 32 for compression, CPU, memory, and record-size trade-offs.

[timecodes · 8]
  1. Kirk's first committed psql patch
  2. Meson versus Autotools build benchmarks
  3. Resuming batched WAL image compression
  4. How WAL record assembly handles page images
  5. Removing per-page compression and hole logic
  6. Allocating contiguous multi-page buffers
  7. Record-data links block the implementation
  8. Batch-size rationale and benchmark plan
#B-tree#WAL#compression@Nikolay Samokhvalov
-hackersCF #5621
stalled

correlated mapping
· 64:33 · youtube ↗

B-tree build and batching WAL records

Batch or compress WAL records during B-tree construction.

The group explored reducing WAL volume during B-tree index creation by batching full-page images, which could later let compressors reuse context across similar pages. After tracing the sorted B-tree build and its page-flush routine, they changed it to postpone writes and emit roughly 31 pages in one WAL record. The build and basic tests worked, but compression itself was not implemented, replication coverage was uncertain, and macOS library and overheating problems prevented the intended size and performance comparison.

[timecodes · 8]
  1. Patch status and PostgreSQL CI infrastructure
  2. Why full-page images limit WAL compression
  3. Reusing compression context across B-tree pages
  4. Measuring current multi-page WAL records
  5. Tracing the sorted B-tree build path
  6. Designing deferred page flushing and batching
  7. Implementing the multi-page WAL batch
  8. Successful 31-page records and missing benchmarks
#B-tree#WAL#compression@Nikolay Samokhvalov
-hackersCF #5621
stalled

correlated mapping
· 64:25 · youtube ↗

Compression improvements

Explore compression paths and possible improvements.

The session began by designing an overflow hash table for pg_stat_statements and pg_stat_kcache so new query IDs would not block behind a busy main table; races and merge semantics made it unsuitable for an immediate patch. Discussion then shifted to compression throughout Postgres. Per-column dictionaries were judged invasive, while reusing compression context during B-tree WAL logging looked tractable. They examined pglz and LZ4 stream constraints and concluded that B-tree builds should first batch pages like GiST; no code was written.

[timecodes · 8]
  1. Revisiting pg_stat_kcache lock contention
  2. An overflow hash table for delayed query IDs
  3. Duplicate entries and merge races
  4. From statistics contention to compression
  5. Column dictionaries and system-wide dictionaries
  6. Reusing context for B-tree full-page images
  7. Why appending to compressed streams is difficult
  8. B-tree WAL batching becomes the prerequisite
#compression@Nikolay Samokhvalov@Kirk Wolak
no public artifact mapped
exploration

correlated mapping