hacking.postgres.tv
./menu

postgres=# select * from sessions where topic = 'wal';

#WAL

6 related sessions.

· 78:11 · youtube ↗

Agentic Hacking Postgres

End-to-end tests for archive-fed logical decoding and pause/resume behavior.

The session refined logical decoding from an archive-fed physical standby as a recoverable, production-independent pipeline: after losing the decoding node, a consumer can restore another standby at the acknowledged LSN and tolerate overlap. They reviewed the three-patch implementation and TAP test, found registration and naming issues, debated lock structure, and judged the code ready for discussion after fixes. They then specified an end-to-end WAL-G test with catalog churn, paused consumption, catch-up, and a final marker.

[timecodes · 8]
  1. Recoverable decoding after a standby is lost
  2. Archive-fed WAL decoding without touching production
  3. Naming the feature: logical decoding from archive
  4. Reviewing the three-patch implementation
  5. TAP test registration and test readability
  6. Reviewing pause-on-conflict code and locking
  7. Patch readiness and concrete submission fixes
  8. Designing an end-to-end WAL-G recovery test
#logical decoding#recovery#WAL#AI-assisted hacking@Nikolay Samokhvalov@Kirk Wolak
test design #43
active

direct mapping
· 53:58 · youtube ↗

Logical replication from physical standbys on WAL shipping

Refine recovery pause behavior, crash safety, promotion, and auto-resume.

The group inspected two AI-produced patch branches for pausing archive recovery when replay would invalidate a logical slot. They reconstructed the patch series, reviewed synchronized-slot and promotion behavior, and clarified that dirty slot state is only persisted at restart points, weakening crash guarantees. The concrete conclusion was to favor correctness, consolidate the work into one reviewable three-patch series, preserve readable code over premature lock optimization, and target the July CommitFest after further testing.

[timecodes · 8]
  1. Returning to logical decoding on WAL-shipping standbys
  2. Recovering useful changes from a broken pull request
  3. Reviewing pause and auto-resume patches
  4. Conflict horizons and synchronized-slot behavior
  5. Dirty slot state and restart-point persistence
  6. Live transcript-assisted code investigation
  7. Restructuring the work into reviewable patches
  8. Crash-safety invariant and correctness-first decision
#logical decoding#recovery#WAL@Nikolay Samokhvalov@Andrey Borodin
PR #40
active

direct mapping
· 39:42 · youtube ↗

Logical decoding of archived WALs

Use an archive-fed standby as a logical-decoding engine.

The session began by testing whether Postgres 16's logical decoding on a standby could turn an archive-fed recovery instance into a detached decoding engine. The group concluded a full server is the practical first implementation because it supplies catalog snapshots and output plugins; the missing behavior is to pause replay when it would invalidate a slot and continue when consumption advances. They also identified broader uses: per-record WAL verification, recovery debugging, and filtered, PII-free staging streams.

[timecodes · 8]
  1. Why decode archived WAL away from production
  2. Using an archive-fed standby as the decoder
  3. Standalone WAL tool versus a full Postgres server
  4. Slot conflicts and the need to pause replay
  5. Choosing a recovery mode driven by consumption
  6. Function-based recovery targets
  7. Stepping through WAL for correctness testing
  8. Filtered, PII-free staging from archived WAL
#logical decoding#recovery#WAL@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
blueprint PR #24Sprint 0 #25PR #27
active

direct mapping
· 55:36 · youtube ↗

Hacking Postgres with Kirk, Andrey, Nik

Custom command tags and logical decoding from an archive-fed recovery server.

The workshop first revisited customizable command tags, confirming legacy behavior remains the default and its added branch should be negligible, then explored extensible client protocols and better psql completion. The main direction was extracting data from managed Postgres without burdening production: decode archived WAL into a logical stream. They compared a full recovery server with a standalone tool, concluding the latter would need schema state at a starting LSN, checkpoint/full-page-image handling, and careful treatment of file swaps such as pg_repack.

[timecodes · 8]
  1. Command-tag customization and performance feedback
  2. Ideas for an extensible client/server protocol
  3. Language-server completion for psql
  4. Parallelizing logical-replication initialization
  5. Keeping an independent copy outside managed Postgres
  6. Turning archived WAL into a logical stream
  7. Designing a standalone offline decoder
  8. Checkpoints, full-page images, and pg_repack caveats
#logical decoding#recovery#WAL#hacking process@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
no public artifact mapped
exploration

direct mapping
· 55:46 · youtube ↗

Continue with WAL compression

Continue the B-tree/WAL compression prototype.

After noting Kirk Wolak's committed psql ECHO_HIDDEN improvement and comparing Meson with Autotools, the session resumed a WAL full-page-image compression prototype. Earlier work grouped up to 32 images in one WAL record; Andrey began replacing per-page compression with a contiguous buffer, disabling hole stripping and restructuring record-data links. The code became tangled around rdata ownership and was left uncompilable. They agreed to finish offline, then benchmark batch sizes 1, 4, 16, and 32 for compression, CPU, memory, and record-size trade-offs.

[timecodes · 8]
  1. Kirk's first committed psql patch
  2. Meson versus Autotools build benchmarks
  3. Resuming batched WAL image compression
  4. How WAL record assembly handles page images
  5. Removing per-page compression and hole logic
  6. Allocating contiguous multi-page buffers
  7. Record-data links block the implementation
  8. Batch-size rationale and benchmark plan
#B-tree#WAL#compression@Nikolay Samokhvalov
-hackersCF #5621
stalled

correlated mapping
· 64:33 · youtube ↗

B-tree build and batching WAL records

Batch or compress WAL records during B-tree construction.

The group explored reducing WAL volume during B-tree index creation by batching full-page images, which could later let compressors reuse context across similar pages. After tracing the sorted B-tree build and its page-flush routine, they changed it to postpone writes and emit roughly 31 pages in one WAL record. The build and basic tests worked, but compression itself was not implemented, replication coverage was uncertain, and macOS library and overheating problems prevented the intended size and performance comparison.

[timecodes · 8]
  1. Patch status and PostgreSQL CI infrastructure
  2. Why full-page images limit WAL compression
  3. Reusing compression context across B-tree pages
  4. Measuring current multi-page WAL records
  5. Tracing the sorted B-tree build path
  6. Designing deferred page flushing and batching
  7. Implementing the multi-page WAL batch
  8. Successful 31-page records and missing benchmarks
#B-tree#WAL#compression@Nikolay Samokhvalov
-hackersCF #5621
stalled

correlated mapping