hacking.postgres.tv
./menu

postgres=# select * from sessions where topic = 'monitoring';

#monitoring

6 related sessions.

· 93:47 · youtube ↗

Wait-event tracing

Independent benchmarks, observer effect, coverage, and patch restructuring.

Dmitry Fomin presented a seven-patch wait-event tracing series, split for review but intended to provide complete transition timing and query attribution. The group designed an independent benchmark ladder covering vanilla, compiled-disabled, stats, and full trace modes; corrected pgbench workloads; and distinguished static overhead from observer effect. Preliminary runs showed no clear slowdown, but they added pg_stat_statements, wait-event counts, repeated rounds, raw results, and independent review before submission.

[timecodes · 8]
  1. Why sampled wait events miss short transitions
  2. Seven-patch tracing series and review structure
  3. Planning an independent overhead benchmark
  4. Designing high-transition pgbench workloads
  5. Five measurement levels and observer effect
  6. Stats versus full trace capture modes
  7. Correcting workload duration, rate limits, and logging
  8. Preliminary results and adding pg_stat_statements
#wait events#monitoring@Nikolay Samokhvalov@Kirk Wolak@Dmitry Fomin
benchmark #48coverage #49PR #2benchmark briefCF #6984
active

direct mapping
· 60:51 · youtube ↗

Precise wait-event tracing, part 2

Low-overhead per-event timing and trace capture.

The group compared pg_wait_tracer's external eBPF hardware-watchpoint tracing with Dmitri Fomin's new Postgres patch, which records wait transitions into per-backend lock-free ring buffers. They designed a shared-buffer-thrashing benchmark to stress transition overhead, explored transition, concurrency, and query-profile visualizations, and discussed eventual EXPLAIN integration. Preliminary claims of roughly 2% patched overhead were treated as unverified; the agreed next steps were finish benchmarks, publish the tool, review the code, and prepare a pgsql-hackers proposal.

[timecodes · 8]
  1. How precise wait-event tracing works
  2. ASH dashboard, AAS, and wait hierarchy
  3. Why tracing needs a Postgres patch
  4. Designing a transition-heavy overhead benchmark
  5. Per-backend lock-free ring-buffer design
  6. Transition paths and concurrency analysis
  7. Toward wait events in EXPLAIN
  8. Publishing and presenting the patch upstream
#wait events#monitoring@Nikolay Samokhvalov@Kirk Wolak@Dmitry Fomin
PR #2-hackers v1CF #6984
active

direct mapping
· 60:12 · youtube ↗

Wait-event tracing, part 1

pg_wait_tracer, core timing, USDT probes, and observer-effect trade-offs.

Dmitri Fomin demonstrated pg_wait_tracer, an eBPF-based tool that captures every Postgres wait-event transition with microsecond-to-nanosecond timestamps, replayable trace files, latency heatmaps, and query drill-down. The motivating case was intermittent primary-key lookups jumping from milliseconds to a one-second timeout. The group identified inlined pgstat_report_wait_start/end calls as the obstacle to ordinary probes, produced a small USDT-probe prototype, and planned worst-case overhead benchmarks before proposing always-on core instrumentation.

[timecodes · 8]
  1. ASH, wait-event gaps, and pg_wait_tracer
  2. The intermittent one-second query mystery
  3. Sampling versus complete wait tracing
  4. Finding a trace path with hardware watchpoints
  5. Live ASH-style trace visualization
  6. Latency heatmaps and source-code attribution
  7. Transition chains for root-cause analysis
  8. USDT probes, core patch, and benchmark plan
#wait events#monitoring@Nikolay Samokhvalov@Kirk Wolak@Dmitry Fomin
USDT PR #18timing PR #1-hackers
active

direct mapping
· 87:51 · youtube ↗

Hacking Postgres

Discuss wait-event coverage gaps; follow-up work added COPY file and PROGRAM waits upstream.

Starting from an AI-assisted audit of missing wait-event coverage, the group distinguished genuine waits from CPU work and chose COPY as a testable first target. They reproduced an unreported wait in COPY FROM PROGRAM, prototyped events for program input and output, and used pg_stat_activity and pg_wait_sampling to validate them. The investigation also exposed missing coverage for ordinary COPY file reads and writes, expanding the planned patch to all four paths; the file/program work later reached core.

[timecodes · 8]
  1. AI-assisted audit of missing wait events
  2. Review discipline and excluding CPU work
  3. Choosing COPY FROM PROGRAM as the first target
  4. Reproducing the unreported external-program wait
  5. Prototyping COPY PROGRAM wait events
  6. Testing COPY TO PROGRAM and separating code paths
  7. Discovering ordinary COPY file I/O gaps
  8. Validating coverage with pg_wait_sampling
#wait events#MultiXact#monitoring@Nikolay Samokhvalov@Andrey Borodin@Kirk Wolak
gap analysiscoverage PR #2COPY waits threadupstream commit
committed

direct mapping
· 81:56 · youtube ↗

Vibe Hacking with Dmitry Fomin

Wait-event observability and precise tracing ideas.

Dmitry Fomin presented his first core patch after tracing synchronous-replication latency spikes to blocking fdatasync calls in the standby WAL receiver, even under synchronous_commit=remote_write. His prototype batches flushes by byte threshold or timeout. Review uncovered unsafe defaults for stronger synchronous modes and suggested a separate flusher process as a cleaner long-term design. They rebased the old patch onto master, removed redundant unflushed-byte state, compiled it, and prepared it for pgsql-hackers discussion.

[timecodes · 8]
  1. Production SyncRep latency problem
  2. fdatasync stalls in remote_write mode
  3. Batching WAL flushes by bytes or timeout
  4. What synchronous replication actually guarantees
  5. Reviewing the new WAL-receiver settings
  6. Defaults break stronger synchronous-commit modes
  7. Alternative design with a separate WAL flusher
  8. Rebasing, reviewing, and preparing the patch
#wait events#monitoring@Nikolay Samokhvalov@Andrey Borodin@Dmitry Fomin
pg_wait_tracertiming PR #1
active

correlated mapping
· 63:53 · youtube ↗

Observer effect in pg_stat_statements and pg_stat_kcache

Missing and self-hidden I/O accounting in monitoring extensions.

The group reproduced an observer effect shared by pg_stat_statements and pg_stat_kcache. Reading their hash tables holds a shared lightweight lock while results are materialized; if the tuplestore spills beyond work_mem, the lock may cover temporary-file I/O, blocking registration of new queries. They prototyped conditional lock acquisition for pg_stat_kcache so a query can execute while its statistics entry is skipped. The workaround succeeded in the demonstration, but needed configuration, benchmarks, and confirmation that it addressed the production contention before submission.

[timecodes · 8]
  1. Reviewing the committed bounded psql watch
  2. Observer effects in statistics extensions
  3. pg_stat_kcache contention at high TPS
  4. Tracing the shared hash-table lock
  5. Tuplestore spill extends lock duration
  6. Reproducing blocked registration of new queries
  7. Best-effort tracking with conditional locking
  8. Prototype works, but the production cause remains uncertain
#monitoring@Nikolay Samokhvalov
pg_stat_kcache PR #39core PR #15
active

direct mapping