fix(db): avoid long flush stall on restart#211
Open
EddieHouston wants to merge 2 commits intoBlockstream:new-indexfrom
Open
fix(db): avoid long flush stall on restart#211EddieHouston wants to merge 2 commits intoBlockstream:new-indexfrom
EddieHouston wants to merge 2 commits intoBlockstream:new-indexfrom
Conversation
d102699 to
bef02e3
Compare
philippem
reviewed
Apr 27, 2026
philippem
reviewed
Apr 27, 2026
philippem
reviewed
Apr 27, 2026
enable_auto_compaction() was lowering level0_stop_writes_trigger from the bulk-load value (512) to the RocksDB default (36) on every update() call. At DB open the bulk-load triggers are applied by DB::open, so on any restart L0 can legitimately hold more than 36 files. When the first post-restart update() called enable_auto_compaction(), the trigger tightening instantly put the DB into pre-flush stall territory, and the end-of-batch db.flush() that follows parked inside WaitUntilFlushWouldNotStallWrites waiting for background compaction to bring L0 below 36. On production testnet this reliably cost 77 minutes of indexer freeze per restart (verified by 'Manual flush start' → 'Manual flush finished' in the RocksDB LOG). The actual memtable flush took 62 ms once unblocked; the rest was wait. Split enable_auto_compaction() into the minimal flag-flip and a new apply_steady_state_triggers() that holds the L0 trigger / pending-bytes- limit reset. Invoke the latter exactly once per DB lifetime, inside the F-sentinel gate in start_auto_compactions(), immediately after full_compaction() has drained L0. On DBs where F is already set (steady- state restart), triggers stay at bulk-load values — the comment in DB::open already argues that configuration is fine for steady-state reads given the prefix bloom filters.
Open the DB with RocksDB-default L0 triggers (4/20/36) and only widen to bulk-load values (64/256/512) when the full-compaction sentinel 'F' is absent. Previously every restart used bulk-load triggers permanently, leaving read amplification unnecessarily high after initial sync. Add info-level logging to DB::open and start_auto_compactions so operators can see which trigger profile is active and whether full compaction runs or is skipped.
1ebdfd6 to
5e80660
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix freeze that can occur on restart of a mature electrs DB built from
91b883aor later,and default to steady-state compaction triggers so restarts after initial sync get low read
amplification immediately.
enable_auto_compaction()was tighteninglevel0_stop_writes_triggerfrom the bulk-loadvalue (512) to the RocksDB default (36) on every
update()call, not just once.DB::openapplied bulk-load triggers at startup, so after any restart L0 can legitimately hold more
than 36 files. The reset instantly put the DB into pre-flush stall territory, and the
end-of-batch
db.flush()parked insideWaitUntilFlushWouldNotStallWritesuntil backgroundcompaction brought L0 below 36.
In testnet this caused over 1 hour of indexer freeze on restart.
Fix
DB::open()now defaults to steady-state triggers (RocksDB defaults: 4/20/36). Afteropening, it checks the
Fsentinel key and callsapply_bulk_load_triggers()only whenFis absent (initial sync not yet complete). On normal restarts (F present), the DBstays on tight triggers from the start.
apply_bulk_load_triggers()— new method that widens L0 triggers (64/256/512) anddisables pending-compaction-bytes stalls for initial sync.
apply_steady_state_triggers()— restores RocksDB defaults afterfull_compaction()drains L0. Called once per DB lifetime inside the
Fsentinel gate.enable_auto_compaction()— minimal flag set (disable_auto_compactions=false), safeto call on every
update().Operator logging —
DB::openandstart_auto_compactionsnow log at info level whichtrigger profile is active and whether full compaction runs or is skipped.
Test plan
cargo checkcleancargo test --lib— 8/8 pass, includingnew_index::db::tests::*cargo test --test electrum— 4/4 passcargo test --test rest— 22/22 passManual flush start→Manual flush finishedin RocksDB LOG are within ms