Skip to content

refactor(e2e): fast block build config in e2e tests#23340

Draft
spalladino wants to merge 14 commits into
merge-train/spartanfrom
palla/kill-anvil-test-watcher
Draft

refactor(e2e): fast block build config in e2e tests#23340
spalladino wants to merge 14 commits into
merge-train/spartanfrom
palla/kill-anvil-test-watcher

Conversation

@spalladino
Copy link
Copy Markdown
Contributor

@spalladino spalladino commented May 16, 2026

Motivation

AnvilTestWatcher polls every 200ms and silently mutates global state from three different RunningPromises — proving, dateProvider sync, and L1 time warping. The wiring difference between e2e and sandbox makes failures hard to diagnose. Concrete examples this work was motivated by:

  • A two-dateProvider-read race in Sequencer.work where targetSlot could compute to slot + 2 because syncDateProviderToL1IfBehind warped time between the two reads.
  • e2e_pruned_blocks regressions where finalized stalled at 0 because markAsProven and the archiver's finalized tag advanced out of order.
  • Blacklist huge-time-warp wedges in the pipelined publisher: the in-flight checkpoint references pre-warp state while the watcher keeps operating on post-warp state.

Deleting the watcher from e2e in favor of deterministic per-responsibility replacements gets us:

  • L1 time advances on a known cadence (anvil_setIntervalMining), no surprise warps.
  • Proven tip moves only when a real prover or EpochTestSettler advances it; no async loop racing the archiver.
  • dateProvider mutates only when a cheat code explicitly warps L1, atomically with the RPC.

The AnvilTestWatcher class stays in the tree so the sandbox / local-network entrypoint can keep using it. This PR only changes the e2e fixture wiring.

What's landed

Foundation

  • fix(ethereum): atomically sync TestDateProvider in cheat-code mine/warp pathsmine(), evmMine(), mineEmptyBlock() sync TestDateProvider after the RPC. Removes redundant manual setTime at three e2e call sites. setNextBlockTimestamp deliberately remains un-synced (it schedules but doesn't mine; sync happens on the next mine).
  • feat(e2e): add FAST_E2E_SETUP_OPTS preset — single-block fast-mode timing (3.75s execution budget per slot, 20s pipelined publishing deadline, 48s epoch). Plus a stdlib/timetable/index.test.ts unit test that verifies the derived numbers.
  • refactor(e2e): unconditional anvil interval mining + drop AnvilTestWatcherfixtures/setup.ts integration: drop automineL1Setup, switch to unconditional setIntervalMining(ethereumSlotDuration) for anvil after L1 deploy, default-skip watcher.start() for anvil chains (disableAnvilTestWatcher ?? isAnvilChain). Removes redundant setIsMarkingAsProven(false) calls from 8 sites.
  • refactor(e2e): use EpochTestSettler in place of CheckpointAutoProver — reuses the existing sandbox-path orchestrator (yarn-project/aztec/src/local-network/local-network.ts:213) instead of the bespoke per-checkpoint helper the PR originally added. Advances both the outbox hash and the proven tip once per completed epoch.
  • test(e2e): add e2e_fast_config smoke test — canary exercising three invariants: (a) chain advances under interval mining + pipelining, (b) 20 sequential dependent txs land in distinct blocks (single-block mode keeps batching off), (c) proven tip advances within two epochs (EpochTestSettler works).

Migrations (49 tests)

  • 10 first-batch single-sequencer tests (golden-path, no special timing semantics).
  • 22 second-batch single-sequencer tests.
  • e2e_amm (separate commit because it had a third setup() arg the batch-2 agent missed).
  • 14 tests reachable through TokenContractTest / NestedContractTest, which gained a opts?: Partial<SetupOptions> parameter (caller opts first, fixture invariants like metricsPort / fundSponsoredFPC / skipAccountDeployment override; mirrors the existing FeesTest.setup(opts) pattern).
  • e2e_2_pxes and e2e_lending_contract (initially deferred per the design plan, then re-verified safe).

Held back from this PR

  • e2e_cheat_codes — exercises setNextBlockTimestamp and mine() directly. The cheat-code audit deliberately left setNextBlockTimestamp un-synced; this test reads dateProvider.now() between schedule and mine, so it needs a hand-audit before migration.
  • BlacklistTokenContractTest (7 suites) — huge time warps wedge the pipelined publisher, and they also hit a separate L1ToL2MessagesNotReadyError bug (see "Out of scope" below). Blocked on those source fixes.

Explicitly out of scope (separate PRs)

These came up during the work and are tracked but not addressed here:

  • canProposeAt simulation regression under pipelining. The simulation drops lastArchiveRoot and the pending-tip slot override; a fix exists on origin/sp/publisher-simulation-cleanup and needs forward-port to merge-train/spartan. Blocks e2e_contract_updates, composed/web3signer/e2e_multi_validator_node_key_store, and HA governance opt-in.
  • AztecNodeService.simulatePublicCalls queries unsealed checkpoint. With inboxLag = 2, the simulator calls getL1ToL2Messages(proposedCheckpoint + 1) on a checkpoint that hasn't been sealed yet, raising L1ToL2MessagesNotReadyError. Pointer: yarn-project/aztec-node/src/aztec-node/server.ts:1519-1524. Preferred fix: a positive messagesSealed(checkpoint) check on the message store, gating the next-checkpoint-message-append on it. Blocks 7 blacklist suites + e2e_bot + e2e_avm_simulator + e2e_fees/{private_payments,failures}.
  • Publisher funder loop stops firing after cycle 1. RunningPromise cycle in yarn-project/ethereum/src/publisher_manager.ts:84-89. Hypothesis: shared viem client between an in-flight monitorTransaction and cycle-2's getSenderBalance hangs the loop. Blocks e2e_publisher_funding_multi.
  • Sequencer.invalidateInFlightCheckpointsBefore(slot) — needed if any test wants to do huge L1 warps that would orphan an in-flight checkpoint. Currently the in-flight job is silently kept and times out.

Status

Audited end-to-end by an LLM review pass + a /codex round-2 review on the final state of the branch. No CI-blocking issues flagged. Mark non-draft to trigger e2e CI. Consider ci-no-fail-fast for the first run so all failures surface at once.

@ludamad ludamad force-pushed the merge-train/spartan branch from 4af2626 to db4ec58 Compare May 16, 2026 19:07
…rp paths

mine(), evmMine(), and mineEmptyBlock() now call syncDateProvider() after mining
so TestDateProvider follows L1 time without callers needing to do it manually.
Remove now-redundant dateProvider.setTime() calls from three e2e test sites.
…tcher markAsProven loop

Adds CheckpointAutoProver, an event-driven helper that subscribes to the
sequencer's checkpoint-published event, waits for the local archiver to
promote the checkpoint (verified via getL2Tips + getBlocks), then calls
rollupCheatCodes.markAsProven(checkpointNumber). Replaces the periodic
polling loop in AnvilTestWatcher.
…tcher

Stop starting AnvilTestWatcher in e2e tests. Anvil now runs in interval mining
mode at ethereumSlotDuration after L1 deploy, so warpTimeIfNeeded and
syncDateProviderToL1IfBehind are unnecessary. The markAsProven loop is replaced
by CheckpointAutoProver when testOnlyAutoProveAfterPublish is true (default via
FAST_E2E_SETUP_OPTS).

AnvilTestWatcher class stays in the tree for the sandbox/local-network entrypoint.
The opt-in automineL1Setup option is removed (three callers cleaned up).
Canary for the AnvilTestWatcher-removal work in this PR. Exercises:
- chain advances under interval mining + pipelining
- 20 sequential dependent txs land in distinct blocks
- proven tip advances via CheckpointAutoProver

Uses FAST_E2E_SETUP_OPTS, the new preset added in this PR.
Missed in batch 2 because of its third argument (syncChainTip: 'checkpointed').
@spalladino spalladino force-pushed the palla/kill-anvil-test-watcher branch from 9208a79 to 1e675e8 Compare May 17, 2026 13:36
@spalladino spalladino added ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure ci-draft Run CI on draft PRs. labels May 17, 2026
@spalladino spalladino changed the title refactor(e2e): kill AnvilTestWatcher in e2e tests refactor(e2e): fast block build config in e2e tests May 17, 2026
…TUP_OPTS

Both already used PIPELINING_SETUP_OPTS. The plan flagged them as second-batch
out of caution; verified neither has hand-rolled cc.eth.warp/dateProvider.setTime
helpers that would conflict with the wave-1 cheat-code audit.
@spalladino spalladino force-pushed the palla/kill-anvil-test-watcher branch from 1e675e8 to d9922ef Compare May 17, 2026 13:50
Replace the PR's bespoke `CheckpointAutoProver` (140 lines + unit tests) with
the existing `EpochTestSettler`, which is already the sandbox's proven-tip
advancement path (`local-network.ts:213`).

`EpochTestSettler` polls `EpochMonitor` and, once an epoch is complete,
inserts the epoch out-hash into the outbox and calls
`RollupCheatCodes.markAsProven(lastCheckpoint)`. Reusing it kills three audit
findings in one go (the dead-weight verify step, unhandled `markAsProven`
rejections, and the misleading `trigger()` JSDoc) and as a bonus advances the
L2→L1 outbox -- so a follow-up can drop `startProverNode: true` from tests
that only spin up a simulated prover to push the outbox.

The trade-off is epoch-granularity proven advancement instead of
per-checkpoint. With `aztecEpochDuration` left at the 32-slot default, FAST
e2e proven would only advance every 384s. Add `aztecEpochDuration: 4` to
`FAST_E2E_SETUP_OPTS` (matching `e2e_simple` and
`e2e_p2p/fee_asset_price_oracle_gossip`) so an epoch is 48s wall time, and
bump the smoke test's proven-tip retry window to 150s for two epochs of
slack.
EpochTestSettler runs once per completed epoch, so the proven-tip retry
budgets up to two epochs (the worst case is the tx landing right after an
epoch boundary, needing the full next epoch to complete).
Migrated tests using FAST_E2E_SETUP_OPTS (e2e_token_contract/*, e2e_nested_contract/*,
e2e_fast_config) consistently failed in CI with "Transaction dropped by P2P node"
because the slot-4 propose tx reverted on L1.

Root cause: with aztecEpochDuration: 4, EpochTestSettler fires markAsProven at the
end of epoch 0 (just after the slot-3 propose lands). The cheat code wraps its
storage write in execWithPausedAnvil, which pauses anvil interval mining for the
duration of the cheat-code RPC and shifts the L1 mining cadence on resume. The
next propose, sent at the slot-4 wall-clock boundary, lands in an L1 block whose
block.timestamp is still in slot 3, and propose's validateHeader reverts with
HeaderLib__InvalidSlotNumber(3, 4) inside the multicall. The outer multicall tx
succeeds, but the propose log is absent, so the sequencer treats the checkpoint
as un-published, the pending chain is pruned, and the in-flight L2 tx is ejected
from the P2P pool.

Drop aztecEpochDuration: 4 from FAST_E2E_SETUP_OPTS and let it default to 32, so
EpochTestSettler does not fire within the typical test window. The smoke test
loses its proven-tip assertion (the same epoch-boundary collision); it still
covers chain progression and single-block-per-slot batching.
Setup pace-locks to anvil's 12s wall-clock interval mining since the watcher's
warpTimeIfNeeded loop is gone, and the default `aztecSlotDuration: 72` means
each L2 slot now takes 72s of wall time. `account_init` deploys ~5 blocks
during `beforeAll` (initial account, banana token, FPC class+instance, fund),
which exceeds the 300s Jest hook budget.

Switch to PIPELINING_SETUP_OPTS (12s/4s slots, pipelining, 30x fee padding,
`startProverNode: true` preserved through the spread order in
`FeesTest.setup`), matching the pattern `e2e_fees/gas_estimation.test.ts`
already uses for the same reason. Manual `getPaddedMaxFeesPerGas` calls in
the test bodies still use the 5x default padding; leaving as-is since the
PR's failure is a timeout, not a fee-rejection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant