refactor(codegen): migrate bb.js BB API codegen to ipc-codegen#23316
Open
charlielye wants to merge 8 commits into
Open
refactor(codegen): migrate bb.js BB API codegen to ipc-codegen#23316charlielye wants to merge 8 commits into
charlielye wants to merge 8 commits into
Conversation
charlielye
force-pushed
the
cl/ipc-codegen-migrate-bb-wsdb
branch
6 times, most recently
from
f8c3cd3 to
f5f4f12
Compare
charlielye
force-pushed
the
cl/ipc-codegen-migrate-bb-wsdb
branch
from
f5f4f12 to
905a0cb
Compare
charlielye
force-pushed
the
cl/ipc-codegen-migrate-bb-wsdb
branch
from
905a0cb to
be99316
Compare
Swap the bb.js core BB API binding producer from barretenberg/ts/src/cbind/ to the new ipc-codegen package. Consumers in barretenberg/ts/src/barretenberg/ and bbapi/ continue to import from cbind/generated/ — only the producer changes. Changes: - barretenberg/ts/package.json: "generate" script now invokes ipc-codegen/bootstrap.sh - ipc-codegen/bootstrap.sh: add bb-schema generation - ipc-codegen/schemas/bb_schema.json: refresh from current bb binary (introduces new fields like ChonkComputeVk.use_zk_flavor and the AvmStat type that consumers depend on) - ipc-codegen/scripts/update_schemas.sh: fix paths after the codegen package relocation from barretenberg/codegen/ to ipc-codegen/ Out of scope: - WSDB migration: requires the SERIALIZATION_FIELDS undef guard in barretenberg/cpp/src/barretenberg/serialize/msgpack.hpp before the new wsdb codegen output can compile. That lands in the wire/domain refactor PR for WSDB on the IPC stack. - Deletion of barretenberg/ts/src/cbind/: the directory's generator sources remain in-tree because the IPC stack (cl/ipc-4-avm-binary onwards) still has aztec-cdb/generate.ts and aztec-avm/generate.ts that import from it. Final deletion follows once all four service consumers migrate. Verify: - yarn workspace barretenberg/ts build:esm # bb.js TS build green - yarn generate # produces cbind/generated/*.ts via ipc-codegen
charlielye
force-pushed
the
cl/ipc-codegen-migrate-bb-wsdb
branch
from
be99316 to
96184d9
Compare
…t output
The previous PR2 commit replaced `yarn generate` with just the ipc-codegen
build + aztec-wsdb generation. That dropped the Rust bindings cbind/generate.ts
emits to barretenberg/rust/barretenberg-rs/src/{api,generated_types}.rs,
which barretenberg-rs/src/lib.rs declares as `pub mod api;` and
`pub mod generated_types;`. cargo build then failed with E0583
"file not found for module" in CI.
Run cbind/generate.ts first (produces Rust + an initial TS pass), then
ipc-codegen (overwrites cbind/generated/* with the ipc-codegen output),
then aztec-wsdb/generate.ts. Order matters: ipc-codegen runs AFTER cbind
so its TS output wins; the Rust files are only written by cbind so they
survive. A follow-up PR can migrate the Rust side and finally remove
cbind/generate.ts.
…e.ts
PR2's stated goal was to replace cbind/generate.ts with ipc-codegen for
the BB API. The previous version of this PR only migrated the TS side
and kept cbind/generate.ts in `yarn generate` to produce the Rust
bindings. This commit finishes the job:
- ipc-codegen/bootstrap.sh: add `gen --lang rust --uds --prefix Bb` for
bb_schema.json, writing to barretenberg-rs/src/generated/. The --uds
flag is what pulls in the Backend trait and IpcError templates the
generated client imports.
- barretenberg-rs/src/lib.rs: declare `pub mod generated { ... }` and
re-export under the existing public-API names (BarretenbergApi,
BarretenbergError, Backend, Fr, Command, Response, etc.) so the 230+
external references don't move.
- Delete the now-redundant hand-written src/{backend,error,types}.rs:
- backend.rs is byte-identical to the ipc-codegen template (modulo
`crate::error::Result` vs `super::error::Result`).
- error.rs is the same enum, just under a different name (IpcError vs
BarretenbergError) — re-exported with `as BarretenbergError`.
- types.rs's `Fr` is now produced by the codegen; its extension
methods (from_u64, from_be_bytes, from_buffer, to_buffer, ...) move
to a new src/fr_ext.rs as an extra `impl Fr` block. `Point` is no
longer exported anywhere; existing curve points (GrumpkinPoint,
Secp256k1Point, etc.) come from the generated bb_types.
- backends/{pipe,ffi}.rs: update imports from the old `crate::error::*`
/ `crate::backend::*` / `crate::api::*` paths to the lib-root
re-exports.
- ts/package.json: drop `ts-node src/cbind/generate.ts` from `yarn
generate`. The remaining pipeline is ipc-codegen (TS + Rust) then
aztec-wsdb/generate.ts (still on cbind libs; that's a separate
follow-up).
- barretenberg/.gitignore: drop the now-obsolete legacy paths
(`src/generated_types.rs`, `src/api.rs`); the `**/generated/` rule
covers the new layout.
Verify:
- yarn workspace bb.js build:esm
- (cd barretenberg/rust && BB_LIB_DIR=<...> cargo build --release)
charlielye
force-pushed
the
cl/ipc-codegen-migrate-bb-wsdb
branch
from
1211029 to
a048306
Compare
…egen's barretenberg-rs/src/backends/ffi.rs and ipc-codegen/templates/rust/ffi_backend.rs were already 163 lines of nearly-byte-identical code — same extern "C" fn bbapi signature, same FfiBackend struct, same trait impl, same test block. Add --ffi to the ipc-codegen call so the template lands at barretenberg-rs/src/generated/ffi_backend.rs, declare it as a feature-gated module under crate::generated, and re-export it from `pub mod backends` to preserve the existing `barretenberg_rs::backends::FfiBackend` path. External users (incl. iOS apps consuming FfiBackend via FFI into libbarretenberg.a) see the same public API. The pipe backend stays hand-written — there's no `pipe_backend` template, and spawning a subprocess is barretenberg-rs-specific anyway.
…ration
The earlier "migrate to ipc-codegen" commit moved the generated Rust
sources from barretenberg-rs/src/{api,generated_types}.rs into
barretenberg-rs/src/generated/{bb_client,bb_types}.rs (alongside the
template-copied backend.rs/error.rs/ffi_backend.rs/uds_backend.rs).
barretenberg/rust/bootstrap.sh still referenced the old single-file
paths in two places:
- cache_upload listed the obsolete files, so tar exited 1 with
"Cannot stat: barretenberg-rs/src/generated_types.rs" after the
successful cargo build, failing the bb-rs bootstrap.
- release() checked for the old paths to decide whether to re-run
`yarn generate`.
Switch both to the new directory layout: `barretenberg-rs/src/generated`
(for cache_upload) and `bb_client.rs` / `bb_types.rs` (for the existence
check). cargo build itself was already passing; this is purely the
cache-upload glue.
The ipc-codegen Rust bindings expose ECDSA/Schnorr signature components and BN254 G2 coordinates as typed Fr (32-byte newtype) where the legacy cbind-generated api.rs used Vec<u8> / [u8; 32] directly. The lib itself compiled, but barretenberg-tests' FFI tests failed because they were: - Calling .len() on Fr (now needs .0.len() to access the inner [u8; 32]). - Passing &Fr where the generated verify methods take Fr by value. - Building Bn254G2Point from raw vec![...] literals (now needs Fr::from_be_bytes([...])). Mechanical adaptation only — no behavioral change. Tests link and the unittest binaries build under cargo test --features ffi.
ipc-codegen now generates wsdb's C++ and TS clients from the committed wsdb_schema.json. The aztec-wsdb server binary is unchanged on next — its hand-written SERIALIZATION_FIELDS still define the canonical wire format, and the schema was extracted from it, so both producers emit byte-compatible clients and the server keeps speaking the same protocol. C++ client: - Lift wsdb_ipc_merkle_db.cpp from cl/more_ipc_v2: adds inline wire<->domain conversion helpers (fr_to_wire, fr_from_wire, fr_vec_from_wire, revision_to_wire, tree_id_to_wire) and threads them through every call site. Public interface unchanged. - wsdb/CMakeLists.txt now compiles generated/wsdb_ipc_client.cpp (from ipc-codegen) instead of the legacy hand-emitted wsdb_ipc_client_generated.cpp. TS: - ipc-codegen emits aztec-wsdb's TS client to barretenberg/ts/src/aztec-wsdb/generated/ (same path the previous cbind-based pipeline used). - aztec-wsdb/index.ts re-exports of AsyncApi + api_types continue to resolve. Deletions: - barretenberg/ts/src/cbind/ (legacy multi-language codegen — superseded by ipc-codegen for every remaining consumer). - barretenberg/ts/src/aztec-wsdb/generate.ts (driver for the deleted cbind/). - yarn-project/world-state/src/native/ipc_world_state_instance.ts (dead code on next; the IPC stack re-adds it against the new generated shape during its cutover).
generate.ts wrote the cpp templates (msgpack_struct_map_impl.hpp, ipc_client.hpp, ipc_server.hpp) verbatim via copyTemplate without pushing them through formatCpp. When ipc-codegen's output landed under barretenberg/cpp/src/.../generated/, the project-wide format-check caught the LLVM-default formatting of those templates and failed. Add the copied template paths to cppFiles so formatCpp picks them up alongside the generated client/server source. clang-format walks up from each file path, so the destination's .clang-format gets applied correctly for each consumer.
Collaborator
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack A PR2 of the two-stack codegen migration (see `/mnt/user-data/charlie/.claude/plans/glittery-snuggling-horizon.md`).
Swaps the bb.js core BB API binding producer from `barretenberg/ts/src/cbind/generate.ts` (legacy schema-visitor) to the new `ipc-codegen/` package introduced in #23314. The output landing place (`barretenberg/ts/src/cbind/generated/`) and the exported names (`AsyncApi`, `SyncApi`, `ChonkProof`, etc.) are unchanged, so consumers in `barretenberg/ts/src/barretenberg/` and `bbapi/` compile without modification.
What this lands
Out of scope (deferred)
Local verification
Test plan