fix(s3): paginate ListObjectsV2 for getBundleHistory (s3.ts)#164
Open
FastHaleqt wants to merge 1 commit intobase:masterfrom
Open
fix(s3): paginate ListObjectsV2 for getBundleHistory (s3.ts)#164FastHaleqt wants to merge 1 commit intobase:masterfrom
FastHaleqt wants to merge 1 commit intobase:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
getBundleHistorylists every object under the S3 prefixbundles/{bundleKey}/and then loads each object to build the bundle’s event history. It does this with Amazon S3ListObjectsV2, which returns at most ~1000 keys per request. If that prefix has more than one page of keys, the old code only read the first page. Any events whose keys appeared on later pages were never listed and never loaded, so the UI/API showed an incomplete bundle history for large bundles.Why this matters
Bundle history is used to understand what happened to a user’s transaction (e.g. metering, lifecycle events). Silently dropping events after the first 1000 keys is hard to notice in normal testing but wrong in production for high-volume or long-running bundles. Pagination is the standard S3 pattern to list an unbounded number of objects under a prefix.
What we changed
listAllObjectKeysForPrefix(s3, bucket, prefix)(insrc/lib/s3.ts, exported) — followsIsTruncated/NextContinuationTokenand passesContinuationTokenon the next call until all keys under the prefix are collected.getBundleHistorynow uses that helper forbundles/{bundleKey}/, then unchanged per-keyGetObject+ JSON parsing. Semantics are the same as before; we only fix completeness of the key list.TIPS_UI_S3_BUCKET_NAME/ defaulttips, sames3Clientsetup as the rest of the file).