Skip to content

Protect operations database from non-string data in error messages fr…#1716

Merged
EnriqueL8 merged 6 commits intohyperledger:mainfrom
davecrighton:djc/protectDBAccess
Apr 22, 2026
Merged

Protect operations database from non-string data in error messages fr…#1716
EnriqueL8 merged 6 commits intohyperledger:mainfrom
davecrighton:djc/protectDBAccess

Conversation

@davecrighton
Copy link
Copy Markdown
Contributor

@davecrighton davecrighton commented Feb 27, 2026

Proposed changes

This change tests the error string before making an update to the operations table checking that it is a valid UTF-8 string and does not contain null bytes. The second check is required because UTF-8 considers a single null byte to be a valid string but Postgres does not.

Part of the fix for #1717


Types of changes

  • Bug fix
  • New feature added
  • Documentation Update

Please make sure to follow these points

  • I have read the contributing guidelines.
  • I have performed a self-review of my own code or work.
  • I have commented my code, particularly in hard-to-understand areas.
  • My changes generates no new warnings.
  • I have added tests that prove my fix is effective or that my feature works.
  • My changes have sufficient code coverage (unit, integration, e2e tests).

Screenshots (If Applicable)


Other Information

Any message for the reviewer or kick off the discussion by explaining why you considered this particular solution, any alternatives etc.

…om reverts

Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
@davecrighton davecrighton marked this pull request as ready for review February 27, 2026 15:28
@davecrighton davecrighton requested a review from a team as a code owner February 27, 2026 15:28
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.94%. Comparing base (1cb77c5) to head (3607be6).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1716      +/-   ##
==========================================
- Coverage   99.94%   99.94%   -0.01%     
==========================================
  Files         342      343       +1     
  Lines       25024    30465    +5441     
==========================================
+ Hits        25010    30447    +5437     
- Misses         10       14       +4     
  Partials        4        4              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
@davecrighton davecrighton force-pushed the djc/protectDBAccess branch from 765a1c5 to fe2e153 Compare March 2, 2026 11:21
Comment thread internal/operations/operation_updater.go Outdated
Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
…can decide whether to take different action on nilPtr

Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
Made-with: Cursor

# Conflicts:
#	Dockerfile
Copy link
Copy Markdown
Contributor

@EnriqueL8 EnriqueL8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question if we should handle the API side of it getting the original error..

Comment thread pkg/utils/dbstring.go
// Nil pointers return "". Strings containing invalid UTF-8 sequences or null
// bytes (which PostgreSQL rejects in text columns even though null bytes are
// technically valid UTF-8) are hex-encoded instead, with a warning logged.
func DBSafeUTF8StringFromPtr(ctx context.Context, s *string) string {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually this should be moved to FF Common

Comment thread pkg/utils/dbstring.go
}
if !utf8.ValidString(*s) || strings.ContainsRune(*s, 0) {
hexString := hex.EncodeToString([]byte(*s))
log.L(ctx).Warnf("String contains invalid UTF-8 or null bytes - encoding as hex: %s", hexString)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is an one way and then on the API surface we don't decode this right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct if an error is hex encoded as it is inserted into the DB it remain hex encoded on the API surface.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More broadly the query is whether we should decode hex bytes coming out the other side of the database in order to rehydrate the initial context.

I assert that the answer is no since the entire call surface expects this field to be a human readable string, strings containing null bytes or non-UTF8 characters likely already pose a risk to consumers that are expecting strings. These would appears as \u0000 in serialized json which at best will present a garbled error message and at worst may be treated as string terminators. It's safer to present a consistently encoded string to the user to deal with these abnormal cases.

Additionally due to hyperledger/firefly-evmconnect#184 and hyperledger/firefly-signer#98 a known source of these malformed error strings will now be decoded in a more friendly way preserving the context of nested ABI.

Therefore this particular change should be viewed as a belt-and-braces protection for preventing unforseen types of malformed error from completely blocking transaction processing (and blocks submission of transaction with following nonces etc) due to:

[2026-02-27T12:37:44.884Z] ERROR SQL update failed: pq: invalid byte sequence for encoding "UTF8": 0x00 sql=[ UPDATE operations SET opstatus = $1, error = $2, updated = $3 WHERE (id = $4 AND namespace = $5) ] dbtx=Qirvs5Un ns=ff1 opupdater=opu_004 pid=....

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for detailed explanation here

@peterbroadhurst peterbroadhurst dismissed their stale review April 22, 2026 10:24

Baton handed to Enrique

@EnriqueL8 EnriqueL8 merged commit e3c9b8d into hyperledger:main Apr 22, 2026
29 of 30 checks passed
EnriqueL8 added a commit to kaleido-io/firefly that referenced this pull request Apr 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants