Protect operations database from non-string data in error messages fr…#1716
Protect operations database from non-string data in error messages fr…#1716EnriqueL8 merged 6 commits intohyperledger:mainfrom
Conversation
…om reverts Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1716 +/- ##
==========================================
- Coverage 99.94% 99.94% -0.01%
==========================================
Files 342 343 +1
Lines 25024 30465 +5441
==========================================
+ Hits 25010 30447 +5437
- Misses 10 14 +4
Partials 4 4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
765a1c5 to
fe2e153
Compare
Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
…can decide whether to take different action on nilPtr Signed-off-by: Dave Crighton <dave.crighton@kaleido.io>
da4894c to
06960e0
Compare
Signed-off-by: Dave Crighton <dave.crighton@kaleido.io> Made-with: Cursor # Conflicts: # Dockerfile
fa1235d to
0baf6f8
Compare
EnriqueL8
left a comment
There was a problem hiding this comment.
Just one question if we should handle the API side of it getting the original error..
| // Nil pointers return "". Strings containing invalid UTF-8 sequences or null | ||
| // bytes (which PostgreSQL rejects in text columns even though null bytes are | ||
| // technically valid UTF-8) are hex-encoded instead, with a warning logged. | ||
| func DBSafeUTF8StringFromPtr(ctx context.Context, s *string) string { |
There was a problem hiding this comment.
Eventually this should be moved to FF Common
| } | ||
| if !utf8.ValidString(*s) || strings.ContainsRune(*s, 0) { | ||
| hexString := hex.EncodeToString([]byte(*s)) | ||
| log.L(ctx).Warnf("String contains invalid UTF-8 or null bytes - encoding as hex: %s", hexString) |
There was a problem hiding this comment.
So this is an one way and then on the API surface we don't decode this right?
There was a problem hiding this comment.
Correct if an error is hex encoded as it is inserted into the DB it remain hex encoded on the API surface.
There was a problem hiding this comment.
More broadly the query is whether we should decode hex bytes coming out the other side of the database in order to rehydrate the initial context.
I assert that the answer is no since the entire call surface expects this field to be a human readable string, strings containing null bytes or non-UTF8 characters likely already pose a risk to consumers that are expecting strings. These would appears as \u0000 in serialized json which at best will present a garbled error message and at worst may be treated as string terminators. It's safer to present a consistently encoded string to the user to deal with these abnormal cases.
Additionally due to hyperledger/firefly-evmconnect#184 and hyperledger/firefly-signer#98 a known source of these malformed error strings will now be decoded in a more friendly way preserving the context of nested ABI.
Therefore this particular change should be viewed as a belt-and-braces protection for preventing unforseen types of malformed error from completely blocking transaction processing (and blocks submission of transaction with following nonces etc) due to:
[2026-02-27T12:37:44.884Z] ERROR SQL update failed: pq: invalid byte sequence for encoding "UTF8": 0x00 sql=[ UPDATE operations SET opstatus = $1, error = $2, updated = $3 WHERE (id = $4 AND namespace = $5) ] dbtx=Qirvs5Un ns=ff1 opupdater=opu_004 pid=....
There was a problem hiding this comment.
Thanks for detailed explanation here
Cherrypick PR hyperledger#1716 from upstream
Proposed changes
This change tests the error string before making an update to the operations table checking that it is a valid UTF-8 string and does not contain null bytes. The second check is required because UTF-8 considers a single null byte to be a valid string but Postgres does not.
Part of the fix for #1717
Types of changes
Please make sure to follow these points
Screenshots (If Applicable)
Other Information
Any message for the reviewer or kick off the discussion by explaining why you considered this particular solution, any alternatives etc.