Add support for duplicate custom request headers#2504
Open
ayanrajpoot10 wants to merge 2 commits into
Open
Conversation
WalkthroughHeader handling is refactored across the codebase to support multiple values per header key. ChangesMultivalue HTTP Header Support
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@runner/runner.go`:
- Line 246: The code currently overwrites existing entries with
httpxOptions.CustomHeaders[customHeader] = []string{""}, which drops duplicate
identical -H inputs; change it to append an empty string to the existing slice
instead (e.g. httpxOptions.CustomHeaders[customHeader] =
append(httpxOptions.CustomHeaders[customHeader], "")) so repeated identical
customHeader values are preserved in httpxOptions.CustomHeaders rather than
replaced.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 07d4fc43-8304-40fd-870e-9a1a094e64f9
📒 Files selected for processing (4)
common/httputilz/httputilz.gocommon/httpx/httpx.gocommon/httpx/option.gorunner/runner.go
| // rawhttp skips all checks | ||
| if options.Unsafe { | ||
| httpxOptions.CustomHeaders[customHeader] = "" | ||
| httpxOptions.CustomHeaders[customHeader] = []string{""} |
There was a problem hiding this comment.
Preserve repeated identical -H entries in unsafe mode.
Line 246 still overwrites prior values for the same raw header string in unsafe mode. Repeating the exact same -H input loses duplicates.
Suggested fix
- httpxOptions.CustomHeaders[customHeader] = []string{""}
+ httpxOptions.CustomHeaders[customHeader] = append(httpxOptions.CustomHeaders[customHeader], "")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@runner/runner.go` at line 246, The code currently overwrites existing entries
with httpxOptions.CustomHeaders[customHeader] = []string{""}, which drops
duplicate identical -H inputs; change it to append an empty string to the
existing slice instead (e.g. httpxOptions.CustomHeaders[customHeader] =
append(httpxOptions.CustomHeaders[customHeader], "")) so repeated identical
customHeader values are preserved in httpxOptions.CustomHeaders rather than
replaced.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
common/httpx/httpx.go (1)
437-454:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winCoalesce canonical header keys before
Del/Addto preserve case-variant duplicates.
net/httpcanonicalizes header keys forHeader.Del/Header.Add, butSetCustomHeadersdoesr.Header.Del(name)per raw map key; if the input contains bothX-Testandx-test, the laterDelwipes the values added for the earlier one. Merge values by canonical key first, then apply once per canonical key.Suggested fix
import ( "context" "crypto/tls" "fmt" "io" "net" "net/http" + "net/textproto" "net/url" "os" "strconv" "strings" "time" @@ func (h *HTTPX) SetCustomHeaders(r *retryablehttp.Request, headers map[string][]string) { - for name, values := range headers { - r.Header.Del(name) + normalized := make(map[string][]string, len(headers)) + for name, values := range headers { + canonical := textproto.CanonicalMIMEHeaderKey(name) + normalized[canonical] = append(normalized[canonical], values...) + } + + for name, values := range normalized { + r.Header.Del(name) for _, value := range values { switch strings.ToLower(name) { case "host": r.Host = value if h.Options.Unsafe { r.Header.Add("Host", value) } case "cookie": // cookies are set in the default branch, and reset during the follow redirect flow fallthrough default: r.Header.Add(name, value) } } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@common/httpx/httpx.go` around lines 437 - 454, The SetCustomHeaders function currently calls r.Header.Del(name) and r.Header.Add(name, value) per raw map key which breaks when input contains case-variant duplicates (e.g., "X-Test" and "x-test"); modify HTTPX.SetCustomHeaders to first coalesce the incoming headers by their canonicalized key (use http.CanonicalHeaderKey or strings.ToLower?; prefer http.CanonicalHeaderKey) into a single map[string][]string, then iterate that canonical-key map once: call r.Header.Del(canonicalKey) and add all merged values (handling special cases like "Host" and "Cookie" exactly as existing logic does) so duplicates differing only by case are preserved instead of overwritten.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@common/httpx/httpx.go`:
- Around line 437-454: The SetCustomHeaders function currently calls
r.Header.Del(name) and r.Header.Add(name, value) per raw map key which breaks
when input contains case-variant duplicates (e.g., "X-Test" and "x-test");
modify HTTPX.SetCustomHeaders to first coalesce the incoming headers by their
canonicalized key (use http.CanonicalHeaderKey or strings.ToLower?; prefer
http.CanonicalHeaderKey) into a single map[string][]string, then iterate that
canonical-key map once: call r.Header.Del(canonicalKey) and add all merged
values (handling special cases like "Host" and "Cookie" exactly as existing
logic does) so duplicates differing only by case are preserved instead of
overwritten.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 238619ef-5db5-4303-8324-82612a847bd9
📒 Files selected for processing (1)
common/httpx/httpx.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
This PR adds support for duplicate custom HTTP request headers in
httpx, aligning its behavior with tools likecurland the HTTP specification, which allows multiple headers with the same key.Previously,
CustomHeadersused amap[string]string. When multiple headers with the same name were provided (for example,-H "X-Test: one" -H "X-Test: two"), earlier values were overwritten and only the last value was sent in the request.This PR fixes issue #2503.
Changes:
CustomHeadersfrommap[string]stringtomap[string][]string, matching Go's nativehttp.Headerstructure.httputilz.ParseRequest()to correctly preserve duplicate headers from raw requests.r.Header.Set()withr.Header.Add()inSetCustomHeaders()so all header values are appended instead of overwritten.Before
Command:
Output:
After
Command:
Output:
Both header values are now correctly preserved and sent in the request.
Proof
go build ./...go test ./...-Hflags and confirmed both headers are included in the final requestChecklist
Summary by CodeRabbit