fix(audit): distinguish README comparison failure from content mismatch

[sadlilas]

Symptom

During the Microsoft ecosystem audit run on 2026-05-04, the check-readme-sections step in recipes/repo-audit.yaml reported a false positive on microsoft/amplifier-collections:

• Finding: README.md sections exist but content differs from template (Trademarks)
• Actual: The Trademarks section was verbatim identical to the reference template

Re-running the recipe's extraction logic against the same files and reference template produced TRADE_MATCH=true, confirming the content matched perfectly and the "false" result was a comparison failure being mis-reported as a content mismatch.

Root Cause

The match flags in check-readme-sections use a binary false-default pattern:

TRADE_MATCH="false"
if [ "$HAS_TRADE" -gt 0 ] && [ -n "$TRADE_START" ]; then
   # ... extraction and diff ...
   if diff -q ... ; then TRADE_MATCH="true"; fi
fi

This collapses three distinct conditions into one outcome:

1. Section does not exist (HAS_TRADE=0) → TRADE_MATCH=false ✓ (correct finding)
2. Section exists and matches template (diff succeeds, no output) → TRADE_MATCH=true ✓ (correct)
3. Section exists but differs from template (diff finds differences) → TRADE_MATCH=false ✓ (correct)
4. Comparison failed (reference fetch empty, extraction malformed, sed/grep error, partial file read) → TRADE_MATCH=false ✗ (misleading — this is "I couldn't compare", not "they differ")

Conditions 3 and 4 both produce TRADE_MATCH=false and surface to the user as:

"README.md sections exist but content differs from template"

But condition 4 is a transient failure requiring retry, while condition 3 is a real content mismatch requiring a README edit. Users cannot distinguish the two from the current finding text.

The Fix

Three-state match flags replace binary defaults:

• true = comparison completed, content matched
• false = comparison completed, content differed
• unknown = comparison could not be completed (missing/empty files, extraction failure, sanity checks failed)

Sanity-check the reference README before comparing:

• Verify reference is non-empty
• Verify it contains both ## Contributing and ## Trademarks headings
• Set REF_README_OK=false if not; skip comparisons that rely on it

Skip comparisons when extraction yields empty content:

• If either extracted section is empty, mark match as unknown rather than silently failing the diff
• This prevents malformed extraction (missing heading, sed error) from silently appearing as "content differs"

Distinct findings per condition:

• missing-section → ERROR severity, "section not found" (user action required: add section)
• unknown-match → WARNING severity, "comparison could not complete; reference unavailable or extraction failed; retry recommended" (transient, requires retry)
• false-match → WARNING severity, "content differs from template" (real mismatch, user action required: update section)

The JSON contract still emits boolean *_verbatim fields (for machine consumers), but they now derive correctly from the three-state flag: only true maps to true; both false and unknown map to false. The distinct signal for "comparison failed" is now carried by severity and finding text, allowing users and downstream tooling to respond appropriately.

Scope Deliberately Limited

Not included (deferred pending evidence):

• Per-repo reference directories: I hypothesized a race condition in the shared reference/ directory (one dir written by all parallel repo-audits in step 3). Investigation showed this is plausible but not proven — one false positive over 112 repos is consistent with a transient blip, not a structural race. Restructuring the working directory is speculative. Deferring until the same symptom reproduces or multiple repos show false positives.

• Canonicalization logic changes: The current whitespace-normalized awk diff (awk '...' | sort | diff) is sufficient when extraction succeeds. No change needed.

Verification

• YAML syntax: Validated via python3 -c "import yaml; yaml.safe_load(...)"
• Bash syntax: Validated via bash -n on extracted command blocks
• Uniqueness: All 79 replacements verified unique before applying
• Diff stats: 79 insertions, 19 deletions across recipes/repo-audit.yaml

---

Generated with Amplifier