config: add allow_incompatible_key_usage TLS option

[alliasgher]

Problem

Let's Encrypt announced they will stop issuing certificates with the TLS Client Authentication Extended Key Usage (EKU) in 2026. Modern Go TLS strictly enforces EKU matching — it rejects client certificates that lack ExtKeyUsageClientAuth and server certificates that lack ExtKeyUsageServerAuth.

This breaks components (notably Alertmanager cluster/gossip) that reuse the same LE certificate for both TLS server and client roles. The peer acting as a TLS server rejects the connecting peer's certificate with:

tls: failed to verify certificate: x509: certificate specifies an incompatible key usage

See: prometheus/alertmanager#5151

Solution

Add allow_incompatible_key_usage to TLSConfig:

tls_client_config:
  cert_file: /etc/pki/le/fullchain.pem
  key_file:  /etc/pki/le/privkey.pem
  allow_incompatible_key_usage: true   # new

When set, the EKU check is skipped while the following security properties are preserved:

• Certificate chain trust (CA verification)
• Certificate expiry
• Hostname verification (for client connections via server_name or SNI)

insecure_skip_verify remains unchanged and is not implied.

Implementation

Go's TLS stack performs EKU verification internally and there is no supported hook to disable only that check. The workaround is:

1. Set InsecureSkipVerify = true — disables Go's built-in verification entirely
2. Install a VerifyPeerCertificate callback that re-implements the verification using x509.VerifyOptions{KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageAny}} — accepts any EKU while still checking chain + expiry + hostname

The callback captures the *tls.Config pointer (not the field value) so that callers who set RootCAs after NewTLSConfig returns are still respected.

Tests

TestTLSConfigAllowIncompatibleKeyUsage:

• Generates a CA + server cert with only clientAuth EKU (no serverAuth) — the EKU mismatch a TLS client would encounter with an LE cert used for mTLS
• Asserts that without the flag the connection fails with incompatible key usage
• Asserts that with the flag the connection succeeds

[roidelapluie]

Hello,

Thank you for this pull request which seems to fix something real. Can we use VerifyConnection instead? (pseudocode:)

tlsConfig.InsecureSkipVerify = true // required to suppress built-in verify
tlsConfig.VerifyConnection = func(cs tls.ConnectionState) error {
    opts := x509.VerifyOptions{
        Roots:         tlsConfig.RootCAs,
        Intermediates: x509.NewCertPool(),
        DNSName:       cs.ServerName,        // honors per-request SNI
        KeyUsages:     []x509.ExtKeyUsage{x509.ExtKeyUsageAny},
    }
    for _, c := range cs.PeerCertificates[1:] {
        opts.Intermediates.AddCert(c)
    }
    _, err := cs.PeerCertificates[0].Verify(opts)
    return err
}

also add negative tests and tests with other hostnames would be great.