Networking10 min read

mTLS & Certificate Management

Mutual authentication and practical certificate operations

mTLS — Mutual TLS Authentication

In standard TLS, only the server authenticates itself to the client by presenting a certificate. The client verifies the server's identity, but the server has no cryptographic proof of who the client is. mTLS (Mutual TLS) extends this so that both sides present and verify certificates.

Standard TLS vs mTLS

Standard TLS

Server presents certificate
Client verifies server identity
Client is anonymous at the TLS layer
Client authentication happens at the application layer (cookies, tokens, API keys)

mTLS (Mutual TLS)

Server presents certificate
Client verifies server identity
Server requests client certificate (CertificateRequest)
Client presents certificate
Server verifies client identity
Both sides are cryptographically authenticated

mTLS Handshake

ClientHello (cipher suites, key_share, SNI)

ServerHello + key_share

{EncryptedExts + CertificateRequest + ServerCert + Finished}

{ClientCert + CertificateVerify + Finished}

Key Difference from Standard TLS

The server sends a CertificateRequest message in its handshake flight, telling the client "I need you to prove your identity with a certificate." The client responds with its certificate and a CertificateVerify message — a signature over the handshake transcript using the client's private key. The server validates the client certificate against its configured CA trust bundle.

mTLS Use Cases

Zero-Trust Service Mesh

In zero-trust architectures, every service-to-service call is authenticated and encrypted. No implicit trust based on network location. mTLS provides identity at the transport layer — you don't just know "something on 10.0.3.42 called me," you know "the payment-service with certificate CN=payment.prod.svc called me."

Kubernetes Service Meshes

Istio and Linkerd inject sidecar proxies (Envoy for Istio, linkerd2-proxy for Linkerd) that transparently handle mTLS between services. The application code doesn't need to manage certificates — the mesh control plane automates certificate issuance, rotation, and distribution.

API Authentication

mTLS is stronger than API keys or bearer tokens. An API key can be stolen and replayed from anywhere. A client certificate requires possession of the private key, which never leaves the client. Banks, payment processors, and cloud providers use mTLS for partner API authentication.

Tip

Istio mTLS modes: Istio supports STRICT (only mTLS connections accepted — plaintext is rejected), PERMISSIVE (accepts both mTLS and plaintext — useful during migration), and DISABLE. Always aim for STRICT in production. PERMISSIVE is a migration aid, not a steady state.

mTLS in a K8s Service Mesh (Istio)

Service A Pod
App sends plaintext
to localhost (sidecar)
Envoy Sidecar A
Intercepts traffic
Initiates mTLS
Envoy Sidecar B
Terminates mTLS
Verifies client cert
Service B Pod
Receives plaintext
from sidecar

Note

SPIFFE: Istio uses SPIFFE (Secure Production Identity Framework for Everyone) identities. Each workload gets a SVID (SPIFFE Verifiable Identity Document) — an X.509 certificate with a SPIFFE ID as the SAN URI, e.g., spiffe://cluster.local/ns/default/sa/payment-service. This gives every pod a cryptographic identity tied to its K8s service account.

Let's Encrypt & the ACME Protocol

Before Let's Encrypt (2016), obtaining TLS certificates required paying a CA, manually generating CSRs, and going through a validation process. Let's Encrypt automated the entire flow using the ACME (Automatic Certificate Management Environment) protocol (RFC 8555).

ACME Challenge Types

Challenge	Mechanism	Pros	Cons
`HTTP-01`	ACME server fetches `http://<domain>/.well-known/acme-challenge/<token>`	Simple, no DNS access needed	Requires port 80 open, doesn't work for wildcards, doesn't work behind CDNs that cache
`DNS-01`	Create a TXT record: `_acme-challenge.<domain>` with a specific value	Works for wildcards, no HTTP server needed, works from private networks	Requires DNS API access, propagation delay
`TLS-ALPN-01`	Serve a self-signed cert with an ACME-specific ALPN extension on port 443	Works when only port 443 is available (no port 80), no DNS access needed	More complex setup, limited tooling support

Tip

DNS-01 is the power move. It's the only challenge that supports wildcard certificates (*.example.com). It works even if your server isn't publicly reachable. And it integrates cleanly with infrastructure-as-code: Terraform/Pulumi can create the DNS record, certbot or cert-manager verifies it, and the cert is issued. The tradeoff is that you need programmatic DNS access (Route53, CloudFlare, Google Cloud DNS, etc.).

ACME Flow (HTTP-01)

POST /acme/new-order (domain: example.com)

Order created + authorization URL

GET /acme/authz (fetch challenge)

HTTP-01 challenge: token + thumbprint

POST /acme/challenge (ready)

Validates http://example.com/.well-known/acme-challenge/...

POST /acme/finalize (send CSR)

Certificate issued (download URL)

cert-manager in Kubernetes

cert-manager is the de-facto standard for automated certificate management in Kubernetes. It watches for Certificate resources, obtains certificates from configured issuers, stores them as Secrets, and handles automatic renewal.

Key Resources

Resource	Scope	Purpose
`Issuer`	Namespace	Defines how to obtain certificates. References a CA (ACME, Vault, self-signed, CA key pair).
`ClusterIssuer`	Cluster-wide	Same as Issuer but available across all namespaces. Typically used for Let's Encrypt.
`Certificate`	Namespace	Declares the desired certificate (domain, issuer, secret name). cert-manager creates/renews it.
`CertificateRequest`	Namespace	Internal resource — represents a single issuance request to the issuer.

cert-manager with Let's Encrypt (DNS-01)

console

# ClusterIssuer for Let's Encrypt production
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - dns01:
        route53:
          region: us-east-1
          hostedZoneID: Z2ABCDEF123456

console

# Certificate resource — cert-manager handles the rest
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: production
spec:
  secretName: api-tls-secret          # K8s Secret with tls.crt + tls.key
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - api.example.com
  - "*.api.example.com"             # Wildcard — requires DNS-01
  renewBefore: 720h                  # Renew 30 days before expiry

cert-manager Issuers

ACME (Let's Encrypt)

Automated via HTTP-01 or DNS-01 challenges. Free. 90-day certs. The most common issuer for public-facing services.

HashiCorp Vault

Vault's PKI secrets engine acts as a private CA. Perfect for internal mTLS certs. Supports short-lived certs (hours/days) with automatic rotation.

Self-Signed / CA

For dev/test environments or bootstrapping. SelfSigned issuer generates a self-signed CA, then a CA issuer uses that CA to sign certs. Not for production public-facing services.

Note

Ingress integration: cert-manager integrates with Ingress resources via annotations. Adding cert-manager.io/cluster-issuer: letsencrypt-prod to an Ingress automatically triggers certificate issuance for the Ingress's tls.hosts. No separate Certificate resource needed.

Short-Lived Certificates

Traditional certificates have long lifetimes — 1-2 years (now capped at 398 days for public CAs). Short-lived certificates last hours to days. This is a paradigm shift with significant security benefits.

Long-Lived Certificates (Days-Years)

Compromised key is usable for months
Revocation is required but unreliable:
- CRLs are often cached and not checked
- OCSP adds latency; some clients soft-fail
- Revocation propagation is not instantaneous
Manual renewal = operational risk (expired certs = outage)
Large blast radius on compromise

Short-Lived Certificates (Hours-Days)

Compromised key is usable for hours at most
No revocation needed — the cert expires before revocation infrastructure could react anyway
Automated renewal is mandatory (forces good practices)
Small blast radius — exposure window is tiny
Istio/Linkerd issue certs with 24h lifetime by default

Warning

Prerequisite: Short-lived certificates require robust automation. If your renewal pipeline breaks, you have hours (not months) before everything fails. This is why tools like cert-manager, Istio's citadel, and Vault's PKI engine are critical — they handle the entire lifecycle automatically.

Tip

Industry trend: Apple, Google, and Mozilla have been pushing for shorter certificate lifetimes for public CAs. As of 2025, the CA/Browser Forum is moving toward 90-day maximum lifetimes (matching Let's Encrypt's existing practice), with a path toward even shorter (47-day and eventually 10-day) certificates. Automation is no longer optional — it's table stakes.

OpenSSL CLI — Essential Commands

OpenSSL is the Swiss Army knife for certificate operations. These are the commands you'll use regularly in production.

Inspect a Certificate

console


    $ openssl x509 -in cert.pem -text -noout
    
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:a1:b2:c3:d4:e5:f6:07:08:09:0a:0b:0c:0d:0e:0f
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Mar  1 00:00:00 2026 GMT
            Not After : May 30 00:00:00 2026 GMT
        Subject: CN=api.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:api.example.com, DNS:*.api.example.com
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

Note

Key flags: -text outputs human-readable details. -noout suppresses the PEM-encoded cert output. Add -dates instead of -text to see only validity dates. Add -subject or -issuer for quick checks.

Verify a Certificate Chain

console


    $ openssl verify -CAfile ca-bundle.pem -untrusted intermediate.pem cert.pem
    cert.pem: OK

console

# -CAfile    → trusted root CA(s)
# -untrusted → intermediate CA(s) (not in trust store but part of the chain)
# Last arg   → the end-entity certificate to verify

# Common failure modes:
# "unable to get local issuer certificate" → missing intermediate in the chain
# "certificate has expired" → self-explanatory
# "self signed certificate in certificate chain" → CA not in trust store

Test a TLS Connection

console


    $ openssl s_client -connect api.example.com:443 -servername api.example.com
    
CONNECTED(00000003)
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R3
verify return:1
depth=0 CN=api.example.com
verify return:1
---
Certificate chain
 0 s:CN=api.example.com
   i:C=US, O=Let's Encrypt, CN=R3
 1 s:C=US, O=Let's Encrypt, CN=R3
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFLTCCBBWgAwIBAgISA...
-----END CERTIFICATE-----
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3421 bytes and written 373 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384

What to look for in s_client output

Certificate chain: Should show end-entity + intermediates. Missing intermediates cause "unable to verify" errors in clients.
Verification: OK — chain validates against the system trust store.
Protocol / Cipher: Confirm TLS 1.3 and a strong cipher suite.
Server Temp Key: Should show X25519 or P-256 (ECDHE — forward secrecy).
"No client certificate CA names sent" — means the server is NOT requesting mTLS. If it is, you'll see the list of acceptable CAs.
-servername flag: Sets the SNI field. Required for virtual-hosted servers to return the correct cert.

Generate a Self-Signed Certificate

console


    $ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 \
  -subj "/CN=localhost" -nodes
    
Generating a RSA private key
........++++
.............................++++
writing new private key to 'key.pem'
-----

console

# -x509      → output a self-signed certificate (not a CSR)
# -newkey     → generate a new private key (rsa:4096 = RSA with 4096-bit key)
# -keyout     → where to write the private key
# -out        → where to write the certificate
# -days 365   → validity period
# -subj       → subject DN (skip interactive prompts)
# -nodes      → no DES — don't encrypt the private key (no passphrase)

# For ECDSA instead of RSA (smaller, faster):
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
  -keyout key.pem -out cert.pem -days 365 -subj "/CN=localhost" -nodes

Generate a CSR (Certificate Signing Request)

console


    $ openssl req -new -newkey rsa:4096 -keyout key.pem -out csr.pem \
  -subj "/CN=api.example.com" -nodes
    
Generating a RSA private key
........++++
writing new private key to 'key.pem'
-----

console

# Inspect a CSR:
openssl req -in csr.pem -text -noout

# Sign a CSR with your CA:
openssl x509 -req -in csr.pem -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -out signed-cert.pem -days 90

Other Useful Commands

console


    # Check certificate expiry date
    $ openssl x509 -in cert.pem -enddate -noout
    notAfter=May 30 00:00:00 2026 GMT

console


    # Check a remote server's certificate (without full s_client output)
    $ echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null | openssl x509 -noout -dates -subject -issuer
    
notBefore=Mar  1 00:00:00 2026 GMT
notAfter=May 30 00:00:00 2026 GMT
subject=CN=api.example.com
issuer=C=US, O=Let's Encrypt, CN=R3

console


    # Verify that a private key matches a certificate
    $ diff <(openssl x509 -in cert.pem -noout -modulus) <(openssl rsa -in key.pem -noout -modulus)
    (no output = they match)

console


    # Convert PEM to PKCS#12 (for importing into Java keystores, browsers, etc.)
    $ openssl pkcs12 -export -in cert.pem -inkey key.pem -out bundle.p12 -name "api.example.com"

Warning

Debugging checklist when TLS fails:

Is the certificate expired? (openssl x509 -dates)
Is the certificate chain complete? (missing intermediate is the #1 cause of "works in browser but not in curl/Java")
Does the SAN match the hostname? (openssl x509 -ext subjectAltName)
Is the server sending the right cert? (openssl s_client -servername)
Is the private key matching the certificate? (compare modulus)
Is the protocol/cipher supported by both sides? (check s_client output)

mTLS — Mutual TLS Authentication

Standard TLS vs mTLS

mTLS Handshake

mTLS Use Cases

mTLS in a K8s Service Mesh (Istio)

Service A Pod

Envoy Sidecar A

Envoy Sidecar B

Service B Pod

Let's Encrypt & the ACME Protocol

ACME Challenge Types

ACME Flow (HTTP-01)

cert-manager in Kubernetes

Key Resources

cert-manager with Let's Encrypt (DNS-01)

Short-Lived Certificates

OpenSSL CLI — Essential Commands

Inspect a Certificate

Verify a Certificate Chain

Test a TLS Connection

Generate a Self-Signed Certificate

Generate a CSR (Certificate Signing Request)

Other Useful Commands