openshift-lb-envoy

Envoy in front of one public IP, fronting many OpenShift clusters plus any plain DNS→IP services. SSL passthrough — backends own their certs; Envoy just routes by SNI. All driven from docker-compose.yml — no custom image, no separate config file.

                                  ┌──► hammer2 SNO  (172.16.2.50)
Internet ──► <public_ip> ──► Envoy ──► anaeem  SNO  (172.16.2.52)
              80 / 443 / 6443     ├──► asaran  SNO  (172.16.2.54)
                                  └──► sat     (172.16.2.5, generic service)

How it routes

Listener	Mechanism	Routes by
:443	TLS ClientHello SNI → TCP passthrough	*.apps.<cluster>, service hostname
:6443	TLS ClientHello SNI → TCP passthrough	api.<cluster>
:80	HTTP Host header	*.apps.<cluster>, service hostname
one per TCP_FORWARDS port	plain TCP proxy, no matching	the listener's own port

No TLS termination at Envoy. oc/kubectl work because mTLS isn't broken. Browsers see OpenShift's certs.

Architecture

Two services in docker-compose.yml:

• config — stock python:3.13-alpine. Decodes a base64-embedded Python script, reads the CLUSTERS and SERVICES env vars, writes /etc/envoy/envoy.json into a shared Docker volume, exits.
• envoy — stock envoyproxy/envoy:v1.32-latest. Depends on config completing successfully, then reads the generated JSON.

No docker build. No bind-mounted source files. Ship docker-compose.yml to the host and run.

Usage

Edit the CLUSTERS env var in docker-compose.yml:

CLUSTERS: |
  [
    {"name": "hammer2", "base_domain": "hammer2.na-launch.com", "ingress_ip": "172.16.1.50"},
    {"name": "anvil",   "base_domain": "anvil.na-launch.com",   "ingress_ip": "172.16.1.51"}
  ]

Each cluster needs name, base_domain, ingress_ip. api_ip is optional and defaults to ingress_ip (SNO).

For anything that isn't an OpenShift cluster — any hostname that should be forwarded to an IP with SSL passthrough — use the SERVICES env var:

SERVICES: |
  [
    {"hostname": "sat.na-launch.com", "ip": "172.16.2.5"}
  ]

Each service needs hostname and ip. Optional keys:

• name — stat/cluster name (defaults to the hostname, sanitized)
• https_port — backend TLS port (default 443)
• http_port — backend HTTP port (default 80; set 0 to disable HTTP routing)

hostname may be a wildcard like *.example.com. HTTPS is routed on :443 by SNI (passthrough, backend keeps its cert); HTTP on :80 by Host header. Both CLUSTERS and SERVICES are optional individually, but at least one entry must exist; duplicate hostnames across the two are rejected at generation time.

For raw TCP that carries neither SNI nor an HTTP Host header — a bare protocol speaking straight over a socket — use the TCP_FORWARDS env var:

TCP_FORWARDS: |
  [
    {"name": "duel_gs", "ip": "172.16.2.64", "ports": "7000-7009"}
  ]

Each item needs name, ip, ports. ports accepts a "low-high" range string, a single int, or a list of ints. Every port in the range gets its own listener that proxies straight through — no tls_inspector, no filter_chain_match, nothing to key on but the port itself, so it's for trusted internal ranges only, not public hostnames. Ports must not collide with the router's own 8080/8443/6443/9901, or with each other; each one also needs its own entry in the envoy service's ports: list below, since Docker only publishes what's listed there.

Then:

docker compose up -d --force-recreate     # apply / reapply
docker compose logs envoy
docker compose down

--force-recreate is what re-runs the config container so env-var changes take effect.

Deployment

1. NAT / port-forward the public IP to the compose host: 80 → 5080, 443 → 50443, 6443 → 56443 (the compose file publishes Envoy on those host ports).
2. DNS per cluster / service:

   *.apps.<cluster>.<domain>   → <public_ip>
   api.<cluster>.<domain>      → <public_ip>
   <service_hostname>          → <public_ip>
   

3. Reachability: the host must reach each cluster's ingress / API IPs on 80/443/6443, and each service IP on its http_port/https_port.
4. docker compose up -d.

Verify

# OpenShift's cert should appear, not anything from Envoy:
openssl s_client -connect <public_ip>:443 \
  -servername console-openshift-console.apps.<cluster>.<domain> </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject

# Console:
curl -vk https://console-openshift-console.apps.<cluster>.<domain>/

# API via passthrough:
oc login -u kubeadmin https://api.<cluster>.<domain>:6443

Inspect the generated Envoy config:

docker run --rm -v $(basename $PWD)_envoy-config:/cfg alpine cat /cfg/envoy.json

Editing the generator

If you need to change the Envoy config schema (add headers, tweak ports, etc.), edit generate.py, then refresh the base64 in compose:

./embed.sh
docker compose up -d --force-recreate

generate.py and embed.sh are dev-time only — not needed at deploy time. The base64 in docker-compose.yml is the canonical script at runtime.

Constraints

• One cluster per base_domain, one service per hostname — SNI can't disambiguate duplicates (the generator rejects them).
• Admin interface bound to 127.0.0.1:9901; SSH-tunnel for stats. Don't publish it wider — it's unauthenticated and mutating, and Docker-published ports bypass the host firewall.
• Pure passthrough — no per-route HSTS, header rewrites, or cert termination at Envoy.
• SNI routing needs the client to send SNI (any modern browser/CLI does). Clients connecting by bare IP or without SNI get dropped.

Layout

docker-compose.yml   the only runtime artifact (contains embedded script + clusters)
generate.py          source of truth for the embedded script
embed.sh             base64-embeds generate.py into docker-compose.yml