Cluster Bursting Demo: anaeem LB → virt + anaeem burst

A loadgen pod on anaeem drives traffic at a local nginx LB. The LB splits between anaeem's burst app and virt's on-prem app, weighted 5:1 in favor of anaeem. KEDA on anaeem scales the burst app 0 → 5 based on the request rate counted by virt's nginx-prometheus-exporter sidecars.

                         loadgen (on anaeem)
                                │
                                ▼
                          nginx LB (on anaeem)
                          ┌──────┴──────┐
                          │             │
                  weight 5│             │weight 1
                          ▼             ▼
                  ┌──────────────┐   anaeem ────► virt app (nginx + exporter sidecar)
                  │ anaeem burst │   apps          counts requests it serves
                  │ app (0..5,   │   router        ▲
                  │ KEDA scaled) │                 │
                  └──────────────┘                 │ scrape
                                             virt Prometheus
                                                  ▲
                                                  │ public edge-TLS Route
                                            KEDA on anaeem

All cross-cluster traffic originates from anaeem (which can resolve virt's apps router). This is required in this lab — virt cannot resolve anaeem's apps router, so it cannot push, scrape, or call anaeem directly.

Each app pod serves an HTML page showing its pod name + cluster name (blue=virt, green=anaeem) so you can see where each request lands.

Files

virt/
  01-namespace.yaml      demo namespace
  02-app.yaml            app Deploy (nginx + nginx-prometheus-exporter sidecar)
                         + Service (ports 8080 + 9113) + Route
  03-prometheus.yaml     Prometheus (pod-discovery scrape on :9113) + Route

anaeem/
  00-keda-operator.yaml  Custom Metrics Autoscaler operator + KedaController
  01-namespace.yaml      demo namespace
  02-app.yaml            burst app Deploy (replicas: 0) + Service
  03-lb.yaml             nginx LB (weighted upstreams) + Service + edge-TLS Route
  04-loadgen-pod.yaml    pod that loops curl against lb.demo.svc:80
  05-scaledobject.yaml   KEDA ScaledObject querying virt's Prometheus

scripts/
  00-setup-virt.sh       apply virt; record Prometheus URL
  01-setup-anaeem.sh     install KEDA if missing; apply burst app, LB, ScaledObject
  02-drive-load.sh       start loadgen pod
  03-stop-load.sh        delete loadgen pod
  99-teardown.sh         clean both clusters

Run

# 1. Log in once and name the contexts virt / anaeem
oc login -u <user> -p '...' https://api.virt.na-launch.com:6443
oc config rename-context $(oc config current-context) virt
oc login -u <user> -p '...' https://api.anaeem.na-launch.com:6443
oc config rename-context $(oc config current-context) anaeem

./scripts/00-setup-virt.sh
./scripts/01-setup-anaeem.sh
./scripts/02-drive-load.sh

# In another shell — watch burst + LB distribution:
watch -n 2 'oc --context anaeem -n demo get scaledobject demo-burst; \
  oc --context anaeem -n demo get hpa keda-hpa-demo-burst; \
  oc --context anaeem -n demo get pods -l app=prometheus-example-app'

LB="https://$(oc --context anaeem -n demo get route lb -o jsonpath='{.spec.host}')"
for i in $(seq 1 60); do curl -sk "$LB/" | grep -oE 'from <b>[a-z]+</b>'; done | sort | uniq -c
# or open $LB in a browser — green = anaeem, blue = virt

./scripts/03-stop-load.sh
./scripts/99-teardown.sh

MANUAL-v2.md walks the same flow with every oc command spelled out (no scripts) and assumes KEDA is already installed on anaeem.

How the pieces fit

1. App pods on virt. nginx serves the blue HTML page on / and exposes /stub_status on localhost. A nginx-prometheus-exporter sidecar scrapes that and exposes nginx_http_requests_total on :9113. Every request virt serves is counted.

2. Prometheus on virt. Discovers app pods via the Kubernetes API and scrapes their exporter port. Exposes /api/v1/query on a public edge-TLS Route — no auth, the metric isn't sensitive.

3. Burst app on anaeem. Same nginx + HTML pattern, green, replicas: 0. No exporter — virt is the metric source.

4. LB on anaeem. nginx with two upstreams:

   • prometheus-example-app.demo.svc:8080 weight 5 (local burst pods)
   • prometheus-example-app-demo.apps.virt.na-launch.com:80 weight 1 (virt)

   When anaeem has 0 burst pods, its Service has no endpoints so nginx falls back to virt via proxy_next_upstream. Once KEDA scales burst pods up, ~5/6 of traffic stays on anaeem; virt sees only the 1/6 trickle plus spillover during cooldowns.

5. KEDA ScaledObject on anaeem. prometheus trigger pointed at virt's Prometheus Route, querying sum(rate(nginx_http_requests_total[1m])). Above activationThreshold=2 KEDA scales 0→1; above threshold=5 it scales toward maxReplicaCount=5.

Why this topology

The lab has asymmetric DNS: virt cannot resolve anaeem's apps router, but anaeem can resolve virt's. The previous version with the LB on virt and an exporter on the LB worked when both routes were reachable, but breaks when the virt→anaeem direction is unavailable.

This version puts every cross-cluster initiator on anaeem:

• LB upstream call → virt Route ✓
• KEDA Prometheus poll → virt Route ✓
• Loadgen → LB Service (in-cluster, no DNS issue) ✓

Nothing on virt ever calls anaeem.

Out of scope

• Inter-cluster TLS (the LB-to-virt hop is plain HTTP over the apps router).
• Hardening the virt Prometheus Route (anyone who can reach *.apps.virt... can read the rate metric).
• HPA on virt — virt stays at fixed 2 replicas; only anaeem scales.