Skip to content

Room 52

CHAOS ENGINEERING

The Resilience Forge

  • Chaos engineering is the discipline of experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions.
  • You’ll break things on purpose - and prove your systems can survive.

📜 Mission Briefing

  1. Deploy a microservices app with 3 components: frontend, backend, cache.
  2. Create a chaos experiment: randomly kill pods in the backend Deployment.
  3. Set up PodDisruptionBudgets (PDBs) to protect minimum availability.
  4. Inject network latency using a NetworkPolicy that blocks traffic temporarily.
  5. Observe how the system recovers (or doesn’t) from each failure.
  6. Tune liveness/readiness probes to improve resilience.
  7. The password is found when the system self-heals after chaos: chaos52x.
Key Commands
Command Purpose
kubectl delete pod -l app=backend --force Kill backend pods
kubectl get pdb List PodDisruptionBudgets
kubectl get events --sort-by=.lastTimestamp Recent events
kubectl top pods Resource usage
kubectl describe pod <pod> Pod health details
kubectl logs -f -l app=frontend Stream frontend logs
Microservices Stack
# backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: nginx:1.25
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 3
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
PodDisruptionBudget
# backend-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: backend-pdb
spec:
  minAvailable: 2           # At least 2 pods must always be running
  selector:
    matchLabels:
      app: backend
Chaos Injection Script
#!/bin/bash
# chaos-test.sh - Inject failures and verify resilience

NAMESPACE="default"
APP="backend"

echo "🔥 Starting Chaos Experiment..."
echo "================================"

# Step 1: Verify baseline
READY_BEFORE=$(kubectl get pods -l app=$APP --no-headers | grep Running | wc -l)
echo "📊 Baseline: $READY_BEFORE pods running"

# Step 2: Kill random pods
echo "💀 Killing random backend pods..."
kubectl delete pod -l app=$APP --field-selector=status.phase=Running --force --grace-period=0 2>/dev/null &

# Step 3: Immediately check disruption budget
sleep 2
PDB_STATUS=$(kubectl get pdb backend-pdb -o jsonpath='{.status.disruptionsAllowed}')
echo "🛡️  PDB allows $PDB_STATUS more disruptions"

# Step 4: Wait for self-healing
echo "⏳ Waiting for self-healing..."
kubectl rollout status deploy/$APP --timeout=60s

# Step 5: Verify recovery
READY_AFTER=$(kubectl get pods -l app=$APP --no-headers | grep Running | wc -l)
echo "📊 After chaos: $READY_AFTER pods running"

if [[ "$READY_AFTER" -ge 2 ]]; then
  echo "✅ System survived chaos! Password: chaos52x"
else
  echo "❌ System did not recover. Check PDB and probe configuration."
fi
Network Chaos - Isolate a Service
# network-chaos.yaml - Block all traffic to backend temporarily
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: chaos-isolate-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
    - Egress
  # Empty ingress/egress = deny all traffic

# Apply network chaos
kubectl apply -f network-chaos.yaml
echo "🔇 Backend isolated from all network traffic"

# Watch frontend logs - it should report errors
kubectl logs -l app=frontend --tail=10

# Remove chaos after 30 seconds
sleep 30
kubectl delete -f network-chaos.yaml
echo "🔊 Network restored"
Recovery Observations
# Watch pods recover in real-time
kubectl get pods -l app=backend -w

# Check events for restart patterns
kubectl get events --sort-by=.lastTimestamp --field-selector reason=Killing

# Verify PDB was respected
kubectl describe pdb backend-pdb

💡 Hints

  • PDBs only protect against voluntary disruptions (like kubectl delete), not node crashes.
  • Liveness probes restart unhealthy containers; readiness probes remove them from Service endpoints.
  • initialDelaySeconds is critical - too short and you’ll kill pods that are still starting.
  • Network policies are additive - once any policy selects a pod, only explicitly allowed traffic passes.

🔑 Password

Deploy the stack with PDB, run the chaos script, and the password appears when pods self-heal.