Skip to content

Room 36

KUBECTL DEBUG

The Diagnostic Lab

  • kubectl debug lets you attach ephemeral debug containers or create pod copies for troubleshooting.
  • A distroless pod has no shell - debug it with an ephemeral container.

📜 Mission Briefing

  1. Apply distroless-pod.yaml - runs a Go binary in a distroless image (no shell).
  2. Try kubectl exec - it will fail (no /bin/sh).
  3. Use kubectl debug to attach a debug container: 
    kubectl debug distroless-pod -it --image=busybox --target=main
  4. From the debug container, read /proc/1/environ to find the password.
  5. The password is debugpod.
Key Commands
Command Purpose
kubectl debug POD -it --image=IMG Attach ephemeral container
kubectl debug POD -it --image=IMG --target=CONTAINER Share PID namespace
kubectl debug POD --copy-to=NEW-POD --image=IMG Debug copy
kubectl debug node/NODE -it --image=IMG Debug node
kubectl debug
# Ephemeral debug container (shares pod namespaces)
kubectl debug my-pod -it --image=busybox --target=main
# --target=main: share PID namespace with container "main"
# Inside debug container: can see processes of "main"

# Debug copy (create a copy of the pod with modifications)
kubectl debug my-pod --copy-to=debug-pod --image=busybox -it
kubectl debug my-pod --copy-to=debug-pod --set-image=main=ubuntu -it

# Node debugging (creates a privileged pod on the node)
kubectl debug node/worker-1 -it --image=busybox
# Access host filesystem at /host
# chroot /host  # get a host shell

# Ephemeral containers:
# - Added to running pods without restart
# - Share namespaces with the pod
# - Cannot be removed (pod must be deleted)
# - No ports, resources, or probes
# - Great for distroless images
Hints

Hint 1

kubectl apply -f distroless-pod.yaml
kubectl exec distroless-pod -- /bin/sh  # Will fail

Hint 2

kubectl debug distroless-pod -it --image=busybox --target=main -- sh
# Inside: cat /proc/1/environ | tr '\0' '\n' | grep PASSWORD

Hint 3

The password is debugpod.

🔓 Unlock Room 37

openssl enc -aes-256-cbc -d -a -pbkdf2 \
        -in ../room_37/README -out ../room_37/README.txt -pass pass:PASSWORD
cat ../room_37/README.txt