Room 49

ETCD OPERATIONS

The Vault of All Knowledge

etcd is the brain of Kubernetes - every object, every secret, every config lives here.
Lose etcd, lose your cluster. Mastering backup and restore is essential.

📜 Mission Briefing

Explore etcd in the Kind cluster - find the etcd pod and its data directory.
Take an etcd snapshot backup using etcdctl.
Verify the snapshot integrity.
Practice restoring from the snapshot to a new data directory.
Inside the snapshot, a saved secret contains the password: etcdback.

Key Commands

Command	Purpose
`kubectl get pods -n kube-system -l component=etcd`	etcd pod
`kubectl exec -n kube-system etcd-* -- etcdctl ...`	etcdctl commands
`etcdctl snapshot save /tmp/backup.db`	Take snapshot
`etcdctl snapshot status /tmp/backup.db --write-table`	Verify snapshot
`etcdctl snapshot restore /tmp/backup.db`	Restore snapshot

How etcd Works in Kubernetes

# etcd is a distributed key-value store
# Kubernetes stores ALL cluster state in etcd:
# - Pods, Services, Deployments, ConfigMaps, Secrets
# - RBAC rules, Namespaces, Custom Resources
# - Leases, Events, Endpoints

# etcd data path: /registry/{resource}/{namespace}/{name}
# Example: /registry/pods/default/my-pod

# etcd in Kind cluster
kubectl -n kube-system get pod etcd-escape-cluster-control-plane -o yaml
# Key flags:
#   --data-dir=/var/lib/etcd
#   --cert-file=/etc/kubernetes/pki/etcd/server.crt
#   --key-file=/etc/kubernetes/pki/etcd/server.key
#   --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

# Taking an etcd snapshot
# Method 1: From inside the etcd pod
kubectl exec -n kube-system etcd-escape-cluster-control-plane -- \
  etcdctl snapshot save /var/lib/etcd/snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify snapshot
kubectl exec -n kube-system etcd-escape-cluster-control-plane -- \
  etcdctl snapshot status /var/lib/etcd/snapshot.db \
  --write-table

# Output:
# +----------+----------+------------+------------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | 3fba... |       28 |        320 |    1.2 MB  |
# +----------+----------+------------+------------+

# Restoring from snapshot
# 1. Stop kube-apiserver (move static pod manifest)
# 2. Restore to new data directory
etcdctl snapshot restore /tmp/snapshot.db \
  --data-dir=/var/lib/etcd-restored

# 3. Update etcd pod manifest to use new data-dir
# 4. Restart etcd and kube-apiserver

# In production:
# - Regular scheduled backups (cron)
# - Store backups off-cluster (S3, GCS)
# - Test restores periodically
# - etcd cluster should have odd number of members (3, 5, 7)

# Reading etcd directly (bypassing API server)
kubectl exec -n kube-system etcd-escape-cluster-control-plane -- \
  etcdctl get /registry/secrets/default --prefix --keys-only \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# etcd encryption at rest
# Kubernetes can encrypt secrets in etcd
# EncryptionConfiguration resource defines providers:
# - aescbc, aesgcm, secretbox (encryption)
# - identity (no encryption - default)

Hints

Hint 1

# First, create a secret that holds the password
kubectl create secret generic escape-etcd --from-literal=password=etcdback
# Then take a snapshot

Hint 2

# Read the secret from the snapshot
etcdctl snapshot status /var/lib/etcd/snapshot.db --write-table

Hint 3

The password is etcdback.

🔓 Unlock Room 50

openssl enc -aes-256-cbc -d -a -pbkdf2 \
        -in ../room_50/README -out ../room_50/README.txt -pass pass:PASSWORD
cat ../room_50/README.txt