Infrastructure Runbook:
Production Stack (Protostar Development)
1. System Overview & Topology
- OS: Debian 12 (Bookworm) minimal kernel
- Primary Node: 4 vCPU, 16GB RAM (Hetzner Dedicated / Cloud
VPS)
- Container Runtime: Rootless Podman with Systemd Quadlets
- Database: PostgreSQL 16 (Primary with daily offsite
encrypted dumps)
- Edge Proxy: Nginx with strict SSL TLSv1.3 configuration &
Let's Encrypt
2. Emergency Incident Response
High CPU / Memory Exhaustion:
- SSH into bastion host using Ed25519 key + MFA.
- Run container resource inspection:
podman stats
--no-stream
- Inspect system logs:
journalctl -u
webapp.service -n 100 --no-pager
- If necessary, restart isolated service:
systemctl
--user restart webapp.service
3. Backup Verification Protocol
- Automated nightly backups execute at 02:00 UTC.
- Verification checksum runs automatically against S3-compatible remote storage bucket.
# Infrastructure Runbook: Acme Corp Production Stack (Protostar Development)
## 1. System Overview & Topology
- **OS:** Debian 12 (Bookworm) minimal kernel
- **Primary Node:** 4 vCPU, 16GB RAM (Hetzner Dedicated / Cloud VPS)
- **Container Runtime:** Rootless Podman with Systemd Quadlets
- **Database:** PostgreSQL 16 (Primary with daily offsite encrypted dumps)
- **Edge Proxy:** Nginx with strict SSL TLSv1.3 configuration & Let's Encrypt
## 2. Emergency Incident Response
### High CPU / Memory Exhaustion
1. SSH into bastion host using Ed25519 key + MFA.
2. Run container resource inspection: `podman stats --no-stream`
3. Inspect system logs: `journalctl -u webapp.service -n 100 --no-pager`
4. If necessary, restart isolated service: `systemctl --user restart webapp.service`
## 3. Backup Verification Protocol
- Automated nightly backups execute at 02:00 UTC.
- Verification checksum runs automatically against S3-compatible remote storage bucket.
Incident Response Runbook:
Security Compromise Protocol
1. Severity Classification & Triage
- Sev 1 (Critical): Active unauthorized root access, data
exfiltration, ransomware, or complete service outage.
- Sev 2 (Major): Repeated failed brute-force attacks
succeeding on non-privileged endpoints or anomalous outbound traffic.
- Sev 3 (Moderate): Web app scanning, suspicious log
entries without confirmed impact, or expired certificate alerts.
2. Phase 1: Containment & Isolation (First 15 Minutes)
- Isolate Node: Apply strict firewall rules blocking
inbound/outbound traffic except management IP (
sudo ip link set
dev eth0 down).
- Preserve RAM/State: Capture active connections and
process tables before rebooting.
- Revoke Credentials: Immediately rotate all SSH keys, DB
passwords, and API tokens.
3. Phase 2 & 3: Forensic & Eradication
- Review auth logs (
journalctl -u ssh
--since "2 hours ago").
- Run package integrity checks (
sudo debsums
-s).
- For Sev 1 breaches, never trust a compromised host—provision a fresh immutable instance from
clean IaC.
# Incident Response Runbook: Security Compromise & Anomaly Protocol (Protostar Development)
## 1. Severity Classification & Triage
- **Sev 1 (Critical):** Active unauthorized root access, data exfiltration, ransomware encryption, or
complete service outage.
- **Sev 2 (Major):** Repeated failed brute-force attacks succeeding on non-privileged endpoints, or
anomalous outbound traffic.
- **Sev 3 (Moderate):** Web application scanning, suspicious log entries without confirmed impact, or
single expired cert alerts.
## 2. Phase 1: Containment & Isolation (First 15 Minutes)
1. **Isolate the Affected Node (Do Not Power Off):** Apply strict firewall rules blocking
inbound/outbound traffic except management IP.
- For on-premise: `sudo ip link set dev eth0 down`
2. **Preserve System Memory & State:** Capture active connections and process tables:
- `sudo ss -tulpn > /var/log/incident_netstat_$(date +%s).log`
- `sudo ps auxf > /var/log/incident_processes_$(date +%s).log`
3. **Revoke Active Credentials:** Immediately rotate all SSH keys, database passwords, and API tokens.
## 3. Phase 2: Forensic Analysis & Log Inspection
1. **Inspect Authentication Logs:** Review auth logs for unauthorized sudo usage or timing anomalies:
- `journalctl -u ssh --since "2 hours ago" --no-pager`
2. **Scan for Unauthorized Binaries:** Run integrity checks via package manager:
- `sudo debsums -s || rpm -Va`
3. **Inspect Rootless Container Namespaces:** Check for unexpected running containers:
- `podman ps -a --no-trunc`
## 4. Phase 3: Eradication & Remediation
1. **Rebuild vs. Clean:** For Sev 1, never trust a compromised host; provision a fresh immutable
instance from clean IaC templates.
2. **Patch the Root Vector:** Close the exploited vulnerability and verify patch levels.
3. **Verify Log Pipeline Integrity:** Confirm immutable logging components (`Vector` -> `immudb`)
successfully recorded audit logs.