Technical Debt: The Hard Way

Late 2025, I was rushing to set up and deploy my capstone undergraduate final project. The application heavily relied on complex image text-recognition packages. With the deadline breathing down my neck and no budget for proprietary APIs, I turned to open-source packages.

Frustrated, I started vibe coding just to ship it.

I blindly chained Python packages together, stacking dependencies I barely understood. The project passed the presentation, but on my production server, I had unknowingly planted a ticking time bomb.

Symptoms: A Sluggish Server

A few days later, I received an ominous email from my hosting provider: VPS SSH traffic limit reached. My server was saturated.

SSH was painfully slow — almost 2 seconds per keystroke. I quickly ran a process snapshot:

ps aux

There it was — a rogue process pinned at 100% CPU, eating my resources.

The Battle

I tried to kill it:

pkill <process_name>

Then I brought down the container stack, restarted Docker, and logged out, thinking the problem was solved.

Hours later, the alert triggered again. The process had resurrected. I went nuclear:

kill -9 <PID>

It kept coming back. I killed it multiple times with no success.

I started isolating projects one by one until only my latest undergraduate project remained. Even after running:

docker compose down <project_id>

the ghost process persisted.

Exhausted, I turned to AI for help and ended up running destructive commands.

The Innocent Fatal Flag

I executed this command:


bash
docker compose down -v <container_id> && docker rmi <image_id>

The server instantly became responsive. Resource usage dropped to zero. Feeling victorious, I applied the same to all Docker instances, restarted my client projects, and went to sleep.

The next morning, a client called:

"I can't log in… I had to create an account again… and all of our data is gone!"

My stomach dropped. All database records across client containers were wiped.

What Really Happened

The -v (or --volumes) flag in docker compose down -v deletes the persistent volumes attached to the containers. All my production databases lived in those volumes on the host filesystem.

I had assumed Docker operations wouldn’t touch the raw data on disk. I was wrong.

Deep Dive

1. Inspecting the Damage

ps aux

a → all users
u → user-oriented format (CPU, MEM, owner)
x → processes without a controlling terminal

2. Process Termination

pkill <name> → sends SIGTERM (graceful shutdown)
kill -9 <PID> → sends SIGKILL (instant, no cleanup)

The process kept reviving because of Docker’s restart: always policy or a systemd service.

3. The Dangerous Flag

docker compose down -v removes containers and the named/anonymous volumes declared in your docker-compose.yml. This permanently deletes the data directories on the host.

Post-Mortem & Lessons Learned

This painful experience forced a complete shift in how I operate:

Isolated Infrastructure: Experimental/student projects never share hardware with client production systems.
Automated Backups: Regular off-site backups via cron jobs.
Mechanical Sympathy: I now deeply understand every tool, flag, and dependency I use.

Real engineering is about intentionality. Like a civil engineer who analyzes every material in a bridge, I now choose every dependency, command, and architecture with care.

Sam 🙂