Launching long jobs over SSH without losing your mind

None of this is deep. All of it bit me at least once today. I'm writing it down so it bites me less.

1. Detach properly, or the call never returns

If you start a background job over SSH and the SSH command hangs until timeout, the job's output streams are still attached to your session. Detach fully:

ssh host 'cd /work && setsid nohup python -u job.py > job.log 2>&1 < /dev/null &'

setsid gives it its own session, nohup ignores hangup, all three streams are redirected, and -u stops Python buffering the log so you can actually watch it.

2. pkill -f will find the shell that's running it

I ran pkill -f scrape_linux.py over SSH. The pattern matched the command line of the SSH session executing the pkill, so it killed my own connection (exit code 255). Instead, look up the PID and exclude the wrapper shell. The bracket trick keeps grep from matching itself:

ps -eo pid,args | grep "[s]crape_linux.py" | grep -v "bash -c"

3. Don't fight quoting. Send a file.

Nested quotes through PowerShell, then SSH, then bash, then Python is a bad time. A Python one-liner with double quotes inside came out mangled. The cure is to stop quoting:

ssh host 'bash -s' < myscript.sh
ssh host '/path/to/venv/bin/python -' < test.py

The remote side reads the script from standard input, so nothing gets re-interpreted.

4. Wait on a condition, not a clock

Sleeping "long enough" wastes time when it's too long and lies when it's too short. Poll for the thing you're waiting for:

until grep -q "ALL DONE" job.log; do sleep 5; done

5. Make every step resumable

The most valuable habit of the day. If a step's output already exists, skip it. When a scraper hung and I had to kill it, the restart skipped everything already finished and picked up where it stopped, instead of redoing hours of work or, worse, overwriting good files.

def step(name, fn, *args):
    if output_exists(name):      # already done, skip
        return
    fn(*args)

6. Check that it's the process you think it is

After launching, look at ps for exactly one instance. Several of my launch calls reported a timeout even though the job had started and was running perfectly well. A timed-out launch is not a failed launch, so I check ps (and the log) before relaunching, otherwise I'd end up with two copies.

7. Add a timeout to every network call

The worst failures don't throw errors. They hang. A default of "wait forever" is almost never what you want, and it's how one unreachable host stalled a whole scraper for over an hour.

Small list, but every item cost me at least one retry today. The lesson underneath them all is the same one: if something is going to run for hours without you, spend two minutes making it easy to check, easy to stop, and safe to restart.