Automating a Render Queue with Python and systemd

My render node used to be driven by a folder of .bat files and hope. This post walks through the small Python daemon that replaced them — about 300 lines, no framework, running under systemd on Debian.

Requirements

• queue render jobs from any machine on the LAN;
• survive renderer crashes without losing the queue;
• notify me when a job finishes or fails.

The queue

Jobs are plain JSON files dropped into a watched directory — no database, no broker. The filesystem is the queue, and mv is atomic:

from pathlib import Path
import json, shutil, subprocess

QUEUE = Path("/srv/render/queue")
ACTIVE = Path("/srv/render/active")
DONE = Path("/srv/render/done")

def next_job() -> Path | None:
    jobs = sorted(QUEUE.glob("*.json"))
    return jobs[0] if jobs else None

def run(job_file: Path) -> int:
    job = json.loads(job_file.read_text())
    active = ACTIVE / job_file.name
    shutil.move(job_file, active)          # atomic claim
    proc = subprocess.run(
        [job["renderer"], "-scene", job["scene"], "-out", job["output"]],
        capture_output=True, text=True, timeout=job.get("timeout", 14400),
    )
    shutil.move(active, DONE / job_file.name)
    return proc.returncode

Crash recovery falls out of the design: on startup, anything left in active/ is moved back into queue/ and simply runs again.

Supervision

systemd does the babysitting — restart on failure, journal logging, resource limits:

[Unit]
Description=Render queue worker

[Service]
ExecStart=/usr/bin/python3 /srv/render/worker.py
Restart=on-failure
RestartSec=10
MemoryMax=48G

[Install]
WantedBy=multi-user.target

What I deliberately left out

• A web UI. ls /srv/render/queue answers most questions.
• Priorities. Renaming a file to sort first is the priority system.
• A message broker. The filesystem has been reliable for decades.

Boring technology is a feature. Every dependency you don’t add is one that can’t break the night before a deadline.

The full script lives in the Lab with setup notes.