The batch job nobody knew

Ask for the architecture diagram and you get the request path: users, services, database. Ask what runs at 2 a.m. and the room goes quiet. The batch layer is every estate's dark matter: invisible on diagrams, heavy in consequence, and discovered mainly by its absence.

The record for what dormant, unowned code costs belongs to Knight Capital, where a leftover code path and a partial deploy burned $460 million in forty-five minutes; the SEC's order reads like an inventory finding published four years too late. Your version is smaller and closer: the reconciliation job that stopped running in April, noticed in June, during an audit.

Walk the batch floor the way we do:

1. Harvest every scheduler. cron, Task Scheduler, the job tables inside the database, the ERP's own scheduler, CI systems moonlighting as cron, and the vendor appliances with their own calendars. Estates average four schedulers; the inventory usually lists one.
2. For each job: what does it read, what does it write, who consumes the output, what happens if it runs twice, what happens if it never runs again. The last two questions are the whole risk profile, and nobody has written the answers down.
3. Find the chains. Job C assumes job B finished, which assumed job A's file landed. The chain has no owner, no monitor, and a timing assumption from 2017 that daylight saving violates twice a year. Every estate has one chain whose failure mode is "payroll is wrong."
4. Check who gets told. A batch job whose failure email goes to a departed employee's mailbox is not monitored. It is mourned in advance.

Then triage into the standard three piles. Owned and needed: give it monitoring, an owner, a runbook line, and a place on the inventory. Needed but duplicated or deranged: fold it into the sanctioned pipeline during the migration it will otherwise sabotage. Unknown: scream test with a long fuse. Disable, wait a full business cycle including month-end and quarter-end, because batch consequences travel on the accounting calendar, not the sprint calendar.

The migration angle is the reason this entry exists: batch jobs are where cutovers go to die. The request path gets rehearsed; the 2 a.m. layer gets discovered Sunday morning when the file did not land. Put the batch inventory in week one of any migration, and give the chains the same parallel-run treatment as the request path. The jobs nobody knew are only dangerous while nobody knows them. Counting them is most of the cure.