The problem
Every month the compliance team had to show, for each subcontract on the program, whether the workforce on site met the program's participation goals. The evidence lived in a compliance portal that only produces reports one at a time, through a browser. Someone would log in, pull the report for each subcontract, read the figures out into a spreadsheet, check them against the current contractor roster, and work out the shortfall against each contractual target. Then the package was filed to the shared drive, the summary rows were pushed into the reporting database, and an email went out to say it was done.
That took days of one person's month, and it was the kind of work where a slip is invisible. A stale roster, a report pulled for the wrong period, a value typed into the wrong row: nothing fails loudly, the package just goes out wrong. The job was too repetitive and too long for anyone to do perfectly twelve times a year.
The constraints that shaped it
I wrote down the rules before I wrote the code, and they held for the whole project.
- Refuse rather than guess. When a contractual figure is missing, the system does not infer one from a filename, a similar contract, or last month. It highlights the cell, marks that work order for review, and carries on with the rest. A review result is the system working correctly.
- Earned completion. A subcontract counts as done only when the real report exists on disk. Nothing is marked complete because a step seemed to work.
- Verify what a check actually looks at. A log line records what was attempted; the delivered thing is the only evidence. Check the file, the sent item, the row.
- One interruption, only on change. The team should hear from the system when a new subcontract needs an input it cannot know, and otherwise never.
- Assume the desktop is flaky. The portal hangs, the browser wedges, the spreadsheet application drops its connection mid-save, the shared drive serves the old file for a minute after a write. Every one of those has to be survived, not merely reported.
What I built
The pipeline. A runner takes each work order and, for each subcontract in it, drives the portal in a browser it controls, pulls the report, converts it to a workbook sheet, reconciles the sheet against the roster, and computes the goal shortfalls from verified contractual inputs. When a work order finishes clean it publishes: files to the shared drive, rows to the database, a completion email. Forty-plus subcontracts, about eighty minutes end to end.
The verification layer. The part that took longest. The source workbook is chosen by a signed pointer that states which file is authoritative, after two incidents where the "newest matching file" turned out to be a stale partial. Saves to the shared drive are confirmed by waiting for the file hash to change and then settle, not merely to stop changing. Every completed subcontract is checked against a real artifact, and the browser lane is released in a way the process can verify, so a hung page never blocks the next one.
A liveness dashboard. A small sidecar heartbeats run state to a phone-sized page: which work order, which subcontract, the ETA, free memory, and whether the process is alive at all. A watchdog restarts the sidecar if it dies and writes the shutdown record that a killed process cannot write for itself. An unattended system that cannot be glanced at is not unattended; it is unwatched.
Rollback-able promotions. Every change is a numbered build. A certification gate checks the candidate is line-identical to its parent apart from its own change, audits that every name the tests fake still exists in the real source, and runs the candidate's suite. Promotion records the new production build and saves the outgoing one with its own rollback script. Rolling back is promoting an older folder; fifty-odd builds later, that chain is the project's history.
Agent test mode, and the trap it created. To test safely, a mode redirects every write into a sandbox and refuses production paths outright. It let me run the whole cycle dozens of times without risk. It also let a class of defect hide, which I come back to below.
Results
The first fully clean pass on a single work order completed every subcontract with zero exclusions and zero orphaned processes. The first production run across all work orders published every package, wrote every row, and sent the completion email with a person watching but never touching it. Since then production runs are supervised rather than performed. The team's monthly effort dropped from several days to reading a completion email and, occasionally, confirming one number a new subcontract needs: somewhere between twenty and forty hours a month returned.
What went wrong, and what I learned
The verification code was where the bugs were. Every significant error on this project, six in two weeks at the worst stretch, was in a check rather than in the work. A verifier reported a repair had failed because it hashed the shared drive's stale copy. A "right sender account" check read the log instead of the sent item, and the assignment had silently done nothing for weeks. The habit that came out of it: when a check surprises you, verify what the check is looking at before believing what it says.
A test mode hid production defects. Test-only wrappers meant the code I was reading was not always the code that ran; I nearly widened timings in a path that never executes in production. Now I check the production log for the feature's marker before tuning anything, and every certification ends with a line I take literally: passing the gate is not evidence the feature works. Prove it fires.
Stub contracts. A stub is a hypothesis about the application. One certification passed every test on a feature that did nothing, because the tests faked a name the application had stopped using. The fix is structural: an audit that asserts every faked name exists in the real source, run on every candidate, so the two can never drift apart quietly.
The deeper lesson is that "refuse to guess" applies to the system's checks as much as to its outputs. A missing value blocks; a wrong value just calculates. The system was heavily defended against the first; the defences against the second had to be built deliberately.
How it carried into the public products
RateWatch and Bid Digest both came out of this project, not as code but as rules. Both read real public sources and never infer. RateWatch treats a changed document as the signal rather than trusting a parse, because "changed and stable" was the only check that ever held. Both tell a person exactly once, only when something has changed, and each run leaves a record that proves it ran. If a source cannot be read, the run says so and stops rather than publishing a confident blank.