Telling Technology · Learning in public
All episodes
EP.46 — FAKE PROJECT
The nightly audit · learning to tell plans from projects

My AI audits every project every night — and just learned to spot a fake one

Every night an agent sweeps every project folder I own and decides what's alive, what's stale, and what should quietly drop off tomorrow's report. Last night two folders sailed straight through as real projects — because they had a beautifully written plan file. Behind it, their content folders held one thing: a .gitkeep.

Two project folders side by side: a beautifully written plan file next to a content folder holding only a .gitkeep
⌁ 20+ folders swept nightly · 2 false positives caught · 1 filter rule fixed
It starts with the housekeeping nobody sees →
01 The setup

There's a janitor under the audits

You've seen the loud version: EP.25 fanned out 20 agents at 01:00 to audit every repo on the machine, and EP.30 had a fresh model judge three competing rewrites of my own system. Underneath both sits something much less glamorous — a master-architect agent, fired by architect-nightly.ps1, that walks every project folder in the workspace and answers one boring question per folder: is this thing actually alive? Alive ones get watched. Stale ones get flagged. Dead ones drop off future reports so the morning brief stays short.

architect-nightly.ps1 — the housekeeping pass
sweep: every project folder in the workspace ask: alive / stale / drop from future reports signal: a real, non-placeholder plan file output: tomorrow's project list
The unglamorous layer under the big audits. It doesn't find bugs — it decides what's even worth auditing tomorrow.
The nightly agent fan-out: parallel agents spinning up at 01:00 to audit every repo on the machine
EP.25 · the loud layer
the 01:00 fan-out — the janitor pass decides what it even looks at
02 The blind spot

A perfect plan and nothing behind it

The "is this real?" check was built to catch obvious template junk — leftover {{ }} tokens, a TODO: describe this project line, a file that's clearly never been touched. Two folders had none of that. Their plan files were fully written, specific, and genuinely good: purpose, structure, commands, conventions. So the filter waved them through as real work. Then you open the content folders they describe and find exactly one file in each: .gitkeep. A placeholder whose entire job is to hold an empty folder open.

the check that passed them
> is this a real project? plan file: present, 90 lines, no {{ }} tokens ✓ placeholder text: none ✓ verdict: REAL content/: .gitkeep ← never looked
Every rule it had, it passed. The filter measured the description of the project and never once opened the project.
A well-written plan file on the left, and on the right the content folders it describes, each holding only a .gitkeep file
2 folders · both passed
a great README is not a shipped project
03 The fix

Make it look inside the box

The change is one tightened rule in the discovery filter: a project only counts as real when its content folders hold more than a keep-file. Not "does a plan exist" — "does anything exist that the plan promised". It's a small diff and a slightly harsher machine: several folders that used to report as healthy now report honestly as scaffolding. That's the point. And it happened twice in one night — a second repo's plan file had been flagged as a stale leftover scaffold two nights running before it finally got rewritten to describe what the repo actually does.

master-architect.md — discovery filter
- before: plan file exists && no placeholder junk -> REAL + after : plan file exists && no placeholder junk && content folders hold MORE than a keep-file -> REAL result: 2 false positives reclassified as scaffolding
A well-written plan is now evidence of intent, not evidence of work. Only files the plan promised count as work.
A before/after diff of the discovery-filter rule in master-architect.md, adding a content-folder emptiness check
one rule, two lines
the whole fix — before / after
⚠️

This makes the report meaner about my own work

The two folders it caught are mine. The filter didn't find someone else's fake project — it found the gap between what I planned and what I built, and now it will keep saying so every night until one of us fixes it. A .gitkeep is a promise; the audit just stopped accepting promises as delivery.

04 The receipts

Four screens from the night it learned the difference

The filter rule, the empty folder that broke it, the fix-plan doc working through its own false positives, and the decision logged so the next agent inherits the reasoning instead of re-deriving it. Tap any image to enlarge it and read the exact prompt that drew it.

A file tree showing a rich plan file above content folders that each contain only a .gitkeep
.gitkeep, and nothing else
A fix-plan document listing audit false positives, each marked verified-safe or needs-a-real-decision
resolve, or defer honestly
Before and after of a plan file flagged as a stale scaffold two nights running, then rewritten
flagged twice, then fixed
A decisions log entry recording the tightened discovery-filter rule, its cost, and a one-line rollback
logged, not guessed
Caught by the audit that runs on this very workspace

A plan is not a project

That's the whole lesson, and it's the same line that decides whether one of these nightly briefs becomes a real page or stays an idea forever. The AI that sweeps 20+ of my folders every night just learned the exact distinction my own show runs on: written down and built are not the same status.

A verdict board: two folders reclassified from real project to scaffolding, with the new rule stated underneath — content folders must hold more than a keep-file
2 folders reclassified · 1 plan file rewritten · rule change logged 2026-07-24
06 Steal this

The four-line rule that stops an AI grading a README

If anything you run sweeps folders and decides what's "active" — a dashboard, a nightly audit, a project picker — this is the rule that keeps a good description from counting as good work. Everything in this episode is free and open — clone it, run it, make it yours.

agent master-architect runner architect-nightly.ps1 rule content > keep-file doc fix-plan + decision log
the rule # Is this folder a real project, or just a well-written plan? # 1. plan file exists and isn't template junk ({{ }}, "TODO: describe") -> necessary, NOT sufficient # 2. open the content folders the plan promises # 3. if every one of them holds only a keep-file (.gitkeep) -> SCAFFOLDING, not a project # 4. log the reclassification with a reason — an audit that changes its mind silently is worse than one that never changes it
↧ soon

The master-architect pack is being scrubbed of real folder names and machine paths before release. Comment ARCHITECT on the post and the bot DMs you the moment it's public.

Next episode

Next: the folders it just demoted — build them, or bury them?

The audit is now honest about two of my own folders, which means it will nag me about them every single night until something changes. The next episode is the decision it forced: ship the thing the plan describes, or delete the plan and admit it was never a project. One of those is a lot harder to film.

A dark shot of two demoted project folders on a nightly report, each waiting on a build-or-bury decision
drops soon · follow so you don't miss it