Run a security assessment by hand and you notice pretty quickly how little of it is actual hacking. You agree on the scope, point a tool at it, and wait. You read the output, decide what is worth chasing, run the next tool, and wait again. Recon, then enumeration, then vulnerability analysis, and the whole time you are keeping notes so that days later you can turn the pile into a report someone will actually read. A single authorized external assessment, done manually, can burn the better part of a week. Most of that week is coordination and paperwork, not clever offense.
Here is the thing though: I think almost none of that is judgment work. Agreeing on the scope is judgment. Deciding that an odd result is a real finding is judgment. Nearly everything between those two points is a workflow, and it is roughly the same shape every single time. So I set out to automate that workflow and hand people back most of the week, without losing the things that make an assessment worth trusting in the first place.
The result is ReconForge: a governed, project based platform where AI agents coordinate a scoped security assessment, every command leaves behind evidence, and the final report is built from that evidence rather than from an AI's vibes. Not a chatbot with a shell attached, and definitely not "Kali in the cloud." This is the first post in a series about how I built it.
Let me start at the top: the architecture.
The Trap: "Just Give an AI a Shell"
The lazy version of this idea writes itself. Spin up a Kali container, hand an LLM a shell tool, tell it "go assess this target," and let it loop until it feels done. It demos beautifully. But it also falls apart, I think, the moment you care about any of the things that make an assessment trustworthy.
Where did that finding come from? What exact command produced it? Can you run it again next week and get the same result? If the model hallucinated a vulnerability, how would you even know? An autonomous agent chewing on raw terminal output in a loop gives you confident prose and no receipts. For a tool people are supposed to trust with real targets, that is a non starter in my book.
So the first design decision was a boundary: the AI coordinates and interprets, but the truth lives outside the AI. That single rule shaped everything else.
The Big Idea: Three Layers, Cleanly Separated
ReconForge is split into three layers, each with one job. Think of it like a restaurant. There is the dining room where you sit and order, the kitchen where the cooking is coordinated, and the pantry where the actual ingredients live. Nobody wants the chef inventing what is in the fridge, and nobody wants the diners wandering into the walk in freezer.
Layer A, the Product Layer, is what the user touches. You create a project, define a target and its scope, pick a template, launch a run, and watch it happen live. Agent cards, a timeline of events, an artifact browser, a findings panel, a report at the end. This is the dining room.
Layer B, the Orchestration Layer, is the brain. A central orchestrator loads the template, applies the scope and policy constraints, starts the right agents, supervises them, and correlates what they find. It is the kitchen: it does not chop the vegetables itself, it coordinates the people who do.
Layer C, the Execution and Persistence Layer, is the substrate. This is where commands actually run (a Kali Linux environment reached over SSH) and where everything is written down: PostgreSQL for structured state and an append only event log, and S3 compatible object storage for raw evidence. This is the pantry, and it is the source of truth.
The nice part about this separation is that every piece is swappable. Kali is just the current execution substrate; the backend talks to it through an abstraction, so it could be a local container today and a Kubernetes pod tomorrow without the orchestration layer noticing. The project's truth is never trapped inside the container.
The Objects That Make It Feel Like a Product
A platform needs nouns, not just verbs. ReconForge has a small vocabulary that everything else is built on:
- Project: the engagement workspace, holding scope, targets, and all history.
- Target: an asset in scope (a domain, an IP, a web app URL).
- Template: a preconfigured workflow that decides which agents run and under what constraints. Users pick a template and press go; they do not write commands.
- Run: one execution of a template against a target.
- Finding: an issue with evidence, a severity, and impact enrichment.
- Artifact: raw tool output (stdout, stderr, generated files), immutable and hashed.
- Event: a timeline record of something that happened, which is what powers the live UI.
The whole user journey is just these nouns in a line:
Create Project → Define Target → Select Template → Launch Run → Watch Live → Review Findings → Export Report
Notice what is missing from that flow: a text box where you type nmap flags. Templates define constraints, not commands. That is deliberate. In my view, the person running the assessment should be choosing intent ("passive recon, stay in scope") and letting the system figure out the how.
Agents in a Pipeline, Not a Free-For-All
Layer B does not run one giant do-everything agent. It runs a small team of specialists, each with a bounded job, and it runs them as a pipeline so nobody works on data that is not ready yet.
Recon discovers the attack surface. Enumeration enriches what Recon found with services and versions. Vulnerability and Compliance both work off those enriched assets in parallel, because they do not depend on each other. And Reporting fires last, once everything upstream has finished, so it always writes from the full picture.
This is pipeline parallelism, and it buys me two things at once: real concurrency where it is safe (Vulnerability and Compliance genuinely run side by side), and logical ordering where it matters (nobody hunts for vulnerabilities on assets that have not been enumerated yet). I will go deep on how the orchestrator supervises this in the next post, because there is a lot of interesting machinery in keeping five independent agents honest.
The One Rule I Refuse to Break: Evidence First
If ReconForge has a religion, it is this: never trust only the agent's interpretation.
Every time an agent runs a command, the system captures the command itself, the exit code, the full stdout and stderr, any files it generated, timestamps, and which agent kicked it off. All of that goes into object storage as an immutable artifact before the agent is allowed to summarize what it means. The agent's tidy summary is a convenience layer on top of the raw truth, never a replacement for it.
This is what makes the reports credible. A finding is not "the AI thinks port 8080 is risky." It is "here is the exact command, here is its output, here is the artifact ID you can go read yourself." It also means I can re-parse old evidence later when I improve the analysis, debug a weird run by replaying its event timeline, and hand an auditor a paper trail that actually holds up.
Raw evidence is cheap to store and priceless to have. Skipping it to save a little space is the kind of shortcut you regret exactly once.
What I Learned Building the Foundation
A few things stood out while laying this groundwork:
Decide where truth lives before you write a line of orchestration. The single most important choice was "the container is not the source of truth, the database and object store are." Everything downstream got simpler because of it.
Separation of layers is what makes swapping painless. Because the execution substrate sits behind a clean seam, I can change how commands run without touching how the assessment is coordinated. Boundaries are boring to build and wonderful to have.
Constraints beat commands for the user. Letting people pick a template instead of typing tool flags is not dumbing it down. It is moving the expertise into the system so the product is usable by more than just the person who built it.
A small object vocabulary goes a long way. Project, Target, Template, Run, Finding, Artifact, Event. Seven nouns, and suddenly the API, the UI, and the database all speak the same language.
What's Next?
This post was the map. In the rest of the series I am going to walk through the interesting parts one at a time:
- Part 2: how the orchestrator supervises parallel agents with LangGraph, and why a workflow beats a free running agent loop.
- Part 3: getting an LLM to return something you can actually trust, using structured output and a pure function runner.
- Part 4: the six layers of resilience that keep a long run alive when agents misbehave.
If you are building anything that lets an AI take real actions in the real world, I hope the evidence first idea sticks with you. The flashy part is the AI. The part that makes it trustworthy is everything you write down around it.
Till next time! ✌️