The hardest problems weren’t AI problems

I thought I was building a question-answering system.

I eventually realized I was building a trust system.

That sounds like a small distinction. It wasn't.

Over the last few months I've been building something intentionally narrow.

Not another AI assistant. Not another chat interface. Just a private system that answers questions about AI assurance using a corpus I'd spent the better part of a year assembling: controls, evidence requirements, framework mappings, and supporting source material spanning the EU AI Act, ISO 42001, NIST AI RMF, OWASP, MITRE, and a couple hundred other references.

The original goal sounded almost trivial.

I wanted to ask questions like:

• Which controls actually apply here?
• What evidence would convince me this control is implemented?
• Does this control satisfy a requirement, or merely contribute toward it?
• If two frameworks appear to say the same thing, where do they actually differ?
• If the system gives me an answer, how do I know it's justified by evidence instead of merely sounding plausible?
• Which parts of the answer can I prove, and which parts require judgment?
• What should be learned?
• What should be proven?
• Which failures should be impossible instead of merely unlikely?
• When should I trust the model?
• When should I trust deterministic software instead?

The more I worked on it, the more I realized those last questions were the real project.

If there were ever a place where I expected a language model to perform well, this was it. The corpus was intentionally narrow, carefully curated, heavily structured, and fully sourced. Every control already carried explicit evidence requirements and typed mappings back to authoritative documents. There were no arbitrary web searches or unbounded knowledge sources.

Even then, building something I actually trusted turned out to be much harder than I expected.

Eventually I ended up with a system that runs entirely on a single machine, makes no external calls, cites the source behind every claim, and refuses to answer when the available evidence isn't sufficient.

None of the individual techniques are especially novel. Retrieval, reranking, model distillation, and deterministic verification are all well-established ideas.

What surprised me was how differently they behaved once they became part of a system whose outputs actually had to be trusted.

More importantly, what I thought were AI problems repeatedly turned out to be software engineering problems.

The engineering lessons were older than the AI

Before I started, I assumed I'd spend most of my time tuning prompts, comparing models, or experimenting with fine-tuning.

Instead, most of my time disappeared into retrieval quality, evaluation methodology, benchmark design, relationship modeling, and data representation.

More than once I became convinced the model was broken.

More than once the benchmark was wrong.

Sometimes retrieval had failed.

Sometimes the relationships I needed existed in the data but weren't represented in a way the retrieval system could ever discover.

Sometimes my evaluation metric simply wasn't measuring what I thought it was measuring.

Changing the model wouldn't have fixed any of those problems.

Looking back, one of the biggest surprises was how often traditional software engineering principles solved issues that initially looked like AI failures.

• Separating responsibilities.
• Building deterministic invariants.
• Treating evaluation as production code.
• Questioning measurements before optimizing against them.
• Reducing hidden assumptions.
• Keeping independent verification separate from probabilistic reasoning.

Those ideas aren't new. They're the same engineering habits that make distributed systems, operating systems, databases, and safety-critical software reliable.

They turned out to matter just as much here.

When I say deterministic invariants, I mean properties the entire system guarantees regardless of what the model decides.

A citation either exists or it doesn't.

A quoted passage either appears in the cited source or it doesn't.

Those weren't things I wanted the model to estimate.

They were things I wanted software to prove.

Separating reasoning from verification

One design decision influenced almost everything else.

Very early on I decided the language model would never decide whether it had successfully justified its own answer.

Instead, the model proposes an answer and cites the evidence it relied upon.

Everything after that is deterministic.

Separate code verifies that every cited source exists, confirms quoted text actually appears in that source, checks that the evidence actually supports the claim being made, and rejects answers that don't meet those requirements.

The model reasons.

The software verifies.

Those sound similar.

They're not.

Keeping those responsibilities separate simplified far more of the system than I expected.

It also fundamentally changed how I thought about trust.

For this kind of problem, the answer itself isn't really the product.

Being able to independently verify why the answer was given is.

The validator solved a problem I didn't know I had

One of my favorite discoveries wasn't planned.

The validator was originally written to verify production answers before they reached a user.

Months later I realized it had quietly become something else.

It was filtering the training data.

When I generated teacher responses for distillation, the same deterministic validator naturally removed answers that weren't sufficiently grounded or couldn't be verified.

Without intending to, I'd built a mechanism that improved production assurance, evaluation quality, and training quality using the same piece of software.

Looking back, it seems obvious.

It wasn't obvious while I was building it.

It also changed how I think about distillation.

If you already have a reliable verifier, you don't just have an evaluation framework.

You have a mechanism for constructing cleaner training data.

The benchmark lied to me twice

One of the more humbling moments came from my evaluation framework.

At one point I thought I'd made a major improvement.

The numbers looked fantastic.

Eventually I realized I was evaluating on questions the model had already seen during training.

The benchmark wasn't measuring generalization.

It was measuring memorization.

Later I made the opposite mistake.

One metric stubbornly refused to improve, so I assumed the model still had work to do.

After inspecting individual examples, I discovered the model had been answering correctly all along.

My metric simply wasn't recognizing valid answers.

The model hadn't changed.

The measurement had.

That experience reminded me of something software engineers have known for decades:

Before optimizing against a metric, make absolutely certain the metric is measuring reality.

Not every behavior belongs in the model

Another lesson surprised me because it felt almost backwards.

One stubborn failure involved requests for the exact text of licensed material that I intentionally hadn't included.

Initially I treated it as a modeling problem.

Eventually I stopped.

The correct behavior wasn't something that required judgment.

It was a rule.

So I implemented it as one.

No retraining.

No prompt engineering.

No additional examples.

Just deterministic software enforcing a deterministic policy.

Looking back, I probably spent too much time trying to teach the model things that software could guarantee perfectly.

That lesson extends well beyond language models.

• Use AI where judgment is required.
• Use software where correctness can be guaranteed.

Retrieval became the ceiling

Another assumption didn't survive contact with reality.

I kept looking at answer quality.

Eventually I realized answer quality couldn't exceed retrieval quality.

If the right evidence never reached the model, reasoning couldn't recover it.

Improving retrieval improved everything downstream.

Changing how relationships were represented in the corpus mattered more than changing prompts.

Adding a reranker mattered because the measurements showed it consistently surfaced better evidence, not because rerankers are fashionable.

That pattern repeated throughout the project.

The model was rarely the bottleneck. The system around it usually was.

Although I should admit that Nemotron Nano and I aren't currently on speaking terms.

Looking back over the project, the biggest improvements didn't come from swapping models.

They came from improving the system around the model.

Why I think this matters

This wasn't just relevant to one small assurance system.

I suspect the same pattern exists across many enterprise AI projects.

Teams spend weeks debating which model to use while treating retrieval, evaluation, verification, and data quality as implementation details.

My experience was almost the opposite.

Once the model reached a reasonable baseline, the largest improvements came from improving everything around it.

That doesn't mean models don't matter.

They absolutely do.

It means trustworthy systems emerge from the interaction of many components, not from any one component alone.

What I'll take into the next project

The system still has plenty of room to improve.

The evaluation set is still relatively small.

There are edge cases I want to exercise much more aggressively.

Retrieval quality can continue improving as the corpus grows.

None of that feels unusual.

That's just engineering.

If there's one lesson I'll carry into future projects, it's this:

Before changing the model, understand the system around it.

Many of the biggest improvements came from cleaner data, better retrieval, deterministic verification, stronger evaluation, and more careful measurement, not from finding a different model.

Before this project, I would have told you that building trustworthy AI was primarily about choosing the right model.

I don't think that anymore.

I still care deeply about models.

They're remarkable pieces of engineering.

But today I think about them the same way I think about databases, search indexes, or message queues.

They're essential components inside a much larger system.

Trust doesn't emerge from any one of those components.

It emerges from the engineering discipline that connects them.

Ironically, the farther I got into building AI software, the more I found myself relying on engineering principles that have existed for decades.

I didn't expect that.

Looking back, I probably should have.