You're using agentic AI wrong.

Full episode transcript

Your agents drift because you gave them suggestions, not a system. I've spent time on X and a few different forums lately, reading software engineers — people I honestly figured understood this better than I do — and a lot of them are baffled that their agents and their sessions wander off from what they asked for. It's the same complaint over and over. It started fine, it held for a while, and then somewhere in the middle the thing forgot who it was working for and what it was building. They're all treating that as a model problem. It isn't one. Here's what I notice reading those threads. There's no structure underneath any of it. It's auto memory, or at most a claude.md or an agents.md file. That's the entire governing layer for their work. So the debugging goes straight at the model — newer version, different harness, longer prompt, be more explicit this time. And that buys a session or two. Then it drifts again, because nothing about the part that actually failed ever got touched. Auto memory is a suggestion. It is not a governing system. It picks up what it happens to pick up, it surfaces when it decides something feels relevant, and you get very little say in what it thought mattered. And a single context file has a hard ceiling — there is only so much you can put in there before you're picking which parts of your own business the agent is allowed to know this week. Most people are running an entire operation on those two things. Think about what that asks of one file. It has to hold what your company does, who the work is for, how decisions get made, what you already tried and threw out, what you will not do under any circumstance, and what you're aiming at a year from now. All in one document short enough to be read every single time. It can't. So people trim it down to whatever broke most recently, and the file stops being a standard and turns into a scar log. What you actually need is an operating standard. Something that understands your organization, your business, your job — the universal goals, the short-term goals, the long-term goals, and the nuance — without you having to ask for it or explain it in every single session and every single prompt. That last part is the whole point. If you're re-establishing who you are at the top of every conversation, you don't have a system. You have a habit. And this is the part I think people miss, because it sounds like more overhead and it's actually the opposite. The more you grow it, the more it understands how the pieces of your system interact with each other. More connections get made, the bonds between them get stronger, patterns start getting recognized. It gets better at the job as it gets bigger. That is not how a context file behaves. A file gets worse as it gets bigger. It's kind of like a human brain in development. And I want to be careful here, because I'm not making a one-to-one comparison between an AI brain and a human brain. I'm not saying that. But it follows a similar pattern. The more connections that are made, the better it gets at finding what it needs in any given situation. That's the property I actually care about. Not the metaphor. So what goes in a thing like that? Not trivia. The stuff you'd have to explain to a competent new hire before they could make a call without you. What the business is. What we're optimizing for right now. What we're building toward. Where the exceptions live. And which of those outranks the others when they collide. That last one is where most setups fall over, because priority is nuance, and nuance is exactly what doesn't survive as a bullet in a list of instructions. Now the part people skip. This is not a static set of instructions. You don't write it once, feel good about yourself, and walk away. It has to be grown, maintained, and reshaped over time as your company, your job, your life, whatever it's describing, changes and evolves. It takes constant work to stay consistent. That's not a footnote on the idea. That is the cost of the idea, and if you're not willing to pay it, don't start. I'd rather be honest about that than sell you a setup step. Because the failure mode here isn't that the thing is wrong on day one. It's that it's right on day one and nobody touches it while the business moves underneath it. Then it's confidently describing an organization that doesn't exist anymore, and your agents are following it perfectly. That's worse than having nothing, because now the drift comes with authority behind it. What I've found from having this in my own system is that my problems these days have nothing to do with my agents not understanding what I want. That whole class of problem is basically gone. I'm not re-explaining the business at the top of every session. I'm not catching an agent halfway through a job that quietly decided the goal was something else. What replaced it is maintenance. Finding efficient ways to keep the system current, deciding what actually gets recorded and what doesn't, keeping the whole thing from turning into sprawl. That's a real cost and I'm not going to pretend otherwise. But it's a net positive to the long-term health of my organization, and I'd make that trade every time, because it's a problem that compounds in my favor instead of resetting itself every session. If you're in those threads right now, frustrated, I'd stop looking at the model. The model is probably fine. Look at whatever is actually governing it and be honest with yourself about whether that's a system or just a place you dumped some preferences. If it's the second one, no amount of prompt engineering fixes it. You're patching the symptom in the one place that can't hold the cure. The people I saw complaining aren't bad engineers. They're good engineers who assumed the memory layer was infrastructure when it's a convenience. That's the whole gap. You're not using agentic AI wrong because you don't understand the model. You're using it wrong because you never built the thing that's supposed to be in charge.