You're using agentic AI wrong.

Your agents drift because you gave them suggestions, not a system.

I've spent time on X and a few different forums lately reading software engineers — people I honestly figured understood this better than I do — and a lot of them are baffled that their agents and their sessions wander off from what they asked for. It's the same complaint over and over. It started fine, it held for a while, and then somewhere in the middle the thing forgot who it was working for and what it was building. Almost everybody treats that as a model problem. It isn't one.

There's nothing underneath

What I notice reading those threads is that there's no structure in their systems. It's auto memory, or at most a claude.md or an agents.md file. That is the entire governing layer for their work, and then they're surprised when the work isn't governed.

Auto memory is a suggestion. It is not a governing system. It picks up what it happens to pick up, it ranks it without asking you, and it surfaces when it decides something feels relevant. You find out what it thought mattered after the fact, usually when it gets it wrong. That's a convenience feature, and people are leaning on it like infrastructure.

The single context file has a different problem: there's only so much you can put in there. Think about what that one document is being asked to hold. What the company does. Who the work is for. How decisions get made. What you already tried and threw out. What you won't do under any circumstance. What you're aiming at a year from now. All of it short enough to be read every single time, in every session, without eating the space you actually need for the task. It can't. So people trim it to whatever broke most recently, and the file quietly stops being a standard and becomes a scar log.

What you actually need

What you need instead is an operating standard: something that understands your organization, your business, your job — the universal goals, the short-term goals, the long-term goals, and the nuance — without you having to ask for it or explain it in every single session and every single prompt.

That last part is the whole point, and it's the part I'd use as the test. If you are re-establishing who you are and what you're doing at the top of every conversation, you don't have a system. You have a habit.

What goes in it isn't trivia. It's the stuff you'd have to explain to a competent new hire before they could make a call without you. What the business is. What you're optimizing for right now. What you're building toward. Where the exceptions live. And — this is the one that gets skipped — which of those outranks the others when they collide. Priority is nuance, and nuance is exactly what doesn't survive as a bullet point in a list of instructions. If your setup can't tell an agent that speed loses to reversibility on hard-to-undo actions, or that a particular client's constraint beats the general rule, then every one of those calls comes back to you.

Why it gets better instead of worse

Here's the part I think people miss, because it sounds like more overhead and it's actually the opposite. The more you grow it, the more it understands how the pieces of your system interact with each other. More connections get made, the bonds between them get stronger, patterns start getting recognized. It gets better at the job as it gets bigger.

That is not how a context file behaves. A file gets worse as it gets bigger — harder to read, more contradictory, more expensive to carry. A connected system gets cheaper to use as it grows, because the next thing you add lands next to things it relates to.

It's kind of like a human brain in development. I want to be careful with that, because I'm not making a one-to-one comparison between an AI brain and a human brain. I'm not saying that. But it follows a similar pattern: the more connections that are made, the better it gets at finding what it needs in any given situation. That property — retrieval getting better because the structure got denser — is what I actually care about. Not the metaphor.

The practical version: write it in pieces that point at each other, not as one long document. A system page names the workflows it feeds; a rule names the decision it governs. The linking is what compounds, and it's the part people skip because a flat file is faster to start.

The bill nobody wants to talk about

This is not a static set of instructions. You don't write it once, feel good about yourself, and walk away. It has to be grown, maintained, and reshaped over time as your company, your job, your life, whatever it's describing, changes and evolves. It takes constant work to stay consistent.

That's not a footnote. That is the cost of the idea, and if you're not willing to pay it, I'd rather you didn't start — because the failure mode here isn't that the system is wrong on day one. It's that it's right on day one and nobody touches it while the business moves underneath it. Six months later it's confidently describing an organization that doesn't exist anymore, and your agents are following it perfectly. That's worse than having nothing. The drift now comes with authority behind it.

So the maintenance has to be part of the work, not a project you'll get to. When a decision changes, the record changes in the same sitting. When something gets renamed or retired, the things pointing at it get fixed. When you notice yourself explaining the same thing to an agent twice, that's the signal that something belongs in the standard and isn't there yet.

Where this lands

What I've found from having this in my own system is that my problems these days have nothing to do with my agents not understanding what I want. That whole class of problem is basically gone. I'm not re-explaining the business at the top of every session, and I'm not catching an agent halfway through a job that quietly decided the goal was something else.

What replaced it is maintenance — finding efficient ways to keep the system current, deciding what actually gets recorded and what doesn't, keeping the whole thing from turning into sprawl. That's a real cost and I'm not going to pretend otherwise. But it's a net positive to the long-term health of my organization, and I'd make that trade every time, because it's a problem that compounds in my favor instead of resetting itself every session.

The people I saw complaining aren't bad engineers. They're good engineers who assumed the memory layer was infrastructure when it's a convenience. That's the whole gap. You're not using agentic AI wrong because you don't understand the model. You're using it wrong because you never built the thing that's supposed to be in charge.

Full episode transcript

Your agents drift because you gave them suggestions, not a system. I've spent time on X and a few different forums lately, reading software engineers — people I honestly figured understood this better than I do — and a lot of them are baffled that their agents and their sessions wander off from what they asked for. It's the same complaint over and over. It started fine, it held for a while, and then somewhere in the middle the thing forgot who it was working for and what it was building. They're all treating that as a model problem. It isn't one. Here's what I notice reading those threads. There's no structure underneath any of it. It's auto memory, or at most a claude.md or an agents.md file. That's the entire governing layer for their work. So the debugging goes straight at the model — newer version, different harness, longer prompt, be more explicit this time. And that buys a session or two. Then it drifts again, because nothing about the part that actually failed ever got touched. Auto memory is a suggestion. It is not a governing system. It picks up what it happens to pick up, it surfaces when it decides something feels relevant, and you get very little say in what it thought mattered. And a single context file has a hard ceiling — there is only so much you can put in there before you're picking which parts of your own business the agent is allowed to know this week. Most people are running an entire operation on those two things. Think about what that asks of one file. It has to hold what your company does, who the work is for, how decisions get made, what you already tried and threw out, what you will not do under any circumstance, and what you're aiming at a year from now. All in one document short enough to be read every single time. It can't. So people trim it down to whatever broke most recently, and the file stops being a standard and turns into a scar log. What you actually need is an operating standard. Something that understands your organization, your business, your job — the universal goals, the short-term goals, the long-term goals, and the nuance — without you having to ask for it or explain it in every single session and every single prompt. That last part is the whole point. If you're re-establishing who you are at the top of every conversation, you don't have a system. You have a habit. And this is the part I think people miss, because it sounds like more overhead and it's actually the opposite. The more you grow it, the more it understands how the pieces of your system interact with each other. More connections get made, the bonds between them get stronger, patterns start getting recognized. It gets better at the job as it gets bigger. That is not how a context file behaves. A file gets worse as it gets bigger. It's kind of like a human brain in development. And I want to be careful here, because I'm not making a one-to-one comparison between an AI brain and a human brain. I'm not saying that. But it follows a similar pattern. The more connections that are made, the better it gets at finding what it needs in any given situation. That's the property I actually care about. Not the metaphor. So what goes in a thing like that? Not trivia. The stuff you'd have to explain to a competent new hire before they could make a call without you. What the business is. What we're optimizing for right now. What we're building toward. Where the exceptions live. And which of those outranks the others when they collide. That last one is where most setups fall over, because priority is nuance, and nuance is exactly what doesn't survive as a bullet in a list of instructions. Now the part people skip. This is not a static set of instructions. You don't write it once, feel good about yourself, and walk away. It has to be grown, maintained, and reshaped over time as your company, your job, your life, whatever it's describing, changes and evolves. It takes constant work to stay consistent. That's not a footnote on the idea. That is the cost of the idea, and if you're not willing to pay it, don't start. I'd rather be honest about that than sell you a setup step. Because the failure mode here isn't that the thing is wrong on day one. It's that it's right on day one and nobody touches it while the business moves underneath it. Then it's confidently describing an organization that doesn't exist anymore, and your agents are following it perfectly. That's worse than having nothing, because now the drift comes with authority behind it. What I've found from having this in my own system is that my problems these days have nothing to do with my agents not understanding what I want. That whole class of problem is basically gone. I'm not re-explaining the business at the top of every session. I'm not catching an agent halfway through a job that quietly decided the goal was something else. What replaced it is maintenance. Finding efficient ways to keep the system current, deciding what actually gets recorded and what doesn't, keeping the whole thing from turning into sprawl. That's a real cost and I'm not going to pretend otherwise. But it's a net positive to the long-term health of my organization, and I'd make that trade every time, because it's a problem that compounds in my favor instead of resetting itself every session. If you're in those threads right now, frustrated, I'd stop looking at the model. The model is probably fine. Look at whatever is actually governing it and be honest with yourself about whether that's a system or just a place you dumped some preferences. If it's the second one, no amount of prompt engineering fixes it. You're patching the symptom in the one place that can't hold the cure. The people I saw complaining aren't bad engineers. They're good engineers who assumed the memory layer was infrastructure when it's a convenience. That's the whole gap. You're not using agentic AI wrong because you don't understand the model. You're using it wrong because you never built the thing that's supposed to be in charge.

Sources