So THAT'S Why My Agent Orchestrator Sucks

Transcript is auto-generated and may contain errors.

Hey folks, we're just going to do a little AI recap. Um, I'll share some of the stuff I've been working on, where I've gotten to, what's not working, because I think that's important to talk about, too. Uh, so to catch folks up, I've been talking about using a couple of things together and, uh, not that I invented them by any means. This is more just for me to explore and play around, but um, been trying to put together like a homegrown Ralph loop. um as sort of like the inner working agent and then around that like an orchestration system and that's built on Symphony which is the spec from open AI. Um they my understanding is like their intention was to build it as a spec that's open and then other people can actually choose the implementation because you know how you build it what language text stack like that doesn't really matter.

It's more just like the philosophy and and their design for it. So I have like a a custom symphony implementation and then I have uh my my templating and instruction framework that is sort of a private thing for me not because it's uh I don't know not because it's top secret but private just because it's like my coding standards like how how I want to build stuff. very very opinionated. So, um, I have all these things layered together, playing together, and the other day, I think I guess it was roughly last weekend or something, I was saying that I I finally had like successful runs going through building really simple applications uh with like basically a you know, like a oneline prompt. And that's not the point to build with a oneline prompt.

that it it can um so I forgot to mention too that there's a step in the beginning that when you build that way it will take a spec made from spec kit and kind of feed that in break up the work items and then orchestrate uh agents to go build them. So again, the the intention is not to have a system that, you know, going forward all that I'll ever ever do is give like a literally the the eval prompt is something like make me a Windows calculator app uh in C. So it's literally that simple. And the goal is that um you know regardless of how little detail you're providing in the beginning that it has something that it can take break into work items and then actually go build. So that way if you're giving more detail that I can go do that. But um I'll have obviously better scenarios.

I I'm just starting with the like something extremely simple and then my goal is to see like if I start becoming more specific. Um can I see that it's picking up the right templates following my my instructions? Uh when I say instructions, I mean some of the the actual like agent guidance that's in instruction files. Um, and then you know this will essentially give me a system that can build and write code the way the way that I want. And obviously I'm not the only person building stuff like this. This is not something I think like I'm going to turn around and sell it. This is literally just so I can learn about how these things work. So, um, what have I learned? Well, last weekend it was working uh to do something simple and then over the course of a week uh it is no longer doing something simple.

So, I'm I'm trying to to see how to tune it because even for a really simple calculator app, uh it's like if I let me put it this way, if I were to give Copilot or Claude Code that like oneline sentence and put it in autopilot, I think it would have significantly better results. So, I was claiming yes, it's working. Like, it's building it. But if you were to score the output of what I built, it's terrible. Like, uh, the odds of it finishing within an hour are pretty slim, apparently. Pretty slim statistically. I've seen it run for 2 hours to do like a broken output. Um, and so I've been trying to basically have co-pilot itself do the analysis after and I have it collecting like tons of data and for some I don't know why. Um, for some reason I feel like I've been very unlucky, but obviously this stuff's not just luck.

It's definitely on me, but uh it's it's doing a really terrible analysis every time. And so like it's missing it's missing the point. So it will say things like oh it's you know um spending this much time thinking versus this much time doing tool calls. I'm like that's literally not that's not important. Like what is it what is it spending time on? Like is it spending time uh running MS build and like and then failing? like is it spending time? Um the tests are like we're not getting the output from that. Like what what is it actually doing? So this has been like really challenging. So I have to keep um God these people on the highway man. Um, I have to really tell it to, you know, every time it's like, "No, go deeper. Go deeper." Like, pay attention to these things.

And then we end up doing, you know, we'll discuss something to try and then run it, no improvement or or it was still wrong, like that actually had nothing to do with it. And then, you know, kind of repeat the process. So, it's been like just obscenely slow um to make progress. But really, like when I I got down to it last night and um I was trying to take a bit of a different approach, I was like, you know, obviously the I'm being let down by the analysis I keep getting. So, I'm just like some of the things it was suggesting to me about like what the agent is doing. I'm like this doesn't make sense because because of this claim I was kind of saying like if I just give co-pilot with no extra guidance you know go do this go do this

really simple task I'm talking about like subtasks now within building a calculator app and it's not a fancy calculator app and I'm like I know co-pilot can do that cuz I've seen it oneshot things that are significantly more complex and obviously are they perfect. No, but like they're better than what I'm getting here. So, I basically said like just show me the the prompt. Like what are what are you actually telling this agent to do? Because clearly it's getting some additional guidance that I don't normally give co-pilot like the the fact that it spins running builds for like 60 minutes. I'm like there's there's no way it's being told to do something that it shouldn't be doing. And so at the root of it, I think like uh and this is going to sound kind of obvious in hindsight. I think the the Ralph loop as the worker is not the right pattern here.

I think the Ralph loop uh is a pattern that can be used, but I don't think it works well in this orchestration setup because um the way I look at it is kind of like the Ralph loop itself is almost like a mini orchestration, like a sequential orchestration um a single sequential orchestration. And so, um, you're basically telling this agent like, you know, do here's a list of things to do. Do one of them at a time. Um, and each time like build, test, make sure you have the output and then like do the next iteration of the loop. So, it's like a very narrow sequential orchestration uh with with exit criteria. And so having that as the inner loop, the reason I believe that things were taking so long is that there's specific instructions about like you know on a on a per task basis build test and then like basically re you're going to be reset to go to the next um task.

But the problem is that the tasks that we're telling this Ralph loop to do because it's already part of a bigger orchestration, they're like really really really small units of work within a user story. So the point being that the tasks are already broken down. So I'm basically telling it to do uh things uh you know I'm going to exaggerate but like almost the equivalent of like go make this class and add this method to it build the whole solution test the whole solution and then like we'll do the next iteration. And so it's spending so much time because there's so much overhead to go do this. It's like if I told you, you know, every time you write a line of code, we're going to have to go build and run all your tests. And so if you wrote that line of code and it wasn't perfect, um because you're not done building the rest of what you need to build, it's obviously not going to work.

So I kind of realized like like and maybe it sounds obvious in hindsight, but like the the Ralph loop is just not the right inner worker. Um, so I'm not getting rid of Ralph loops. I'll have that, I guess, as an option. I don't really know how I'll use it in this orchestration because the whole thing started off as like, let me try building my own Ralph loop to play around with it. Um, and so I don't think the concept itself is necessarily terrible. I just don't think it fits here. So, that's cool because I think that the um the actual inner worker that I'll be using will be a lot more simple. Something uh you know to the thing I started with, I haven't run an eval yet. I think it was supposed to run overnight, so we'll see. Um you know, something as simple as like you know what you need to build.

The instructions are in this other file. So, go read that. um you know build it as it says and then when you're done you know uh say that you're done right obviously we can do more we can do better but as a starting point I I honestly think that's all I need the the uh actual worker sort of like seam between the orchestration and the worker is very thin. As long as it says it's done, uh that work will end up getting committed back on uh on the main branch of the repository and then the orchestrator will pick up sort of the next thing to go get sent out to an agent. So, uh, yeah, I'm I'm hoping that if if it truly can stay that simple, one of the really nice things about that is that, um, if it comes to bring like bring

your own agent kind of thing, cuz I I was thinking about this the other day, like the agents that I'm using are not um, currently they're not like tailored, customuilt, or anything. It was like seriously just this dumb Ralph Loop worker agent. And when I'm saying agent here, I mean the sort of the prompt that that agent is carrying as its set of instructions. It's just really generic, right? And we could do better because you could you could uh I don't know, you could have agents that are more customuilt for like for doing front-end development or database work or something else. So, it would be really cool if I had with my orchestration when it's sort of splitting up the work, if it's like, hey, you're going to go build this part. Um, what that could look like is that I have, I don't know, a dedicated like C expert.

Okay, so C# expert is running and it would have its own tailored instruction file that gives it guidance around how to be a good C expert. I'm trying to balance how much of that we need versus um these file pattern based instructions cuz I've been really liking those. I'm like, if I have good sets of that, does the agent instructions itself need to like what does that actually need to have to be a C expert? I don't know. But at least that's another sort of uh you know tuning in the system to play around with. Uh the other thing is that the the workers themselves are just co-pilot sessions. And so they're not it's not a custom harness or anything. That was one of the things that I didn't want to spend time doing because every time I mess around with a custom harness, it's been like very lackluster.

Like why is it so hard to get anything to work? And it's like because I literally am not using Claude or Co-Pilot as the harness. So at least for this I'm like hey I don't I don't want a custom harness. I want to use a harness I know works well that I am using in other things. So just co-pilot as the harness itself. But because of that I think something that might end up happening is that if I spin up a co-pilot session as a worker co-pilot itself you can run subtasks or sub aents. So maybe how this works is that the orchestration will pick up the next you know available user story and then what we'll do from there is assign that to a worker which is a co-pilot session. that worker will do work. There could be this opportunity where that worker can actually spawn sub agents depending on what the task is.

So maybe you have to go build some feature and it's should have been speced if you're using spec kit and the rest of the orchestration leading up to this. But if it needs to go research something or whatever, maybe it could spin up a sub agent to go do that. Maybe um you know it could use a sub agent so like a more targeted uh agent for writing the tests or things like that. But the point is that if it's using co-pilot as the harness then it can because if it wasn't then I would have to build a harness that knows how to properly spin up sub aents when to do that etc. Um, and so like yeah, just reusing a harness that actually works is kind of uh in my opinion a good move cuz I don't want to worry about that I'll build my own harness for other stuff to mess around with to learn more about that, but definitely not for for coating.

So, um, we'll see how that one does. I'm I'm actually expecting that as long as it didn't break because the actual seam like so say uh we couldn't detect when the agent actually stopped working which I hope it can uh if as long as that worked I think it'll actually have done a better job because the like I was saying the actual work that was being delivered is not not that complicated and I think it was s uh like sabotaging itself. So anyway, that's the update. See you in the next one.