Orchestration and Loop Engineering - Feeding Templates To My AI Workers

Transcript is auto-generated and may contain errors.

Hey folks, we're going to do a little AI recap. Um, I like doing this after the weekend because that's when I get a lot more time to just actually sit down and crank out code. I also had last week mostly off work. I took uh the week off. My wife was traveling back in Canada uh for her birthday, visit her friends. So, kind of had the had the house to myself, watching the animals and yeah, so was doing a lot of programming, but still ended up having to to work basically every day in some capacity. That's fine, though. Uh, it was nice to have the flexibility to like, you know, to focus on what I want to do and then pivot back over to to work stuff as needed. So, what have I been doing? Well, this will just be an extension of things I've been talking about recently.

So, I'll just give you a heads up. If you've watched my other videos on how I've been building stuff with AI and you're like, "Well, that sounds dumb, Nick. I don't like it." You're probably not going to like this one either. But if uh if it's been kind of interesting, then hopefully this will continue on that path. So, for context, uh I use co-pilot CLI a lot for things. I think at the beginning of this year for I mean the first few months of this year I've been saying like I need to try using other tools a little bit more too but I'm having a really difficult time getting out of the co-pilot CLI just because uh the I don't know the flow is is working quite well so it's it's too disruptive right now to to pivot away from it and especially because I have my my token limits uh lifted So it's like a lot of incentive to to stick around.

So um my general flow is that I'm building things across a few different repositories. I have um brand ghost which is the business I have on the side for uh posting content to social media and doing uh SEO work for businesses uh on basically autopilot. So I have that that I work on. I have Neler which is my dependency injection framework. I have a another project for uh for helping build a uh sort of like a consumer app to help a friend out. And then what else? I have like just a handful of like supporting things. And so I find that I have like essentially a PowerShell terminal window open with like 12 tabs and I kind of just go between them prompting co-pilot to to do things. And so there's a someone was calling it they had a special word for it, but basically talking about how like uh you know this this context switching from trying to manage too many uh AI sessions is like a is a real thing.

Um not brain rot, it's something else they had. But anyway, they were saying like, "Hey, like it's it's real that, you know, it's uh kind of messing with their heads a little bit." So, one of the approaches or things that I've been trying to do that I talked about more recently was trying to build up um sort of this automated flow where I use something that looks like a Ralph loop if you've heard of Ralph loops. So a worker that basically takes a task um iterates on it with some constraints and then uh you know if it if it doesn't meet the constraints within the period of time it's throwaway work so you you scrap it and the idea being that like um ideally we have agents that get the right thing done and you know get the work out of the way but if it can't then we acknowledge that it's it's throwaway right like you didn't waste human time doing it Great.

So it does this loop and then I have a orchestration layer on top which follows uh this this open- source uh sort of definition from open AI called symfony. So I have my my own implementation of that in C. And then uh the third layer is sort of this integration with spec kit. That car was not anywhere near close. I didn't even have to touch the brakes. It's beeping at me. So, you start the whole thing off uh by being able to feed it a spec. Now, the cool thing or what I think is cool about all this is that those none of those steps are required. So, if you just want to use the inner like Ralph loop, you can do that. Um if you want to use it with the orchestration, you can do that without, you know, giving it a spec.

But what I'm trying to do is make it so that even if you just have like a a oneshot prompt and you're like I want I'm literally trying it with a prompt that says I want a desktop um a desktopbased calculator app in C. Like that's that is the prompt and the idea is that it will use spec kit to build out the spec and then use the orchestration layer to schedule all the work and then have all the agents go do it. Um so I finally had the first successful like end to end of this. And it's not that the previous ones like didn't I don't know how to say it's like not that I couldn't have said it it passed but I certainly wasn't happy with it especially given how simple like with a prompt like that all that I'm expecting is like

a sort of like a wind form app or something equivalent some Cbased you know tech stack uh could be WPF it doesn't matter cuz I'm not specifying applying it in the in the prompt. What I'm expecting is that I get my specification documents created that I get a a simple app that has, you know, some buttons on it to to do calculations, right? To to be like the desktop calculator you have on like or your mobile phone or whatever simple calculator UI. And um given how simple it is, I'm expecting I said like, you know, it's got to be done within an hour, right? That should be plenty of time to be able to do this. And um what's been happening is that it's like partially it's getting stuck on some things. So it's kind of like just the glue between all the parts. It would be like, oh, it spent, you know, 30 minutes and had to retry on this one step.

So, he used half the budget on one step that failed. And then when I look into it, it's like, oh, the the agent session like literally just froze or the agent session was stuck trying to build something in a loop, but like um you know, it like didn't have the dependency or something silly. And uh it's just exhausting sort of the budget that I give it. Um so finally had the first success of this which is pretty cool to see. Um, the other part worth mentioning is that I have a set of templates that I'm building up so that when I say something like build me a desktop app um, in C, like I have templates that are basically the foundation that I would expect to have for any code base that if I wanted to use um, like Windforms for example, here's exactly the minimum minimum expectations I would have.

So here's a solution file. Here's a couple of projects. I like building everything with plug-in architecture. So like here's how that would look. Here's what I use for dependency injection. And then I do this across all of these different types of of apps. And I I don't have like a crazy amount of them. There's over 10. I think under 20. And so the idea is that if I say to this thing, go build me something that there should be a template that I can start with. And so part of the integration, again, a lot of these pieces are optional, is that when it's building the spec, it can go ask my template library, do we have a good template for this to start with? Because if so, let's use that as the basis. So in this case it's uh it's using an Avalonia template that I have which is super handy for just you know getting started with a a C desktop application.

The other thing I should mention too is that when I'm talking about these templates it's not just like um I don't know going to Visual Studio and pressing new project and getting something made. Like that's part of it. The other part is that by default it has all of my coding standards uh turned on in ways that are like uh not only built-in analyzers to or like llinters if you're not familiar with the C uh terminology. So has that stuff turned on by default for what I want. Um it also has my agent instructions. So I have a library of shared instructions that are they're they basically match on the file path as well. So I can say like if you're dealing with a file that ends in like test.cs like it's basically a test file in C here are my standards. And the other cool thing is that that's also templated.

So there's like components within the template. So, if I'm using XUnit versus TUnit, I can swap in those instruction files to be more pointed. And so, when you link all this stuff together, my goal is that I'm able to to basically kick off new projects, right? Have them scaffolded uh really well from the beginning uh on autopilot. And then the other part of this system is that it's not just used for scaffolding something in the beginning. The idea is that once this uh it's a it's a small little web server that listens, but you can basically queue up work to it. And so you can just keep sending it work and it will uh schedule agents. And if you build a backlog, the idea is supposed to be that there's a sort of like an orchestrator that will look through the items that are given and based on the priority and the dependencies will try to basically queue up workers to work through that that backlog.

So this is it's all really an experiment. Some of these pieces like I uh I actively use like every day. So the the templating library with the the the reusable instructions, uh I I work in that every single day. So I'm at the point now where because I'm going across all of these co-pilot sessions, I'm always finding little patterns where I'm like, oh, like, you know, here's one example in this repo where this is certainly a problem. I keep running into it. Let's get that back into the instructions. Um, so bringing that back to the template library or if I'm, you know, working on a new feature, right? It could be a whole new uh project entry point, uh, whatever. Like that's the best example where I'm like, "Oo, I should have a template for this." But even if it's a new feature, um, I might go, "Hey, like this is something I need some coverage for in the templating library." Honey, come on.

You're going to have to move over. Barely not. I will move over. I got to move over one more lane here. Come on. This is Someone's got to speed up or slow down. Apparently no one. Okay, excellent. That was dumb. Um, so yeah, even if this experimental like orchestration thing doesn't pan out, um, that's totally fine because the templating thing is is something I actively use. Uh, it also syncs those templates. Uh, sorry, not the not the entire template, it syncs the instructions across my different projects. Uh so it can track like hey if you're working in this kind of repo setup like you know use these uh use these instruction files. So anyway it's uh it's cool. It's at the point where it's like starting to see some success with the basics. We did a couple of changes. I say we as in like me and my buddy co-pilot did a couple of changes after the the first u calculator run.

And so the calculator was it looked terrible. And so I asked co-pilot I said based on the spec you know was there anything here about UX and it said no. So I said okay great like change let's do another attempt change your prompt because the prompt was on purpose like right? It was like the most simple prompt cuz I wanted to see if given basically nothing will it finish. So now the prompt is almost the same, but then it has a little bit of a an addition that says like I want a consistently styled um user interface that looks modern. So that goes in at the very beginning, right? The prompt is extremely basic on purpose just cuz I'm trying it out. And then hopefully we'll see that uh the agent it's it's crazy, right? There's an agent that's driving the spec kit conversation, which is nuts, right?

I have That's not like I didn't invent this. Uh, so just having an agent having the conversation on your behalf about what you want to build is nuts. Um, just fully hands off. So, I'll see if that actually puts in some UX details into the spec and then if the agents actually follow it after. Um, I should also mention too that if I like came with my own I don't have to use the spec kit part at the beginning. I could absolutely just come with my own spec or my own prompt or skip that entirely, start a new repo and just start sending over work items to it. Um, and work items it does integrate with GitHub. So the work items themselves can just be git issues but all these pieces kind of coming together. Um then what I'd like to do on top of that is once I have it kind of working end to end is have some of these as eval scenarios.

And what does that mean? So I'll take something as simple as the calculator app. I'd like that to be an eval scenario. If that takes an hour to run right now, what I would like to do is make sure that I have some requirements that I'm looking for across all of these scenarios. So like has to finish within this time window, has to um you know consume less uh than this many tokens, has to deliver on the spec. So I'll need like an LLM judge to to go over the spec, go over the output and see if it works. And I'll come up with a few eval scenarios like this. And that way as I tune that whole system, for example, if I change how things orchestrate, right? Um, I think in this one it was only doing one worker because it was really simple.

like there's not a lot of parallel work, but for more complicated ones, if I have multiple workers um and there's conflicts and stuff like that, if I'm tuning how conflict resolution works or how retrying works and things like that, uh can I make optimization so that when a worker starting a session, you know, one of the things that came up as an idea was instead of starting a brand new working session, like it's a huge optimization if you keep the running like the working directory you're already in, right? For for building some of these things like it if we're talking about, oh, maybe it takes a minute to go compile. Okay. Well, 1 minute on a 60-minute budget when you're doing multiple loops, like even if I had to do 10 work items, that's a minimum of just 10 minutes building, right? That's a hu that's actually a huge part of the budget.

So, can I start to tune this in ways uh where I can go run it against all these eval scenarios and and say like, hey, look, this still passed based on my expectations. Um, and I I don't have a ton of experience with evals, so I'm still trying to build up that muscle. But, if you haven't noticed from listening to me talk about this stuff, um, I I kind of think about it in two ways or two main ways. The first way is that uh should park at the end. I'm trying to obviously build things and use AI. So I need to get better at building things so that I can use AI effectively. That's one part. So that's like my templating. That's my uh the my prompt library that uh or my instruction file library that I'm moving along. So that's that's part of it.

And then the other part is like I I'm experimenting to try things because that's how I learn most effectively. And when I've talked about this before, you know, AI was prevalent, the way this looked for me was I used to to build a role playing game. And not because I thought that, oh, this game's going to be complete one day and I can't wait to play it. It was because me spending time building that let me go learn about different parts of software engineering. So, I'm playing around with this agent orchestration stuff, not because I'm sitting here going, "Oh, I'm I'm inventing agent orchestration." Like, I'm I'm not. And there's people that are going to be way ahead of me, and that's totally cool. There's companies that are way ahead. That's cool. Like, I'm happy to go use that stuff. This is me playing around so that I can learn because this is how I learn.

So, hopefully that's helpful in some way. If you got questions about software engineering, AI, career stuff, leave them below in the comments or you can go to codecute.com, submit stuff anonymously that way and I will make a video response for you and everyone else that watches. So, take care. See you in the next one.