The Problem With Mixing Determinism In Agentic Flows

Transcript is auto-generated and may contain errors.

Hey folks, I'm just driving to CrossFit on a Sunday night, which is super weird. Um, I've had terrible uh time management, I guess. We're going to do an AI talk cuz that's just what I've been doing over the past few days. And I'm about to head into uh for my backup on call shift into a primary. So, it's going to be a bit of a week, but yeah, I got a lot of lot of videos. I'm still uploading. I'm a little behind on that unfortunately, but figured I'd talk about some of the things I've been doing in the AI space. So, I'm going to kick it off with some of the uh Eve versel work, which has been pretty cool. Um, it is a shift for us where we had a lot of stuff that was being built with uh like Microsoft agent framework.

And I hinted at this in one of my my videos recently where I was saying I was under the impression that with Microsoft agent framework at the time I started using it that we were sort of getting some kind of capabilities that felt like a harness. And so like when that wasn't really happening, I was like, "Okay, um I started extending my my library called uh Needler uh into something that's now called Foundry to basically extend some of that functionality and kind of give like an agentic workflow with what would hopefully kind of be like a harness." And it was terrible. Um, you know, not not to self- sabotage here, but like, man, it was it was terrible. Um, building a harness is extremely difficult to get done well.

And um the I would argue like I think my biggest argument for why it was so terrible it was like trying to like you can put the pieces together and then when you start looking at like the effectiveness you're like okay like this is outrageously ineffective like how do we make it better and so okay like do you compact do you trim out the history? like it's just getting it tuned the way that like you know things like clawed work and co-pilot CLI work on like trivial things right like what seems like should be a very simple conversation um I feel like was just kind of crazy and so the next part to that was that as you build out this workflow the really challenging thing was that again to try and optimize let's say you have a an agentic workflow where you have like

one step and then another step runs after that and then maybe you do fan out into like two or three sub agents and then you want to bring it back to like you know a final agent just a simple example um seemingly the way that like working with co-pilot and like I get it the way that it would want to optimize it is like let's make it more uh like deterministic right like you have these steps that you clearly know about and You want to make sure that you know you're not wasting uh sort of like resources, right? Whether it's like um going from one phase to another is not working because the right information is not being captured and fed to the next stage or uh or whatever else like in terms of just like routing through this workflow.

like if you make the steps more deterministic in terms of how they they end and feed into the next one in theory like that should do better right like you're you're basically constraining things more so that it should do a better job but in practice what I keep running into is that the more that you try to make these things deterministic instead of making it better it just makes it more brittle right it's like um to to give you a a pair like a I don't know an example of this instead of like instead of making it actually output or arrive at something more deterministically it was really just like putting more restrictions on what it would allow if that makes sense right it didn't actually improve what it was doing it was just tightening the constraint so uh because of that it would just become more and more brittle and it would do that.

Like maybe the way I'm explaining this makes it seem like it's a a simple thing like why don't you just go to the one spot and make it less brittle. It's because it would like systematically find ways to do this. It would always converge on this pattern. So we started using um Verscel's Eve for this. And uh the nice thing like one of the really nice selling points is that it was mostly just like take you know a group of skills that you would have say for like co-pilot or claude and just drop them in and like deploy it and you have a harness that can run on that to do a workflow. I'm like this is kind of what we wanted. So we started doing that like the next day I saw uh I don't think it's just that Microsoft just released it. It's like I just noticed that Microsoft released something called harness.

I still have not yet gone back to go use it, but I updated uh my my foundry package to to try and incorporate some of that. Um because it's it's really the offering that I thought we had in the beginning and and clearly didn't. So, we've been trying to use Versell's Eve. um it's been way better in terms of getting up and running, but when it comes to optimizing started running into the same problems because I would work with co-pilot, we'd be running um these eval fuse so that I have a store to track all of these things. And um essentially what would happen is it would be like okay well here's where here's where we're spending time tokens like here's the inefficiencies okay um and then work with co-pilot and it's like okay well we got to you know you have this fan out

thing happening we got to make this more deterministic and I'm like don't don't you do it don't you do it because it keeps converging on this pattern of like make it more restricted and I get it, but all that it's doing is making it more brittle. So, um was going down this path getting kind of frustrated and one of the guys um that I work with on brand goes uh who had who had mentioned Eve and then kind of pointed us onto it uh while he uh had some time away from it kind of came back and was like, "Hey, like I think um I think they have this other thing." And so he he kind of took one of our workflows and rebuilt it. And he's really really good at this stuff. Uh I feel like I like I talk about this stuff like I'm doing a lot of it, but like he's he's really good.

Um so he kind of whips something up and what I'm not saying that he didn't uh work hard on it or anything. To me it looked like he put you know he makes it look effortless is is kind of my point. So comes back with this thing. It's a port still on Versel's Eve uh a port of one of our workflows and he basically has it without getting super brittle this approach where it can properly do fan out. It can properly come back like it's a lot more structured without having to get brittle. And so a lot of it had to do with um I guess two things. one is the structure of the repository. So I'm going to come back to this at the end, but the structure of the repository um for the agent and then the other thing that was relevant here was that there's another mechanism.

It's just called like dynamic workflows for Eve. And by the way, a lot of this stuff depending on when you're watching this, like I'm talking about this stuff like almost as the features are coming out. like when I saw workflows that were mentioned like even co-pilot was showing me links and it was like this thing is experimental like don't know um and then like you know 2 or 3 days later um this guy that I work with on brand ghost was like has it whipped up and it's already just like working. So uh by the time you watch this maybe that's a retired feature maybe there's something that superseded it. Maybe that's just the obvious thing to use, but um it's just a different way that it can split out and call the sub agents and they end up running more in like an in isolation.

So that's one interesting part is that compared to before we'd have this main agent run with sub agents, but the sub agents weren't really isolated. These ones are like they have their own complete isolation. They're much more sandboxed. So there's a bit more nuance to it. So they can't just like open the same file path. They actually have to go write uh write to blob store, read from blob store. A lot more isolation. And so he has this working and I'm sitting there like, well, okay, This is this is kind of crazy because what he built is not fundamentally. It's not like he uh took what we were building and then rewrote it completely different. Like it's not like it was I don't know like a completely different framework and language. It's literally he took like I could see the uh sort of the plan that that Claude produced.

He uses Claude code and um and he started with our repository and then made like a clone of it to go rebuild this in a different way. So I'm sitting there going like this is nuts. like how how can such like a like I'm trying to figure out is it because of this repo structure that he used or this this workflow calling pattern or the combination and I think the answer is it's actually a combination. So what I can't just do is take his workflow calling pattern and just go move it back over. Um but then I'm like we're kind of caught between these two paradigms. One is that the original approach we were putting a bunch of agents a bunch of skills for agents to use uh in one repository because it would make sense uh to kind of do that. I don't know why we would go do a bunch of different repos.

But the way that he structured his I'm going like I don't understand how we're going to get multiple agents out of this. Like it kind of feels like the repo is for one agent. Um, and so really this just comes down to like my I don't know like lack of familiarity with this like I don't know like Typescript JavaScript ecosystem and and really versel in general. But uh I talked with co-pilot a little bit about this and it explained to me like hey like the the idea is like you can still use one repository. It's just that in the one repo you're going to be using a top level agent. So like basically one of these types of workflows like uh my colleague had put together use you can put a bunch in the same repo but you're going to treat them as each as a separate uh Verscell project.

And it was saying like you can you know you can put a Nex.js JS kind of like I don't know like a facade in front of it if you wanted to but uh otherwise you'd do like these separate forcell projects and I was like I never I just didn't even think that that was like an option. So thought it was really cool to kind of uh I don't know piece these things together. So I haven't got the final result um as I drive to CrossFit right now. That's actually what I'm having copilot do. So kind of crazy how this can work, right? Like I mentioned to you that my colleague took our repository, use claude to go port it to some other format in a new repo, prove that it works, and now I'm porting his stuff back into our repo. um and and structuring it such that we have like this uh these multiple like root level agents in EVE.

So I don't know if it will be done by the time I get home. I suspect not because um it really needs to run these e val uh which are going to take some time. Well, I instructed it to so we'll see. And what I'm hoping is that whether it's like I'm hoping tonight I can see some progress where I'm like this is on the right track and hopefully by the time I wake up in the morning I'll have something I can I can start reviewing. Um but like I was saying it's like I don't think it should fundamentally be too um too crazy. I think it's a lot of file moving to be honest and then pointing um some of these directory structures to particular versell projects. So, uh, pretty cool. But I'm hoping that that isolation, uh, reduces I I think that was one of the big issues is like the way that we had it.

I didn't realize this, but um, it was technically loading up a lot more context than it needed to cuz it was one agent that was trying to to consider all these workflows. Now that I think about it, I thought what I was building to optimize was basically like dynamic skill and uh and tool lookups. So you'd have one agent and then you keep it lightweight and then it basically based on the request figures out what it should route to. But like that's not like that didn't end up working and this other pattern is essentially doing that more structurally. So, um because that is a deterministic thing, right? I was basically going to use an LLM as a router and uh why do that uh when you can make it deterministic? Um so, pretty cool. We'll see how that goes. Uh but yeah, he for you know context, he had it optimized down to like 30% of the token usage um without any quality loss, speed, performance, everything was good.

And yeah, so that's it on the Verscel stuff. Um, I think I don't know. I want I don't know how to like share this because it seems like kind of weird and I'm I can't I feel like I can't be the only person that's having this experience, but um maybe maybe it's only a few of us. I don't know. Uh, I'm finding that like I I'm pretty sure I got some amount of ADHD and um focus like focus is challenging, right? Like when there's something I'm really interested in, it's challenging to to not do it. And um and otherwise like things can be overwhelming, right? I I don't know um if others have kind of like experience like this and I've been finding that with uh different things I'm doing with AI, I'm getting like like a basically like a pretty big issue with like

this feeling of like time warp where I'm sitting down to go do something and I I don't like I I'm not sharing my screen obviously because I'm driving and I'm not at my computer. But, uh, I usually have like 20 co-pilot CLI windows going. Like there's my screen is in four quadrants uh, for co-pilot windows and then a window in the middle for me to be reviewing like issues and PRs. And there's like roughly like 20 sessions going at a time. And I I feel like I sit down to do stuff and then I like look at the time and like I haven't I haven't eaten breakfast and it's like 300 p.m. And then I'm like, "Okay, well that's not good." And then I'm like, "I got to make sure that I'm, you know, getting to bed on time and stuff." And then, hey, like it's uh it's 9:00.

It's like I got to start, you know, finishing up here so I can make sure I'm in bed for 10:00 and then it's like 1:30 in the morning and I'm like I what happened? Um it's Yeah, I don't know. I I like I said, I didn't really know how to bring it up or talk about it, but like that's a thing that's happening right now for me and I I don't know how to feel about it. like um yeah, I I guess I really don't know how to feel about it. A part of me is like, hey, if I'm if I'm building things, like I like to build things. It's not like I'm not enjoying my time and um I'm also trying to be productive. I'm creating things, but I'm getting like a little bit nervous that I um I feel like I don't have an ability to to properly manage my time like more than usual.

I think it's already maybe a little challenging for me. Um what is this person doing? I have to I'm going here. What are you Okay, let's do that then. Um they kind of you can't see cuz they're on the other side, but they basically pulled over into not a spot and blocked a driveway. Um but yeah, I I don't know what to think of that. It's kind of kind of scary. Um but yeah, it's uh like a time management challenge. I'm feeling like I'm kind of getting sucked into a like a black hole of um you know one more prompt or like whatever there there's maybe like it could just be the nature of like it's allowing me to multitask too much and like regardless of it being AI or specifically with code maybe if I had some other experience uh or I could multitask like uh that that it would be the same kind of problem.

So maybe it has nothing specifically to do with AI or coding. Um but yeah, I think it's kind of enabling that uh that problem for me. So I'm saying that out loud to like self- acknowledge and curious if other people are noticing that. Um yeah, something I have to kind of watch because I really felt it this weekend where like today's a good example. I'm going to the gym. It's 7:30 at night. Um because I was supposed to go this morning. Like I don't know what else to say about that, right? Like that's how much time is slipping. So I'm going to go squat, bench, and run. And then we'll we'll do this on call shift this week. But thanks for watching. I will see you in the next one. Take care.