Let's Talk About Evals For AI Agents!

Transcript is auto-generated and may contain errors.

put my windows up here. Uh, hey folks, we're going to talk about evals in AI development, testing, that kind of stuff. And I wanted to talk about this because it's a new area for me. I am inexperienced and with that said, it's like kind of a good thought experiment to talk through things. And of course, if uh any of you are in the comments and have more experience with this or just different experience, I'd love to hear from you. Uh it's an opportunity for me to learn as well. So, one of the recent videos I did here on Code Commute, and sorry, I'm super out of breath cuz I just finished CrossFit and I'm clearly not in good shape. Um, one of the recent videos I did was talking about testing and um, sort of my my philosophy on testing, how I approach things from, you know, different types of tests for different purposes.

And I I look at eval kind of like another whole layer to this sort of thing, especially as uh there's more like AI like built into systems that we're that we're creating for others. So yes, of course, we use AI for our own development, right? Like I I will talk about how I use co-pilot for building things. So not only do we use it for developing, but we embed um uh like depending on the product and service you're building perhaps like agentic workflows inside of the products that we're that we're creating. And then so one of the challenges with that is that because LLMs are not deterministic um if you think about how we typically write test tests um you'll have you know scenarios that you're exercising things that you're asserting on and we're we're always saying like we need to have deterministic tests right

because if you don't good luck with like your your continuous integration and continuous deployment systems where you know you're you're trying to run a suite of regression tests and every time you run it you get maybe different results, right? Like that's not going to be a good time for anyone. So for years we've been saying like well we need determinism and the the super challenging part is that like you can't really apply the exact same thing to to LLMs like especially if you're using a real LLM call. So um I think there's some interesting overlap with I don't know like strategies or techniques we can use in eval to to take just a a moment of pause to like a kind of a line on what I mean by an eval.

So if we are talking about testing some of these uh agentic systems whether it's a you know a model like you're trying to check the behavior of a model you've created what you might call a custom agent whether that's um how an agent uses a particular tool or a particular skill or a chain of all these things together like a workflow. Um the eval is basically the test that's over top of of these things. And so, you know, you could argue in maybe the most basic case, right? If you have a tool that you're exposing to an LLM, that tool can probably be tested traditionally, right? The tool itself is not doing the LLM call. I mean, maybe depending what your tool is doing, it's calling out to something else, but maybe that's something that, you know, you can mock out or or test in a traditional sense because it's not the tool is not um agentic.

It's provided to to an LLM in an agentic system or workflow to use. So you could test the tool traditionally, but you may want to have an eval where you can say given some model and some prompt, does it use the tool properly? Like have I built a tool that an like a model can use properly? And then you might swap out the model. So you might say like I'm going to use Opus 4.6 or Opus 4.8 and like those are two anthropic models. Maybe I want to use a sonnet model. So another one from anthropic. Maybe those all work great and you switch to a GPT model and it doesn't work right. So like you end up testing your your tool which maybe is tested traditionally, but you test it in a scenario with a different model and you would use something like an eval for that.

And I have to sneeze. Pardon me. two. Okay, we're good. Um, and so once you hook it up to actually run with an LLM, that's where you have this non-determinism. If you have a really simple scenario, like super simple, you might still have an eval where you run with an LLM, like a real life LLM, and um and your assertions, your checks are actually deterministic. And that could work, you know, very well depending on what your your situation is and how and how maybe how simple it is. Um, but like in my opinion, there's always, I shouldn't say always, but for at least for now, uh, there's seemingly going to be some risk that you don't have, you know, uh, a perfect result. So, if you were to, you know, depending on how you're telling the LLM to respond, like basically say pass or fail, right?

Um, maybe I'm just making this up. Maybe like, you know, 95% of the time it responds with the word pass when it's supposed to be pass. And then like 4% of the time it's pass with different casing like a lowercase P instead of an uppercase P. And then the remaining 1% of the time it says like true like pass, you know, to indicate like pass was true. And so you can end up with results that are just not perfectly deterministic where you know if you assert that it's exactly pass every time um it won't be. So when trying to build more agentic systems and if in my previous video I was saying like testing is really important to me. It's really important for me to have confidence in my my agentic workflows that I'm building. And so eval being the tool for that. Um I'm just like running into a whole world of pain pain uh good learning but it's frustrating.

Uh and especially I've noticed that using AI to write e vowels has been uh kind of like what feels like a huge step backwards. And I want to explain myself a little bit. Um I feel like today right today using models like I don't know like from even like Opus 4.6 uh through Opus 4.8 I haven't used Fable yet. I've been using the GPT 5.6 uh sole model. um like anything that's more modern feels like for writing tests it's actually doing a really really good job for like traditional tests and like early on I noticed a lot of stupid stuff and I'm not saying it's like perfect now but I noticed a lot of stupid stuff where it was really hard to to um for me to trust agents writing tests because my experience early on was that they seem to go cool like the scenario we want to check is, you know, uh given A um for system B we get result C.

And so what would happen is if you had testable code, it would do it like it would probably get it right. And if your code was untestable or not like super obviously testable, it would fumble around. And so like one thing that you can do is like to try and mock some things out. And there's I mean some people are totally against mocking. I'm not. But it would do things like it would instead of mocking a dependency, it would mock the thing that needs to be tested, right? And so there's a big difference there. When you're mocking a dependency, you're basically saying for this scenario, I'm going to control this dependency. And then that way I can kind of steer the scenario and you know uh see how the the system under test behaves with that when it mocks the system under test you're not actually testing anything real and it would do it confidently.

So it would be like cool I added all these tests you get what look like if you're just reading it like on the surface a whole bunch of tests scenarios seem like they make sense. Uh even if you were to scan the code in the test, if you were to scan it high level, you might go, "Cool, I see the setup. Like it seems like the flow is right. This is good." But if you're not paying close attention, you would miss that. Like instead of creating the system under test, it creates a mock of the system under test or a fake. So it's literally not testing anything real. And I used to run into this a lot, especially um when I've been making a couple recent videos in my recent live stream. I was saying for front-end stuff, that's the kind of thing where like I I have less experience in um admittedly when AI is putting stuff together, I'm looking at it quicker, right?

Because it's one of those things where I have less critical feedback for it. And so there were a lot of misses that way early on. And so more recently, I feel like I have a lot more trust in uh AI writing tests for me. Um I find my biggest complaint now is like it's either sometimes not sticking to patterns that I have. And again, even that's getting better, uh or it has some things that are on slightly more complex scenarios and it will have uh something that's a little bit more flaky than I would like. So, it's like waiting for some result and it's like cool. I'm going to I'm going to wait for a result this way and it would maybe introduce like delays into the test. Uh, and it runs it a couple times and it's like cool, that passed, but then on a different system or like you know in GitHub actions it blows up.

So, just like some stupid things like that now, but mostly pretty damn good. I noticed that with Evals, it feels like I'm even before what the earlier models were doing. And don't get me wrong, I think these are more complicated, but um the level of confidence that it's having, building out scenarios, evaluating them, when I read through what it's doing, I'm like, this is so wrong. It's so um I don't know like it's so uh I what's why can't I find my words today? It's not valuable. It's it's delivering no value and it's it's completely wasteful. And so for me this is a couple couple different signals, right? One is like this is new for me. So, I'm having this experience where I'm like, I don't know the best way to do this. Admittedly, I certainly don't. Um, and then that means that as someone trying to drive and steer the AI, I I am very inexperienced at trying to tell it what to do properly for an eval.

All right? And so it's a learning for me and I have to kind of like learn the hard way and then go okay like I see what it's doing like how do I in the future keep it more on track before it goes down this path. And sometimes like I don't know it's the wrong path until I see it put together and I see it kind of blowing up in my face and I go oh man like that's certainly wrong. So I want to talk about a couple of examples. Um there are like when I'm kind of working with co-pilot and for what it's worth I'm uh for for this conversation the model that I'm using is the GPT 5.6 sole model with like max uh reasoning and then like max context in case you're curious. And so I might have a conversation with co-pilot about the scenarios that I'm interested in covering with an eval.

And so we have this discussion and we agree on the scenarios. And this is more like a highle like this is the the scenario and then like here like high level again not exactly but here's kind of what I'm expecting looks good for this scenario to you know what I would call a pass. And so if I let co-pilot go build that for me, I'm noticing that uh say there's like 10 scenarios I asked it to put together when I read the prompts. So this is an example of like use a skill or use a mini workflow with an agent and like let's see how it behaves. When I read the scenario, I realize that the prompt that it's providing is like it's given in a way that Whoa, thanks car. It's a lot of people slamming on the brakes. Cool. I've only had that happen a couple times in this car where it does it on purpose, but um you probably heard the beeping.

I don't know if you can see it visibly, but um it retracts the seat belt like super hard and there's like nothing in front of us. So, everyone in front of me, line of cars, just like everyone slammed on the brakes and we're already like back up to speed. So, I don't know what's going on, but it's probably someone switched lanes or something and someone hit their brakes. Um, so the scenarios that I would put together when you read the prompt, the prompt is like it's giving away too much information, if that makes sense. So, it's not asking it like a person would. Like if I ask you a question, what I probably won't do is also give you the answer at the same time. Um actually when people do that it almost feels facitious or condescending if I ask you something and give you the answer in the same question.

It's like almost like insulting your intelligence. Um so that's kind of what it's doing. It's like it's saying like I have this scenario like what what's a prompt for this scenario? And so it's almost like defining the scenario in the prompt. And so of course you know there's a huge bias for the agent doing you know the the right thing. And in my opinion that's like cheating the whole point of it because I want to make sure that the scenario is realistic. Is this person going to speed up? No. So that's one thing and I have to keep catching it. Um and again uh when I talk about things like this, this is not me blaming the AI and saying like therefore AI is stupid. This is like I have to take some responsibility over it because what I probably need to do is when I'm having that conversation with co-pilot with whatever model to say like specifically when we're creating the scenarios and we're defining the prompts the prompt itself should not be leading.

It should have like as little information as needed to to make the scenario represented and asked in a way that like a human would if it's truly a scenario that's like a human's asking the question. So I have to like figure out for myself how I do that better. That's one thing. Next is that um when it comes to actually writing evals in terms of like the the framework, let's say um this one is really bizarre to me and I can't like fully rationalize why I see this pattern happening a lot and it's like doesn't matter if it's stuff that I'm doing at work like this or at home. Like maybe I'm the I'm the the problem cuz I'm the common denominator. Uh I've seen this in C. I've seen this in different uh different programming languages and stacks.

But if I say like from Python to TypeScript, um if I'm like, hey, say we get the scenarios and prompts figured out, I notice that when it comes to the the runner for the eval, it always seems to like want to to create its own little runner or harness, even if there's something in place. And I can't figure out why it gravitates so hard to this. So for example, um some of the stuff at work there's uh a particular framework that gets used and so we can put uh you know shared skills and stuff into a spot and they have for the entire repository they have like an eval harness and which is awesome, right? Because you don't have to come up with that yourself. you use the framework that's been built, drop stuff into place, like it's great. You don't have to reinvent the wheel.

And so, no matter what, it keeps trying to reinvent the wheel every single time. And so, if I'm not paying close attention, it will go through like do building all this stuff. And by the time I go to review it, I'm like, "Wait a second." Like, why is there a second runner here? Like, what why did you go build this? And it's like it will say stuff like oh um you know I needed to call this particular tool and like the you know the the main runner like doesn't have that or the more recent one was like scoring. So uh I want to talk about scoring in a moment but basically to to judge the evaluation that it's doing this scenario. It was saying like I need to go invent like a new system for being able to score properly.

And what it did was it built uh a whole new thing that would do like basically reax matching on the output which again I'm going to come back to this in a moment but basically it build its own scoring system. Uh so I've seen it do its own scoring system, its own runner. Um and every time it's like oh I seem to have like a special snowflake case so I just rebuilt the whole thing. And I'm like great. Uh that's stupid because if you like looked closer, the original system like supports this. You like have to pass a different parameter or something. So I don't know why this is happening and it's not specific to this one at work. Uh I'm was saying the other day on a uh code commute video that for brand ghost we're switching over to using um Versel's like Eve framework and it was doing the same thing.

It built its own runner and I was like, "Dude, like literally it has it like comes with one." And I told you to like, you know, we started off by like here's all the documentation, use a fleet, research this, like basically come up with all the information we need and it still decided it was going to make its own runner. And it would say thing like when I ask why cuz I'm always curious so I can learn about where things are going wrong. It said something like, "Oh, I tried to run it and like I hit like a socket error." Like basically I tried something once hit what seems like probably a weird flaky edge case or something. So I just rebuilt my own. And I'm like that this is it's the wrong thing to do, man. Like now now instead of getting onto this new framework, we're getting onto a new framework and immediately deviating to go do something custom.

Like absolutely not. So this has been a real weird like pain in the ass thing that seems to be repeating. And so I don't know. Um like I said, I'm the common denominator, so I got to figure out what I can improve there. Um, on the scoring part, because this one's kind of annoying, too. Um, I guess there's probably some different takes on, you know, what makes an effective eval. Um, I think that you could write them where they have hard gates on them, right? So, you treat them more like a pass or fail kind of, uh, almost like a unit or functional test. But the other thing is like running evals that just have scores and maybe you hard gate on like you know meeting some minimum threshold for these scores.

But the idea being that you run a scenario and then you have different metrics if you will that you care about for the scenario and they go from like you know one to 100 or 0 to one whatever you want to have as your range. And that way you have a bunch of these scenarios. So maybe you can swap out the model um swap out different parts of it and then you can compare and contrast like how does it do with this scenario. So you get scores instead of it just being like all of these pass or all of these fail. You acknowledge that like it's basically it's a sliding window. It's a spectrum across these different facets of the scenario. And so that way like the way I think about this is that if I have a bunch of scenarios, I get some scores. If I'm making adjustments, can I keep like what I would like to do is set like a minimum bar for a score.

So maybe day one that minimum bar is like I need a 50%. Right? If these things can hit 50% uh across these these metrics, I care about those scores, great. And then I go from there and I try to make improvements and now if I can get it to to 60%. Maybe I go cool like the new minimum threshold is maybe it's not exactly 60 but I bump it up from 50 to something else. All right. I I move the floor up and then I keep making adjustments and keep trying to drive that floor up across these scenarios. So that's been the approach I'm trying to take. And uh yeah, it's been doing some really dumb stuff with uh with scoring. Like it it can't seem to figure out without a lot of direction that um there's differences between like a a deterministic check, an LLM based check, a passer fail check, and like a something on a a range.

So for example uh in my experience so far every time I see it doing like you know to evaluate something if it's doing a a direct string check like does this string contain exactly this or match this exact reax I'm like you know red flags alarm bells um because when I look at it usually it's checking for something pretty brittle and I gave you like you know an example earlier of just pass, but usually it's like checking for like 50 variations of something similar. And I'm like, dude, like this is the wrong thing. Like, you're trying to basically use uh very deterministic string checks for something, however it's set up, is like really um semantic. So either we change what's happening and we have it become more strict in the output or you're just using the wrong tool like you need a semantic check. So use an LLM judge to judge it.

So anyway, these are some things I'm exploring, but I just got home. So I will see you in the next one. Take care.