Can I Blindly Trust AI With My Code Yet?

Transcript is auto-generated and may contain errors.

Hey folks, I'm going to do another little quick entry for some AI stuff just because uh had uh yesterday off as well. So, I'm going to be off all this week because my in-laws are visiting. So, um just a little bit more time because I'm not working even though I'm got activities with the in-laws to mess around with some AI and stuff. So, um, the last video I was talking about kind of shifting my workflow over to, um, having what I would say is probably a pretty common thing, like using pull requests and stuff like that. And that's a shift for me because when I've been building with AI, a lot of it has been, you know, uh get it to the point where it's ready to make a commit and then uh I'll kind of do the the local review, make it kind of prove itself to me and then I'll commit and just push up.

And uh the the original reason for that was just like the pull request was just like extra ceremony. like why make me go to you know go on to GitHub review code when I can just like look at it on my machine sort of thing. But as this stuff's been kind of uh growing and scaling across more repositories and stuff for me um I just kind of had this realization like duh like we normally I say normally as in in development teams we have something like putting a review up like with a pull request people you know you run your automation you get a green light people sign off push it through. something like I don't know why don't know why it took so long but got to shift to that. So what I did was I have this uh personal repository for myself. It's just it's called Genesis.

I put my templates um my instructions. When I say templates I mean for like if I wanted to scaffold a new project that kind of thing all the things I kind of learn from uh developing with agents I push into this repository so I can reuse it. And so this is one such thing. Um I basically just converted uh the Genesis repository itself to kind of do this uh sort of like auto approved process. So the idea is that if I if I as in Nick do a uh you know pull request and it passes CI/CD especially because it's my own repository um I don't need AI to sign off on my changes. it'll automerge. Um, and then if AI like C-pilot, if I tell it to go do some work, I can configure the repository to either require a human approval or um or not, right?

Like as long as CI/CD passes, then um it'll it'll automerge. So, um I think for some repos where I'm a little iffy on the patterns and things like I haven't yet proven that AI is building with the patterns and stuff I want, then I would uh automatically sorry I would manually sign off on the changes and then have it merge in after once I approve. Um and then I have some repositories where it just literally makes sense to like let it keep blasting code in because I do have it following patterns I want. Um there's tests in place. It's not um it's not code that people are paying money for. Um like there's just more flexibility that way. And so I I have this now in my Genesis repository and then as of this morning, oh I guess between last night and this morning um kind of having it trial out for real across my uh starting on my other repositories.

And so I what I did was I have sort of this uh scaffolding if you will in Genesis where if I'm making a new project it will get set up uh with you know uh these GitHub actions and things like that for for auto approving automerging. Um, and then I have a skill that I I put together. And I say that I put together as in co-pilot did this. Um, sorry, one sec. I got to switch lanes. I just need some more space. Um, it put together a skill that for all these repositories that I've already built that I can kind of port them over to using uh this pattern. So um basically it will walk co-pilot through doing the conversion. It has like an audit.

Um and so even the first one I did it ported over successfully um using the skill and then co-pilot said like as it was porting it it's like by the way you know this skill and this sort of workflow uh they missed like whoever put this together missed you know uh this kind of scenario. So, um I could feed that back to the original agent and just say, "Hey, like you know, um here's what the first developer said, following your skill, like you know, what are your thoughts on on fixing this? Is this a real problem?" And sure enough, it was like, "Yep, you know, missed a missed an edge case." And it can go back and uh fix up the the template itself and the skill. So, what's cool about that is that I got to trial it on one repository. Uh, the agent that did that corrected itself along the way and it just kind of said, "Hey, heads up." You know, let the original author know.

And now when I go to uh to port the other repositories, again, these are already existing repositories that have plenty of commits and stuff. um when I go to port those ones hopefully the the patterns more established and if it catches anything in the audit as it's going through I'll kind of feed that back into the original thing and that will just kind of uh harden it to any more weird edge cases going forward but again the the whole point of this is that I have some repositories where I can truly just have co-pilot go build something and automatically merge and I feel good about it. Um, now how did I get there and like uh are there still escapes and stuff like that? The answer is uh yes, there are still escapes. Um, I got there by sort of giving it a little bit of trust.

Again, this is not code that's shipping into a paid user hands. So, I'm just being a little bit more flexible. And what ends up happening, and I've talked about this a bunch before, is that I need to assume that by blindly trusting it to land changes. I need to assume that it will start having patterns that I don't like that it will start doing that I would think is, you know, otherwise stupid where I'm like, "Oh my god, how could this be in my codebase?" I it sounds kind of backwards, but like I need to let it happen because I don't know what I don't know. And probably sounds kind of weird, kind of stupid what I'm saying, but hear me out. Um there are patterns and practices that I know, right, that are things where if you ask me like, "Hey, Nick, how would you go build this or how would you guide someone through it?" I can tell you.

Um, but there are also things that like I simply don't know, right? Like I I haven't come across the pattern and then sometimes I'll see it put together and I'm like, "Oh, like now that I see what you're doing, like I wouldn't do that. Here's why." So I need to let AI sort of come across those I need to let it happen or else otherwise I'm in the process up front, right? I'm I'm in the loop. so to speak. And I can't be if I want to scale it. So, I need to let it have some flexibility. I need to let it do some stupid things. And then, like, I'm saying stupid with air quotes. They're just things that like I I wouldn't recommend like maybe it's duplicating code.

Maybe um maybe it didn't refactor something and it's like cheating to like just I don't know like do something in a way where I'm like, "Oh, you shouldn't be exposing something, whatever it happens to be." And then I need to be able to over time see some of these patterns emerge so that I can go okay now that I see how this is coming about like how do I correct this? And what's cool and like the reason this has been working is that um if you've heard me talking about uh these instruction files that aren't just your agents MD file or your claw MD or your co-pilot instructions MD, not just simply the top level instructions. You can do uh file path matching on instructions to be more targeted. And this is absolutely changing the effectiveness for me. So combination of that plus um any other type of guardrail that you can put in place and so as a net developer uh Roslin analyzers are a huge huge win for me.

Um, and for different languages and different text stacks, there's different tools. Like, think about think about it as llinters, right? Like a ROS analyzer is a very powerful llinter. So, it's not just style um, but it's also like uh like patterns because it can actually analyze uh the code, not just like look at white space or something like that. Capitalization, it's it's more than that. So, you tackle the problem from two sides, right? you're you're saying, "Hey, agent, like here's instructions for the right way to build things." So hopefully as it's writing things, it does it properly the first time, right? Ideally, um, but there's always going to be escapes. It's it's not going to be perfect. It's like even with a human, um, if I told if someone told me or I told someone on my team like, "Hey, like this is the pattern we follow.

We we don't use pattern X, right? Someone will go, "Cool, got it." And then inevitably there might be a scenario where they're like, "Uh, I mean, like I'm in a time crunch or whatever and like pattern X just seems to like solve the problem. Like screw it. Like here we go." Um, or they forget whatever I forget and it happens. So to stop that using guards from the other side, something like a Roslin analyzer is like, "Hey, like I see that you tried to do the thing that we said you're not supposed to do. I'm stopping you now." If you only did the second part, you would have a lot of inefficiencies. And I can say that as someone who is putting in more Roslin analyzers and not getting the instruction files uh up to par. And it would just take longer because it would write code and then have to rewrite it, right?

It would write all this code, go to build it, and then it's like, oh, like I'm not allowed to do that. Why aren't I allowed? Let me go figure out and then it has to go rewrite it and and hopefully the analyzers were giving enough guidance on like how to write it properly. That's like retroactively. And so putting in only instructions means that you're hoping that it's going to follow your instructions perfectly. And hopefully you know by now, at least the current state of models, that's not what happens. So getting it from both sides seems to tighten that up a little bit. So that's been good. Um, so basically allowing agents to to start automatically merging has been working pretty well. And to give you uh just a quick example, I was showing this on stream pretty briefly a couple weeks ago, but like one of the side things I'm doing, I uh a couple years ago, so maybe I'll go back a little further.

Um before Microsoft, I used to work at a digital forensics company. And um when I moved out here to Seattle, I had a friend whose whose family uh they basically the kids had never seen their their childhood photos ever, like growing up. And that's because they were on a computer and the computer wouldn't boot anymore. So they were like, "We just kept the computer in the garage, but like you know, we can't it doesn't turn on, right? We can't get the pictures off." So I said, "Give me give me the computer." Um, and so I wrote a tool that can go recover pictures and videos. And I was able to do that just because of basically working in digital forensics for almost a decade. And so I put together a tool that can do that. And then not too long ago, um, especially just because I am trying to mess around with more with agents, I said like, let me just keep asking the agents to build more.

Like, let me just allow them to go build, right? So, I this is something where, you know, no one is actively paying for it. There's no customers. Um it's something that like domain expertise-wise like um while I'm not actively working in digital forensics, there's a lot of there's a lot of experience that I gained from doing it. I I don't know like this combination of like from a software architecture perspective because I've been building software for a long time plus in the niche of like writing a um you know digital forensics product in the past. I'm like this is I feel like this is the perfect example where it's something I can sit back and kind of babysit having agents write code.

So it's been a really good playground where um I can at a high level say like I want to recover you know these types of images now this file format like go collect the data right go research this file format I want you to you know go make sure that we have uh real data to like test against um go build it out and and because it's in D#. I'm like, follow my C sharp patterns. And I have like, you know, two agents that have been running for a little while. Um, but I would say especially over the past week and especially over the last couple days with some of this uh automerging stuff that I've been talking about. I have two of them running.

one is kind of doing um different file formats to recover and the other one is doing more of like what I would call the framework the engine um so to speak and uh and and also like they're both kind of mixing in some refactoring and tech debt but the way it's working is that across these two sessions I'm just like hey like you're focused on this one part of it you're focused on like the file formats and I just let them kind of work. And then periodically I'm checking in on some of the code and I'm like, "Oh, I I notice that we've seen like this pattern come up a lot and like okay, like I don't like that. Here's what we're going to do." Right? So to be a little bit more concrete, one of the most recent examples, um, again, this is a little bit more net and Centric.

So I'll just try to explain. Um, in C we have something that's called a record. And a record is used as a data transfer object. So it doesn't it's not supposed to really have logic on it. It's supposed to make it so that if you wanted to have like a couple of fields or properties, you you can put them onto a type and then use that to pass around. You're transferring data. And so it is something that works really well for um I don't know being being minimal in terms of boilerplate whereas traditionally the way that you would do this kind of stuff to exaggerate it you'd have your type you would have uh the constructor parameters that you're passing in to initialize the type right so if there was you know even if there was just like three fields that you were putting on

right so you have uh three numbers that you're trying to pass into this thing a 3D position let's say you'd have those three constructor parameters you would have three fields uh or three properties and then you would assign those uh properties or fields in the constructor. So that means for every field you kind of have it mentioned in three spots the constructor parameters the fields and then assigning the fields inside the constructor. So like three lines per per thing and with a record it's just one which is really nice. Um now what started to happen was not only was uh were these agents not using records even though I have that in some of the guidance um and I think part of the reason why is that they were over focused on putting guard clauses in place.

And so a guard clause again if you're not familiar is just like uh doing parameter checking right so when when constructing uh let's say this 3D point maybe we're not allowed to have negative um you know xyz values again I'm just making this up so you could have a guard clause that says when you're constructing this if you pass in anything less than zero we'll throw an exception so that means one extra line per um per field. So now all of a sudden these classes that really should in my opinion should just be super simple um they're like sort of ballooning out of control in terms of just boilerplate code. And this is a minimal thing but this is one such example where I'm like I just don't want my code to look like this. I think it's extra bloat for no reason. Um, and it's easy like it's so simple when you have stuff bloating like this that someone misses a guard clause, right?

You have a class that has 10 parameters passed in. Oops, someone forgot guard clause number 10 because it's just lost in this noise. So, um, from looking at that again, few few things have happened. So, I'm like, okay, I don't want that. Do I have something that number one can prevent it? Number two, um is there even like a like a language construct or something that makes this go away? Because I could tell it just use a record instead, right? That'll get rid of most of that. But it doesn't have the guard clauses, right? So like I could tell it, hey, look, like I don't really give a about the guard clauses. Like it's just kind of overkill. just make it a record and call it a day. Uh but instead I could say cool records don't have that support without the same kind of bloat.

So I could go to needler which is the dependency injection and source generation library I have and I said I want I said hey like to the the needler agent I have this pattern I want to come up with a solution for it and so basically let me decorate the fields uh of a a record or a class with with sort of these uh these attributes on them that say this is a field that will be injected in a constructor and is uh needs like a guard that looks like whatever, right? Can't be null, can't be negative, can't whatever. And so when you decorate the fields like this, it will automatically generate a constructor for you. And so I now have this mix of the best of both worlds, right? I have um guard clauses, I have the boilerplate code reduced, and then I have it in my reusable library.

So then I feed that. So I have it into Needler. It's a first class offering. And then I go back to this digital forensics codebase and I say, great. You now have an instruction that says you should use this pattern instead. So use these attributes to put your guard clauses on. Um, and if I stopped there, it wouldn't fix the problem because it relies on the LLM doing the right thing, making the right choice. And that's not a guarantee. So then there's one more step, right? So in this codebase, I need an analyzer that says if you have code that looks like this, it must follow this pattern. Not it should, not please, uh not hopefully, it's it should. So that means that even when it does decide, oops, I'm just going to follow the old pattern, um it can't compile.

And so that's like one such little loop where um I take that feedback from reviewing and I'm like you know not not what I want here's what I want instead and then uh kind of just put it through all the all the stages so that going forward it will build things properly and then I repeat this right so I will periodically look at um changes that are going in and go okay like here's another pattern I don't like or here's one that I do like like the opposite can happen where I'm like, "Hey, this is a cool pattern that like it happened to do and now that I see that I want more of that and um and then I can do the same thing where it's is it just a matter of guide it with instructions? Is this something that like it whipped up a pattern um that I could be reusing even across other code bases?

And if I want to enforce it, do I use an analyzer?" And again, the whole reason I'm doing this is not because I'm like, "Hey, look, uh, you know, digital forensics is going to be my my path to freedom." It's like, this is a really good playground scenario where, um, I can practice this sort of agentic loop and, uh, let it build things. And it's been pretty cool. So um what I'm noticing now is that I have some projects where like this doesn't work well and I'm trying to figure out why. Right? So a good example is there's parts of brand ghost which is something that I I do sell. There are parts of it where if I'm asking an agent like um let's say I want to get it to a point where I could be more handsoff with it. There are parts where I'm like I just can't.

Um, not not even from like a safety precautions uh perspective, more like I can't trust that it's following the patterns I want. Like even if I got it to the point where it was like I'm going to build this whole feature end to end, we're going to look at it in the pull request. I'm like uh it's it's often just not following enough of the patterns I want. And so one thing I realized recently was a lot of my instruction files in Brand Ghost were just out of date. Um there was a lot of custom library stuff that I built into Brand Ghost that uh I had already factored out into some shared libraries. So they had you know they had better analyzers. They had uh you know more robust support.

So this is a good opportunity where I'm like okay if I pause on some of the architectural things in brand ghost if I pause and I go let me shift them to align on my standards that's one step towards having uh you know repeated patterns that I've seen working in other code bases. So let me try that. Um so I'm doing that with some of the brand goes codebase. Um, yeah, just trying to get to the point where I can regardless of whether or not I let the agents automatically merge in because I think everyone will have a different confidence level with that. I certainly do across my own code bases, but getting to the point where I can review the code. If I could be fully hands off, review the code and then I'm just looking at whether or not it's doing what I want versus like sorry, what I want in terms of the feature delivery versus um did it follow the right coding conventions.

I don't want to have to think about that anymore cuz I think there's enough guard rails we can put in place. So anyway, that's what I got going on. I'm going to play around with that more today and we'll see where we get to. Take care. See you in the next one.