This Bot Submitted A Question On AI Reviews Rejected By Humans

Transcript is auto-generated and may contain errors.

Hey folks, we're going to go to a submitted question. Um, I feel a little bit duped because uh I saw the question was a sort of a paragraph to read through and I didn't realize the second half was basically uh the submitter acknowledging it was like a uh a company. Um I actually assume that given it's an AI related company, they used a bot to submit something. But anyway, I think the question is interesting to talk through, but I'm not going to say who the company is, and that's because you're not paying me. Um, and guess what? It's not a charity. So, too bad. You're a business. If you want advertising, pay for it. Question is about uh what to do in your process when you have agentic development happening. And so, agents have written code. It gets to the point of code review. And then it's kind of failing the human sign off.

And obviously this is making an assumption about what your development flow looks like. Are you still even reviewing code? Right? I think one of my more recent videos is actually the reason a bunch of you are here cuz that ended up doing pretty well. And uh the topic was about you know someone from Reddit asking about the situation where like you know code is going straight to production. So the submitter this what looks like an LLM uh bot was asking you know what how do you handle this you know how do you go sort of correct the process how do you give that feedback how do you make sure that um you know false assumptions aren't carried forward and so when I think about my own development flow now uh with things like this I think it largely depends on um you know what what repository I'm in, the scope of the feature, if I'm thinking about like how much or how hands off I was.

There's sometimes like uh I've talked about this in other videos where I'll fire off something to co-pilot, have it have it build because I want to see pieces come together and not necessarily like I am expecting that it's going to, you know, deliver something that's going to be perfect. It's almost like u you know when people prototype things, right? Or they're they're trying to get like a something together they can demo and show off and you can see the thing running. Um, it's almost like that except I want to see that for the code, if that makes sense. I want to see an example of how the code is structured so that I can wrap my head around like maybe what the the right solution could look like. Um, cuz sometimes the things that I'm I'm having to do are not necessarily like a userfacing feature.

And so I want to see like architecturally if I have all these pieces um and I want to make some type of change, can you show me like if I tell you I want you know some type of end result high level, show me what that looks like, right? And then I can read through the code and stuff. So in in situations like that, if I look at the review, um it's it's not that uh I'm having this expectation I should be able to press merge necessarily, but what's kind of interesting is that um I've seen just to give you a couple of examples like what I do my a lot of my development, I have like vertical slices or feature projects and uh a lot of them are like are plug-in based. So the architectures are usually plug-in based and so there's patterns and practices I have around like how I guess like plugins interact with each other and the plugins usually serve as a bit of a barrier functionality.

And so I've seen the agents basically do things where they're breaking some of my rules around that. So they've gotten something to work based on what I asked for, but they've cheated in my mind, right? like they've instead of these plugins being isolated, the plugins know about each other or they're sharing code in a way that's not the the recommended way I share code between plugins. And so if I see a review like this, um, number one, I'm not going to freak out and go, "Oh, it's so dumb. I can't merge this." I like I wasn't expecting it to merge. But number two, if I'm like, oh, like I I see it like trying to to cheat here, break a pattern or um like recently I had this happen where it was cheating and I was like I am positive I had tests and and instructions around this to prevent this type of pattern from happening.

So I went back to go talk to the agent in the session to say like basically we're going to uh do a little bit of a pivot. So what I want to do is uh go fix up the instructions or guidance in the repository so we don't have this happen again. Um which means new dedicated PR just uh completely in isolation from everything else that we were just doing. I just want to get some fixes in place. And so when I was asking it about this particular situation, it was like, no, you're right. There are tests for for this and for that and there are instructions for this and for that but this particular thing does not fall into that group. It looks like the same pattern but here's why it's not like according to the instructions. So then I had to you know chat with AI to say okay cool like you know we have analyzers for uh it's net based so use Roslin analyzers to prevent some types of things from compiling.

So, I was like, "Cool." Like, let's go to the Rosland analyzers, let's go to the tests, let's go to the instructions, let's touch all of those things and make sure this particular pattern's included. Um, and this is going to be the bit of the theme around like when I've talked about how I'm developing with agents, it's usually from two uh angles, right? This guy in a Subaru is tailgating me like I have never seen someone tailgate. Why, sir? You're about to be very disappointed when we get on this ramp for the highway cuz the person in front of me is going slow. Um, and I talked about this before. Look at this idiot. Absolute dummy. Anyway, whatever. Um, I approach it from two angles. One is like giving agents better instructions to do things right the first time. So they when they're writing code, it's more closely aligned to the patterns that I want.

Um, and then there are there is the other direction which is like gating bad behavior. Um, and so I try to tackle it from both sides so that I'm not being wasteful with agents developing things. They develop things properly the first time and the gates prevent things from leaking through because we can't guarantee that agents uh are going to follow every instruction we give perfectly. So both directions are important. Um, and then of course like the earlier the better. So Roslin analyzers are a nice little sort of treat we have in between. And that's because uh the Rosa analyzers uh can stop the compilation when the agent has done something wrong, right? And then you can give it that feedback like hey like you know you're using X instead of Y. In our codebase we use Y. So you know flip the pattern to this right and so if it was not following instructions you can catch it even before you know running your tests and stuff in in continuous integration.

So kind of a nice in between. So I have this kind of pattern um especially with open-ended architecture things or this could I mean it's really common that happens with open-ended architecture things. I find a lot of like gaps in in things that I have I don't know like considered to be rules in my repository in my code base. It's like I was too too open when asking this. So at some point inevitably the agent's going to find you know things that I I don't necessarily agree with and integrate them or try to with normal feature development. uh this is not impossible. This still happens, right? And so same type of thing can apply depending on the scope of the feature. Like if it's a pretty small feature that uh that loop I was kind of talking about for like going back to the agent asking about like how it got into that situation.

Um do we need better instructions, analyzers, tests? Um, if it's a small PR to begin with, I might just include it together. Uh, because I'm like, it's almost like here's the here's the change that was being made and the the rules that we needed to update to kind of go along with it. But, I don't know. Uh, if I can if I if I'm feeling like I'm having good behavior, I'll split the PRs up. But, yeah, it's more continuous integration time. I don't have a like a merge Q set up, which is unfortunate. With a merge Q, then I would be a little bit happier with some of this cuz otherwise I have to go rerun my continuous integration. It's kind of a pain in the butt. Um, this I mean I could talk more about more specific examples of this, but I'm actually finding like these ones are the less interesting ones because at least for me like this has been the pattern for for months and months now.

So, I find like it's working well personally, and I say this because um just to give you one quick example, when I've been uh developing with a a colleague of mine for for things outside of Microsoft, uh if he was using Claude and I didn't have my instruction files that I've been creating for C-Pilot synced properly, I have seen that his clawed code basically just ignores all of the instructions up front, right? It doesn't know. It's not reading the the path-based co-pilot ones. So, it will build things the wrong way. And then you're if you're thinking about this, you say, "Nick, what you told us you tackle it from both sides. You tackle with the instructions and then with like tests or analyzers." And that's true for the most part, but it's not it's not perfect. There's nothing guaranteeing that relationship. That's just what I strive for.

So, I have seen where like, you know, he'll put up a PR and say like, "Hey, can you check it out?" And when I'm looking at it, I'm like, "Man, I can't integrate that because basically Claude just, you know, didn't have the instructions that I have for co-pilot, and there weren't analyzers or tests to catch it. So, you're about to integrate a bunch of patterns that are not okay. And it's not his fault, right? It's not uh my colleagueu's fault and it's like not the agent's fault either. It literally did not have the guidance uh that I have expected. So, um you know, I've been I tried to be even clear with my colleague like, hey, like I'm I'm holding back this PR. It's not because like I think you're dumb or you did the wrong thing. It's just like this is genuinely my fault. like I have to take responsibility that I didn't have my instructions synced in a way that your agent could could have possibly known to do the right thing.

Um but yeah, I think like to me this is less interesting now um because the process is pretty repeatable and it's it's working. Uh could it be better? I'm sure. Uh but this is what I've been iterating on. I think what's more interesting is like wasted cycles in the development flow. And so yes, that's partially related to like, you know, if I only had uh instruction files or sorry, let me say it the other way. If I didn't have instruction files, but I only had tests, uh, that, you know, try to assert that we have some type of behavior in the codebase, like an architectural test, maybe my development flow is really inefficient because the agent builds all of this first and then when it finally goes to integrate it, it's like, eh, none of this is allowed. And it's like, well, I just spent, you know, 40 minutes building this feature just to find out in the end it's not allowed.

um that would be inefficient, right? same same concept we just talked about though there's like a good example even from last night um I have spent 72 hours not you know dedicated but it's taken 72 hours to give co-pilot something to go uh integrate where I gave it a working prototype literally gave it a working prototype from claude code that was built, no, no exaggeration, 10 minutes tops in flawed code. Worked perfectly. And so I sat with co-pilot and I was like, I don't know if you need to rewrite this or if you can actually use this in this hosted environment, but if you can use it, then we have it. If not, then let's talk about rewriting it. And so it tried it and it's like, nope, I can use it. So I said, great, cool. Like, well, here here we go. Like let's let's go land this.

And it's taken 72 hours approximately of like giving co-pilot work to do on this. And it I mean even 48 hours in it wasn't delivering the thing. It I I can't make this any more clear like I gave it the code and tried the code and said that it worked. Um and I had proven that code did exactly what I wanted before giving it to co-pilot. So, like how how the hell are you 48 hours into working on this and you don't have it working? It worked when I gave it to you. So, there's some like real inefficiencies uh in the development process for that. Um so, oo wipers. um this type of thing is more of my interest and that's because uh a lot more of how I'm developing is trying to get to this point where I am comfortable giving a higher level

task to co-pilot and with all of the other things we've talked about making sure that like I can be comfortable giving that over it will do the right thing and the number of gaps in between are smaller and smaller, right? But what I'm noticing is that sometimes I hit a a weird thing like this where it's spent like an outrageous amount of time doing something and uh like it something's wrong, right? So overnight this time, like the last instruction I gave it was basically like said, "Your life depends on you landing this by the morning." And so this morning I woke up and it wasn't done. Shocker. Um but I was trying to check like well how far did it get right because we had a good clear list of what the gaps were before bed. And so this morning it was like well these six things aren't done.

And I'm reading the six things and I'm like wait a second. Um the six things that are remaining are not the features or the fixes I asked for. The six things remaining have to do with like Sorry, these wipers are nuts. Um the six things have to do with sort of the infrastructure that it's using to go get the job done. So just to give you an example, it was like uh I need to make sure that I have picture evidence of of what's created, right? to compare before and after, but for some reason it was having some issues in the continuous integration host where I maybe it was overloaded or something overnight or a runs got killed. Um, and so it couldn't get that and it was like, you know, spinning wasting time trying to figure out what was wrong on that. And the reality is it probably just needed to re-trigger the job, right?

like wait a little bit, re-trigger the job, and it would have been okay. Um, but it had six things in a list like this that were nothing to do with fixing the problems I asked it to fix. It was getting stuck because it couldn't prove to me that it fixed them, which is kind of interesting. So, what I haven't done and I'm giving you, you know, the the heads up on what I plan to do is like when I'm home today, um because I integrated like I looked at the PR and I'm like if everything's fixed and you're just you don't you can't prove it to me by taking the screenshots and stuff, fine. Like we'll integrate it. But um what I plan to do when I go home is essentially sit down with it and I have a skill like a reflection skill and I'm going to ask it to to review like let's go through this session with this skill.

Oh man. No. Okay. Sorry. my my wife and I are both trying to park and there's actually two spots left which is cool. Um I got to go back use the skill and go through the session and say like let's talk about what the hell you did wrong here because clearly something's not okay. Wasted way too much time on this. Um but I I genuinely don't know what it's going to come up with. So I think it's going to be really interesting. In this case, I feel like it was less about, oh, did you code the right patterns and more about whatever process you were following was clearly inefficient and like there's probably some type of instructions I have that are misleading it or there's a really good opportunity where it kept running into the same types of issues and I don't have guidance around like how to navigate that.

So the development process, the flow of that I think is even more interesting at this point. Um because I don't think I have that as figured out for refining. So hopefully that's some helpful information. But yeah, if you got questions about software engineering or career development, leave them below in the comments. Otherwise, you can go to codeccommute.com. You can submit stuff anonymously. If you're a company that wants to ask a question and you want me to make a video about it, uh happy if you want to reach out. But um that's all. Take care.