AI Slop In Code Reviews... What Do We Do Now?

Transcript is auto-generated and may contain errors.

Hey folks, we are going to experience devs subreddit and this thing is going to cause me to lose my mind with all the beeping. And the topic is probably one that's I don't know going to resonate with uh a bunch of people. Not not necessarily me. I think it's interesting, but it's about um how to navigate like pull requests and code reviews when basically everything's turning into AI slop. And it's funny, like as a content creator, this is what I see non-stop, right? Like you got AI posts and then AI comments, just people using AI to talk back and forth and it's super awkward and weird. But like obviously in software development, we are going to keep running into more and more of this. And I wanted to talk about this because number one, I think it's probably relevant for a lot of people, but number two, because I actually don't have to deal with this.

And I think it's probably a good exercise for me to kind of like try thinking through how I would do this. Um, and I say that I don't have to deal with this not because I feel like I'm immune to it or something. I just mean literally because it's not something that's happening yet. I don't know if the May maybe it never will. I don't know. But it's it's certainly not happening right now. And uh I don't want it to be a yet. So I think that means that I probably want to give it some thought. Um I guess my first thought with this topic is and it's kind of two parts, right? One is saying the person from experienced dev subreddit was saying that there's like code that goes up, right? So the person posting the code for the review, it's just tons of AI slop.

And then on the flip side, you get people that are using AI to review the code and just pasting in AI slop comments like they don't seem relevant or they're not accurate or whatever it happens to be. It's just slob both ways. And so I guess I guess my first thought is like sorry I'm just getting on the highway. The first thought is really may maybe it's like the volume of code. Okay. Like that's that's where my brain wants to go because when I think about these problems, I part of me doesn't think that it's like it's because of AI. I think that there are issues and then AI is exaggerating that. So the first example of this I think is probably the volume, right?

Because let's think back before AI and I know this is probably really difficult because that's all that's happening right now is AI everything but um you know people people get frustrated on poll requests and code reviews when it's like hey you put up you know 2,000 file changes like what am I supposed to do with this right like no AI involved when it's just a massive change and you're like you want me to go review this there's even requests that go up and someone's like yeah like I was you know changing a reference or updating a you know uh changing an API name and and so it got used in all these spots and it's like okay part of me wants to just trust you and be like sure that's all you did but at the same time I've literally been on reviews like that

and then someone's like oh yeah and then like while I was doing that I I sprinkled in a couple of extra little changes just to clean things up and it's >> >> So, which one is it, man? Like, do you want me to not look at this because it's all the same thing across 10,000 files or do I need to go look through 10,000 files to go find the little secrets that you sprinkled in so I can actually review them? Right? And these are just sort of things that we encounter in poll requests and code reviews like before AI. And I think one of my sort of biggest pieces of advice and it's obviously it's very generic and it's very simple but it's like smaller pull requests help right you can think through things you can actually get feedback that's valuable um or or there

is no feedback but at least people can understand what the change is and uh I don't know what the right threshold is like I don't have data to say once you go past three files or 200 lines of code like once you pass that it's magically all bad. I I don't know what it is and I think it probably changes realistically depending on team the composition of the team language all this kind of stuff but I think there is a a sweet spot where if you can in like on average stay below or within those bounds of how large a poll request is you can actually get more meaningful feedback. There's all other side conversations on top of this, like how to not be an when you're reviewing someone's code, how to be helpful, how to let people ask questions without feeling attacked, how to not take things personally, all this kind of stuff.

But I think part number one that I'm thinking about here is like generally I find AI will add in mass volumes of code. You have to try really hard. I mean, depending on how much of an expert prompter you are at this point, but I find it's like it's the go-to for every LLM to go add more code. Try it. Like, you know, pay attention to it. I unless you're literally asking it, go delete all this stuff. If you're like, let's go implement this feature or change this functionality, it's almost always just more and more and more code. Um, that's been my experience. I find that I have to, you know, really be um aggressive in telling it to, you know, to refactor things, to extract commonality, to reuse parts, otherwise just net newness. Go go go.

And so part of me again thinks that one of the big reasons we have so much AI slop or the um the the feeling that that's so significant on these poll requests is in part because perhaps the AI is not writing quality code. Sure. But I think the volume of it I think the volume is probably the biggest thing. So my first thing that I would recommend to folks is like how do we get that under control? How do we have more meaningful uh poll requests because they are easier to manage? Okay. And that has nothing to do with AI. I just think that AI exaggerates it. Next thing that I want to think through here is like um this is just practice for people in general again before AI like what is probably pretty good practice when you want to get your code reviewed. You should probably read it and run it first.

And I don't just mean like it's one thing to read it and run it like in your IDE and you're going through your code and you're like, "Yeah, this all looks good to me." Look at the diff, right? If you look at it locally in a tool and you can see like what's changed, like great. In my opinion, that's a good starting point. I would say another really helpful thing is to basically go look at the code in your review tool the same way other people would see it before you go invite them to review it. Why do I say that? Because you can do a precursor, like a a check ahead of time, a cursory check to see what it's going to look and feel like. Because I have had so many times in my career when I was before Microsoft, when I was still putting up more poll requests, well, I guess they technically weren't pull requests, code reviews.

Um, and you know there'd be like a stupid comment I left behind cuz I was commenting out a line to like try something and like or you know I had written code one way and then like after playing with it a bit I was like oh no that's crap and like left it in and like it was dead code nothing was using it and then I pushed it up for review and someone was like dude what the hell does this method do and I'm like uh I don't even know what that's doing there and it's like I would have probably caught it if uh if I had just gone through the tool the same way that other people did, right? Takes a, you know, a couple of minutes extra, but probably saves everyone significant time. And I think that again, this is just another practice that has nothing to do with AI that I think would be helpful for people to do.

I think that AI tooling is exaggerating some of these behaviors for us. Right? The AI wrote it. Like, I don't really know what it's about. Let me just push it up and like move on to the next thing. Right? It's it's sort of exaggerating the speed at which we can produce code and move on to the next. But that's a problem in this case because as you're doing that, you're not taking the time to stop and read it even as the person who's reviewing the code. Some of the the comments on the Reddit thread were like, you know, if you go to use git blame, it's going to have your name beside it, not claude, right? So, and I guess depending if you're using copilot or not, it probably says copilot there.

a lot of my commits in my own uh repository say co-pilot and then co-authored by Nick I'm I feel like I'm the co-pilot now and so taking the time to review it up front I think is just another helpful tip right because if you if you get in this habit you're probably going to notice the AI slop before other people get to it this is probably more challenging for people that are more junior but again this has nothing to actually do specifically with AI, right? If I had junior people on my team and they were putting up big uh pull requests, lots of files touched, and the feedback from the more experienced people on the team was like, "Hey, it's really hard to review stuff like this. Can we consider breaking it down?" You know, we could we could teach them that.

if it was like, "Hey, you should uh you know, you're getting like similar feedback like um it's probably good practice for you to go review your own code before you start adding people on." And if they're like, "Well, I don't really know how to do that. What should I look for?" I think this is a great teaching opportunity, right? If people are junior and they haven't done this kind of stuff, teach them. And it that has nothing to do with AI. It's just that AI is going to exaggerate that problem. So those are two things to start. Poly pulling someone over. Um what else? So we got AI slop in terms of I think volume not getting reviewed. Um I made a video before this one. Uh if I upload things in order, if you've watched that one, I was talking about putting more guard rails in place.

So, do you have consistency in um you know your agent instructions? So, are you using a custom agent? Is that consistent with people? Is your co-pilot instructions or whatever else for your repository? Are those consistent? Um do you like the thing that I've been trying to get more into very recently is like for .NET development we have what are called analyzers. So you can write custom rules. You use C to go write uh code that can analyze your C code to enforce rules. And so kind of meta, but it's pretty sweet. And that way if there's like literally patterns that will compile and be correct, but it's bad practice or not what you want in your codebase, you can literally write your own rules to go check for this kind of stuff. And there's even ways where you can have like uh code fix implementation. So if you're in your IDE, you can press a button that's like how to go fix it for you.

Super cool. Um but I'm trying to put more of these things in place because I'm finding that like I said in the last video, check that one out so I don't have to go, you know, rant for another 20 minutes here. um putting things in place so that the the agent stops doing dumb stuff and I don't have to like every single time I'm prompting it be like here's every possible example I want you to consider like we got to move past this at some point and I I don't think we're there quite yet but I'm hoping putting analyzers in place the AI can go write what is slop it will go to build it and then it's going to have all these rules that go nuh-uh like you're breaking all of these patterns. Don't do that. So, I think more code enforcement, the most simple version of this is llinters, right?

If you're using different languages and you're like, I don't even know what an analyzer is, just think about a llinter, but um maybe more powerful, but you know, linting and that kind of stuff is sort of like a minimum thing that you could and should turn on. But what other types of guard rails can you put in place? Right? This is again if we think about this concept like forget AI for a moment what other kinds of guard rails do we generally put in place I'll give you a really obvious example running tests right so it's like if you think to the most basic set of code it's like oh well we don't want to write tests on this and it's like well we should start writing tests and okay so now we're making testing a habit but you don't have you know continuous integration systems or you don't have systems that are running automation for your tests and you're just relying on developers to run the tests.

They're not going to run the test. They're going to forget to run the tests. We're human. Okay? So, what do you do? You make your automation when you put stuff up for review. Go run the tests. Right? Now, we're taking the human element out of it. You can't screw it up and not run the tests because the computer's going to do it for you. Right? So these are some types of guard rails that we put in place to help other humans not make mistakes. We put llinters in place so that we don't have to and like code formatterers so that we don't have to spend all of our time on reviews just arguing about syntax uh like stylistic things and whites space. And we still do this. We still do it. So, we're just exaggerating the types of problems that we have when we're using LLM single right software.

So, I guess one of the meta points I'm trying to get across here is like what it's like what do we do about this? It's like how would you address like literally just have this same conversation take AI out of it. How would you how would you go address it? So, if let's talk about the comments now from the other side, right? The comments are are garbage. They're AI slob. Okay, pretend AI wasn't writing those comments. What's making them slob? And I don't I'm not saying this to challenge the people that are saying cuz I believe that they're slob. What makes them slob? They're irrelevant or they're incorrect? Like, if it's an incorrect comment, you can have a human leave an incorrect comment. So, like, how would you go address that with a human? And it sounds like in this case from this Reddit thread, it's actually humans that are taking the AI comments and putting them into the pull request.

So, there's still a human doing it. So, what would you tell them? Forget that they're using AI to go generate the comment. What is it about the comment that makes it slopp? Is it just that you know it was written by AI? It's not even a bad comment. Maybe that's a you problem and whatever. You can figure that out. Deal with it. Um but it like is it because it's inaccurate? And again is is that a teaching moment? Is this someone trying to be helpful missing the mark? Because that's people are wrong sometimes and maybe people are being wrong more. It's being exaggerated because they have confidence from the AI tools. Right. So maybe there's just a lot of teaching moments and it's this challenge because people are feeling more confident that they can get some feedback like this. So again, my my advice here is like instead of just being like, "Oh, well, we got to get rid of the AI.

The AI is the bad thing." like no, I think it's probably helpful in many situations, but it's exaggerating different parts of our workflows and we're not used to that. It's like you'll see people on social media being like you know I it's especially like the people that are like they have not actually written software before and they're like I you know did 100,000 lines of code and like we don't need software engineers I did this like yes you are literal proof that we are just exaggerating the rate at which we create potential problems, right? Tell me there's no problems with that code. Well, it's working. Uhhuh. I also write code that I say works, too, until someone uses it and breaks some You just wrote a 100,000 lines in like a weekend and you're telling me, "Oh, it just works." Like, no, it no it doesn't 100%.

And you will find out there's problems the hard way. Oh, come on. Someone's got to move and it's going to be you, buddy. No. Okay, there go my back tires. That's okay. Got new ones ordered. They said these tires are pretty low. And I said, "That's weird. I haven't had them that long. And then they were checking the fronts and they're like, "These ones are great." And I said, "It's a rearwheel drive car." And they said, "Oh, yep. That's that's it. There's half the tread on the back. So, we'll have some freshies." Actually, they called and said they're in, so I got to get those on this week. Yeah. People should not be driving below the speed limit like that. That's not okay. Makes it pretty unsafe. I don't like driving around people like that. Okay, what else we got? I think, you know, overall summary here is like there's a lot of things that we can address.

Um, and I don't think that they're necessarily AI specific. If we start thinking about AI as someone uh like or or someone using AI just as a person that we can exaggerate the volume of code which they create we can exaggerate um and you know that also means that for x number of lines of code there's likelihood for you know y number of bugs like this kind of stuff. If we think about that and then remove the fact that it's AI involvement, like I think that's I understand that is the thing augmenting and that's what's going on. But if you take that out of the equation like the word AI and you just assume it's one person doing this, sorry, when I say one person, my my intention is not a single individual. I just mean that this is people on your team are able to do this on their own.

That's just how they operate. What would you tell them? You wouldn't say, "Hey man, like guess you got to stop writing code cuz you suck." You'd probably say, "Whoa, whoa, whoa." Like, "We got to break the down these pull requests. Have you not seen the other patterns that we have in the codebase? You're not following those, right? You're you're committing emojis into the code. That's not part of like our styling and our guidelines. We just don't do that." And then the next level beyond that is because you'll probably get tired of doing this because people have been doing this for years trying to guide people through coding conventions and stuff like that and pull request processes and how to operate on the team. You start putting in more systematic automation in place to guide people. You force them into the pit of success, right? make it so that as they're trying to go do whatever they're doing, they don't really have a choice but to do the right thing, right?

You can't push code into mainline without the test running. So, you can't just be covering your eyes and submitting and everything. You blow everything up. In some cases, you can bypass stuff, but let's, you know, the happy path, you're not doing that. There's uh like, oh, people aren't writing tests for their code. Cool. Do you have like this isn't the best way to do it necessarily, but a way to do it is like a um a like an analyzer that essentially looks for lines of code coverage. So, it runs the test and goes, "Hey, like your code coverage threshold has dropped below X." Cool. I know there's there's definitely issues with this kind of stuff. Um I'm just saying like there is, you know, one path forward you could go look at. So, what are things that you could start doing to to put gates in place, force people into the pit of success?

That's how I would start looking at this stuff. U and that's again, I'm not encountering that at work with poll requests, but what I'm using AI to build at home, it's not me yelling at anyone else going, "Oh, you put up this big poll request." It's like, "No, that was my bad. I told AI to go build, you know, way too much. It's crazy to navigate. Um, a lot of the time for features like that, I'm just exploring code and I have had to force myself to ask for smaller changes. I have had to start putting in more advanced guard rails. Um, this is what I would do if I had to invite a person on the team. If I had a I don't like comparing AI to juniors because I I just think that that's like a a weird a weird comparison and I don't think it's accurate because AI can do some really really advanced things that I would never expect a new software developer to do.

But also, it can do really dumb things that I feel like if I told a human once, regardless of their skill level, they'd be like, "Okay, I get that. I won't do it again." Yet AI is like, "Haha, I can't hear anything you're saying." And does it again a million times. So I don't I don't like that comparison. But I would say, you know, I think about AI as someone who is new to the team. Let's say that new to the team and overly confident. They start doing things the way they know how to do it. Seems to work from all their experiences. they go do stuff and you're like, "Ah, it's not really how we do it here." Right? You probably noticed this. I mean, maybe it's less of an issue for people that are more and more experienced because they'll uh learn to go kind of go through the code base and adapt to patterns they see.

But when you have people that are new to a team, generally speaking, their first set of uh code reviews that they they have people on, like there's a lot of comments, right? Like, and it's fair. They're just they're learning how to go navigate that codebase. And I think we have a similar thing going on here when we have AI. It's just that it's heavily exaggerated. So, just getting to the office here. That is my spiel on that. Um be curious to hear what you folks think, too. If you completely disagree, love to hear your take on it. Um but yeah, I mentioned in the other video that I recorded before this one. I'm going to do on my main dev leader channel a couple of uh more like polished edited videos and I'm going to walk through some of the stuff I've been doing with analyzers to see if Code Pilot starts doing a better job.

So, thanks for watching. I will see you in the next one. Questions, leave them in the comments or go to code.com. Submit anonymously. See you next time.