We're Pushing Code To Production Without Reviewing It

Transcript is auto-generated and may contain errors.

Hey folks, we're going to go to the experience dev subreddit. Now, this Reddit user was asking about the situation that I think more people are probably getting familiar with regarding uh code that's getting pushed out to production and it seems like people haven't really had the time to review it because there's this pressure that we are generating more and more AI code and you know who who's got time to review the you 10,000 line change from AI because you have 20 other pull requests of the same size and sort of this uh I don't know like vicious cycle around like we got to keep cranking out more and as we're cranking out more there's more pressure that keeps mounting for it and then inevitably there's just more code to review and you know you you don't have time to review the code because you got to be getting other done and the amount of code is so much greater like what's going on with this?

So, this person was saying they feel like they're going crazy because they're like, "Am I the only person who's like, you know, uncomfortable with pushing code to production that no one's like looked at?" And I thought this would be kind of interesting to talk through because I think that one, it's probably more and more relevant for more and more people, right? Um, I don't know, depending on when you watch this, maybe in 6 months or a year from the time I'm recording this, uh, I don't know, maybe it's not an issue. Um, I doubt it. I think it's probably going to get worse before it gets better. Um, but I I think it's interesting and this actually came up when I was talking with people at work as well where uh, you know, it's just another data point that that they're feeling this kind of thing.

And so what ends up happening, right, is you have people that are using AI to generate code and you know, you could you could push back on them in a review or in person or whatever and say like, hey, why did you know what what why did you write this part of the code or what's what's this part of the code for? Acknowledging, you know, AI probably wrote it. What's you know, what's the point of this? And not in a condescending way, but like genuinely like, you know, you're reading through the code, making sense of it, and going, "Well, what what about this part?" And then people people haven't even read the code, right? They're pushing pull requests up asking for you to sign off on them so they can get merged. And like they don't they probably, you know, in many cases haven't even read the code.

Let's let's for the sake of conversation assume they have read the code understanding it's a different question because you'll say you know what's this part for? Why why was this put in place and they they don't know the answer and so inevitably what do they do? Well they have to go back to their their agent session and say hey like what is this for? Um and it's like it's kind of ridiculous right? So you as a reviewer are asking a question about why something was done and the person who was responsible for it doesn't even know. So they have to go ask their agent just to get an answer to give back to you. And depending on that person's skill level, they might not even know understand like what the reason for. They're just kind of relaying it to you. They're like a man in the middle between you and their agent.

And so like where where does this lead us, right? Like it's kind of crazy. So this person like I said um said, "Am I the only person that is uncomfortable with pushing code into production where we don't like we just don't understand what the code is?" So I don't know. When I think about this, part of me is like, hey, I I think there's a world in the future where where we don't have to know. I don't think we're there yet. Uh maybe some people are working at places or working on different types of products where that is the case, which is super cool. Um person making a turn from the wrong lane. Very interesting. Um the the reality is like you know these are part of our software development process that we've done for for a long time right like not and not everyone

does this but generally speaking but forget AI for a second right people something writes code historically that's been people so code gets created and like before it goes live someone wants to understand it You want to make sure that what's about to get shipped, whether that's a live service or released as a a download or shipped on a floppy disc out to people, right? Like people want to make sure like, hey, is this code actually make sense? Like I should double check it. And so we've been doing this for a long time. It's been a pretty standard practice. And like is there a future where you know we we just don't need to be doing that? Like part like I said part of me wants to say yes. Um the same way that like I don't know there's like I I grab you know Nougat packages or libraries depending you know if you're not a .NET developer you don't use Nougat packages.

you get some package and like you didn't review all the source code for it, but you use it, right? You'll go run your integration test and your your CI system and you're like, "Cool, yeah, that works." Like, ship it. But like there there are parts there are parts of things we ship that we don't look at the details for, right? like I will run software on a VM and I don't I didn't review the code that's running in like that makes up the VM like there are just things that you know I don't think it's a crazy statement to say like people have not reviewed in depth all of these components that are running but we do we do give a about the software that we're shipping like we we were the ones making it like we we care about it and So, I do think there's a future where like the actual code that's being shipped where like I I don't need to care as much because I'm not writing the code, right?

Like I'm not writing the code. I'm not, you know, debugging the code by my like by hand, right? Like I'm using tools in this case like AI to be able to do a lot of that for me. And so it's like an uncomfortable thing to say out loud because I I think for many people and this includes me like I'm not there yet. There are some things now this is where I think it's kind of interesting for me in my own personal experience. There are some things that I have that I do treat like that. But the reason I treat them like that is because I don't give a if it breaks. Like I will push to production for my own service that I'm the only user. I'll push stuff to production without reading the code. I don't care if it breaks. I'll tell C-Pilot like, "Hey, it's broken.

Go fix it." Like go you go look at the logs, catch your own that you messed up, see why it's failing, fix it, you know, put some tests in place, maybe update your own instructions so you don't do a stupid thing like that again. and we move on. But it's because the the risk is so low if it breaks that like I just don't care. Like, okay, you're gonna I'm gonna get disrupted in my own, you know, silly side project for 10 minutes. Okay, boohoo. Who cares, right? Um, but like I don't know. Uh, there's stuff for Brand Ghost that I ship and like I have paying customers. I kind of give a about what I'm shipping, right? I I review the poll requests. I love being in a position where I open the poll request from an agent. I can just scan it and be like, "Yeah, checks out.

I feel good about this." Right? I see the test coverage uh for the lines that are changed is high. The test makes sense to me. The patterns like nothing stands out. Cool. It's a quick review. Great. But I review it. I care. And like the same thing at Microsoft, right? Like I'm on a team where we handle trillions of requests per day. That's with a T. Trillions of requests per day. And like we're not just going to go blast code out to, you know, hundreds of thousands of machines across the planet with no one looking at it because that's going to have some real big impact. If something goes wrong, who's going to have to look at it? I'm going to come back to this in a moment. Now, don't get me wrong. I would love to be in a position where we could confidently say you don't have to, right?

Like, you can go blast code out to all these machines across the planet and have confidence. I would love to say that cuz that would be such a cool world to live in to have that much confidence in a code change where you're like, I don't even know the code. uh and if it is broken whatever it will be instantly you know reverted you know the the risk is so low like I would love a world like that it's just that's not the world that's not that's not what the world is today let's get this lane change excellent um so like yeah we do have to review it yeah we do have practices in place that that sort of slow us down so that we can stay on top of this stuff and that's because software development is more than just you know ship PR

right it's a lot more than just was the code written and did it get deployed there's a lot more that happens and you know this especially if you run a live service or if you're someone who ships features and is also responsible for any bugs that come It's a lot more than just write the code. And so, like I said, part of me would love a future like this. I think we're moving in that direction. I think that's cool. Um, that doesn't mean that there's no value in understanding the code you ship, by the way. So, even in a future where you don't need to know, I think it's always helpful to understand things better. I I can't off the top of my head I can't think of a situation in life where like not understanding things better is the right move. I think it's always helpful in some capacity to understand things better.

So in a world where you don't need to know the code, I still think that it's helpful to know it and understand it. Um so coming back to this idea of like okay so you know code code code is being deployed and and people don't understand it who's responsible for it like inevitably cuz it's software even if AI is doing such an amazing job inevitably there will be a bug inevitably even if something's is designed perfectly. There's going to be something that goes wrong in a live service because, you know, something's outside of, you know, ex your control. Something happens. There's an issue. Who like who has to deal with that? And so when we think about this kind of thing, especially in in teams in companies where you know you have more junior developers, senior, all this kind of thing. If you have junior developers who like, and I'm not, by the way, I'm using this as an example.

I'm not saying I'm about to say like junior developers don't know what's going on. This is like a hypothetical kind of thing. assuming you have junior developers that may know less about, you know, the system, the code that's being deployed. Maybe they're leaning on AI even more, right? You've been at the company for a while, so you happen to know the code base a little bit better cuz you were writing it before AI was, you know, blasting in all these pull requests. You have more junior developers who maybe have leaned on AI a lot more. They don't know the code base as well. And so let's say they are responsible for one of these pull requests and there's a bug in it and so now your production service is being impacted like who's responsible right and so like you would you can make the argument

well okay well it's you know junior developers pull request like you know it's it's their bug someone might say well hey you signed off on it like you're responsible I mean reality is you're a team share the responsibility But what's going to happen if this is a, you know, super high impact, high severity situation, you know, people are on call or you're pulled on to like an incident bridge and it's like not it's not about blaming, right? Like so say the junior developer is like willing to jump onto this bridge call and they're like, "Oh my goodness, like it's my feature, my change, it's broken things." like they might be super eager, willing to help and get to the bottom of it, but let's think about this for a second because shit's on fire. You're the more senior developer. And so, inevitably, you're going to find yourself in situations more and more like this where like when things are really on fire, the people that are able to go debug more effectively, right?

No one knows what the code change actually is cuz no one understood the code or no one paid attention to it. Whoever is able to debug and get to the root cause faster, potentially someone with more experience in the system, maybe that happens to be you, like you're going to find yourself in these situations more where it's like you might be the person who's able to get to the bottom of it faster. And so it's kind of crazy to me that, you know, if you're if you're thinking about software development is like, I'm just going to have AI write the code and I'm just going to blast it in that like that works until there's a problem. And then you have a really big problem because like what do you do? Like how do you mitigate it? And there there are answers to this question, by the way.

I'm pausing because it's like I want you to think about it, but like what do you what do you do to mitigate it? And so you could say, well, just revert it. Okay, but like uh unfortunately things aren't always that simple. That might be the right thing to do for your code base. It doesn't necessarily mean that the problem in production is mitigated instantly. by the way. Um, even roll back to last known good. Sure. Okay. But now that's on the server side. Clients are now put into a weird state where you've broken them. What are you going to do about them? Right? So, like it's not always that trivial. Even if, you know, rolling back is a good strategy. Um, not always going to fix things or mitigate. There's nothing in front of me. Stop beeping. Um, so what's my train of thought here? Um, when when this kind of thing is happening, like I was saying, there's there's several different answers to this.

One of them is also, well, I used AI to write it. I don't know the code, so like I should have AI fix it. Right? Let me ask AI how to mitigate it, how to fix it. And you can like that's I'm not saying that that won't work or that may not be an effective strategy, but like realistically like how good are you at doing that? Because if you're really I don't I don't use the f- word on this channel. If you're really flipping good at at using AI to do that kind of thing, then great. like you might be someone who can lean into this more and you can debug and mitigate things faster, especially in production services.

And that might be all fine and well, but like if that's not something you're doing often, odds are like you're going to feel like a lot of pressure when shit's on fire and no one understands the code and someone's got to start debugging it and someone's got to root cause it, mitigate, blah blah blah. So, I I think at least in the foreseeable future, it's in many people's best interest to like genuinely understand um what code you're shipping, right? And so, I wanted to spend the other part of this conversation talking about like, well, why is this come up in the first place? Because you might have been listening to what I'm saying and you're like, yeah, no Nick. like it makes sense to for people to review the code and understand it. You should understand what you're reviewing. You should understand if you're putting up the pull request what your code is.

I think we can all mostly agree on that. Um and so well how do we get into this situation where like people don't and I think a lot of it's like this um I don't know like things are a little a skew with um you know pressure to get things delivered right hey we're a company we use AI and the expectation is well if we're using AI productivity better be up and so how do we measure productivity? I will also pause till you think about this for a moment. How do you measure productivity of software engineering teams? What is what is the answer? And I'd be curious because if you have one, um I have been doing this for several years now. Um I don't think I have ever seen a good quantitative measure of productivity in software engineers that make sense uh across the board.

Personally, and I've seen all sorts of metrics and things that you could consider. Um they might uh correlate, they might not. Um, but I don't think any of them is a good proxy for like actual productivity, right? So, let's use the age-old lines of code, right? I have shipped X lines of code and now that I use AI, I ship 10x lines of code. Therefore, I am 10x more productive. Doesn't work like that. um you know lines of code is a very it's not like a surprise to anyone right it's a metric that anyone can gain and so if you're measuring productivity by lines of code like why wouldn't you just you know write the most verbose code why wouldn't you make a bot that just commits code that is just a bunch of comments in a file right why not right your lines of

code is going up is doesn't that mean you're more productive right it's ridiculous um you got you know people have said like number of bugs caught okay so like in a imagine a perfect world where you never even create bugs are you telling me that by not creating bugs therefore not catching any means that your productivity is not good and then we talk about escape aped bugs. Okay. Well, don't go looking for, you know, don't go looking for bugs. Don't let people send in support tickets. Ignore them. Look, nothing's escaped. These people are going very fast. I thought that was a cop behind me and it's not. So, um, what's another one? Uh, you know, I think one that I've seen a lot coming up now is like poll requests. Oh, this team is doing so many more poll requests. Okay. Does that mean are they doing pull requests because they're fixing other that AI broke and they have to do more pull requests?

Like what what does it mean productivitywise to do more poll requests? And I think that a lot of us have fallen back into this not our not our fault, right? Like I think because of a lot of pressure from the organizations we're in that if you're doing more pull requests that is more productivity and so well how do you get more poll requests done right you well you got to review them too and the pull requests are bigger because AI is writing all this crazy code because we all know that AI does not write the most concise code uh out of the box. So, you know, more pull requests, more code per pull request means like a lot more time for you to review them, but you don't got time because you got to be doing more pull requests in the first place.

So like we end up getting into this like I said earlier this vicious cycle of like more more more less less less time on the review side and then unfortunately understand it less and get into this cycle. So I think that's kind of where it comes from. Um, I would like like I said, I would caution people from kind of falling into this trap, but I I also understand that there's probably so much pressure coming from all angles for doing more, right? You're the engineer, company gave you AI, expectation is way more productivity. What does it look like? Right? So yeah, I don't I'm not saying I know the answer um to this, but I I think that's sort of where this is coming from. So Mrs.

Redditor was asking, "No, like I don't I don't think you're crazy for for wanting to know the code before it shipped." I think that's important, but I also do imagine this world where that we're I believe we're moving towards like you Before it was way more critical that you knew exactly what was being shipped. I think it's becoming less critical that you know exactly what is being shipped. I'm not saying there's no benefit to it. I'm saying it's less critical. And so I'm I challenge myself a lot to like not not in my own uh work at Microsoft especially cuz I like to do a bunch of side projects. How can I build more of that trust? How can I be building things with AI where I can be that confident and then understanding like what is it that gives me that confidence, right? Is it because I've set up really good instructions with the agent so that I know it's following, you know, my best practices, coding conventions that I like.

Right. On top of that, um is there good good um visibility into like test coverage, the rigor around testing that I know that it would be really difficult for it to ship a bug to production. And then am I willing to accept that like you know maybe the if we were to classify the bugs in production the bugs end up being like you know small things around some functionality is a little bit wonky versus like catastrophic failures like losing data people's like passwords and security issues like you know leaking passwords credit card information security stuff um just complete availability issues. Like if you can ensure that that is all minimized like maybe maybe you have more confidence knowing less about the specific code that's being shipped maybe.

So, like I said, in my own development, when you hear me talking about my side projects and stuff, um, I'm trying to lean into that and I again will try to clarify, not because I think that I'm doing something magical that I can just safely trust it and I do it perfectly, like absolutely not. It means that I'm accepting. I'm probably going to have a rocky road ahead of me. there's going to be some storms and I accept it. And so I want to use those those painful situations to like to learn like h how do I work around this? How do I prevent these types of things with agents? How do I catch these types of things? How do we, you know, mitigate them quickly so that I can take those learnings and apply them to like more of my professional career? So, those are some of my thoughts on that.

I've learned that this camera overheats, so I'm going to go stop this video. I have a cooler for my camera at home. And so, I'm going to be using my camera cooler in the back of this thing cuz I guess you can't make a camera that records for half an hour without overheating. Sony. Anyway, thanks for watching. Hope that was helpful or interesting. And if you have uh questions you want answered, leave them below in the comments or go to codecommute.com. You can submit stuff anonymously that way and I'm happy to make a video to answer your question and hopefully help you and some other folks. So, thanks and take care. I'm going to turn this thing off.