How Do We Maintain A High Quality Bar With AI Development?

Transcript is auto-generated and may contain errors.

Hey folks, we're going to go to the Experience Dev subreddit. And this one is from a person who wrote up a pretty lengthy post on trying to maintain quality um with uh like I guess agentic engineering, right? And obviously around this time everything is like super overloaded with terms, but they they wrote like a lot of uh I don't know like detail. So I think they were talking about it in the general sense, not like some hypersp specific methodology or like loop engineering or whatever. They just mean like people are leaning into using agents for writing code and um you know like what are what are people doing to try and and keep standards for for better quality software this way. And I thought it was interesting because they were talking about some things um that I I think that I would be recommending at least as as patterns.

Um and it seems like uh they're trying this kind of stuff out, but perhaps it's not um I don't know, maybe it needs more time, maybe it's a different way of thinking, maybe it's, you know, all sorts of different things. But um they were going on to say that a lot of the stuff they're trying seems to work reasonably well initially. So for example, they're having um you know giving an agent a task for what to go build. They're able to get a demo working pretty quickly, right? I think a lot of us have probably experienced this where you can almost like oneshot a bunch of stuff that you know without the craziest prompt in the world. Now you can get something that's operational whether or not it's good or like production ready completely different story. But at least if you just wanted to get a proof of concept together, like you can a lot of the time like make a lot of pretty significant progress on just getting something that demonstrates that.

So they were saying, "Hey, like that's not really a problem. Like that seems to work pretty well." Um, and then it seems to be when things start scaling up a little bit more. And they're not necessarily like blaming AI. They're not saying like, "Oh, it's because the agents are terrible at writing code and it's all this spaghetti and blah blah blah." They were actually saying, in fact, like when they are reading through the code and stuff, they're like, "It does seem to, you know, look reasonable, like it's not glaringly uh bad." So that's all positive, but they are saying like, you know, as time goes on and they're building software like this, they're they can't help but like realize or or feel that there's like a disconnect between them and the software. So it's almost like a them problem now, right?

um if something's not right or they want to improve things, it's almost like because they didn't write the code, they're not as familiar with the code, they feel less equipped to be able to like, you know, direct the agent on like things to go improve. And I thought this was super interesting because like I I think I can I can get why that's the case. um you know it's like if I you know depending who's watching this like odds are you haven't read some of the code that I've written right so if you were to use some of my software and you were like well let's say something for performance you were like hey Nick I know you wrote this and like um I find that maybe it's slow or there's latency and like we should optimize that and you know I would say sure that's a that's is a great point.

And then without you knowing the code, it's it might be pretty difficult for you to like give me some targeted things that I should go optimize, right? Like you would need to go read my code, understand it, and then go, "Oh, like Nick, I see you're doing this over here and that over there. Uh if we instead tried to take this approach, like we could probably cut down on latency, or you'd have to do less wasteful work." So uh overall you know the throughput would be better but when you're not familiar with the code it's pretty hard to do that right like we are becoming more um detached from the code we're writing I think that's a how how things are working right now when you're delegating work like that I think it's like a natural consequence right that you become more detached it's literally the same thing if I like talk about software that my team builds as an engineering manager, right?

Um it's a it's a very similar thing like not only do I not know the specific lines of code, um I'm I'm just significantly more detached from what the code base actually looks like, at least for how I'm operating currently. Right? in my previous life when I was managing teams and writing code at the same time different story right but that transition going from like I'm writing code on a daily basis to spending more time managing engineers writing less code that they're working on at least that to me that's a very similar pattern right where like I I quite literally know less of the code base because my time is not spent doing that. So what I'm getting at is like I think when I when I look at agents writing code, I see a lot of parallels to you know how how an individual approaches building software.

And so, and this comes up all the time, like I've talked about this in other videos, like a lot of the things that we're doing with agents to try and optimize their behavior, um, how they work on things. If you, especially if you're like orchestrating agents, like I see a lot of parallels between agents and like just how we as humans operate in software development teams. The difference is that like you remove um sort of like the interpersonal human challenges and there's no there's no downtime unless your agents are, you know, stuck on something or you run out of tokens or something. So with uh with these parallels that I I kind of see like with engineering management and then having agents work on things purely from the perspective of code, right? Like there's there's a lot of different things that I can still help my team with without knowing the specifics about the code.

And then there's situations where like if they're stuck on things or they need help like more, you know, more specifically then absolutely like okay like time to roll up the sleeves. Let's go review code together. Let's talk through it. Let's see like what types of things that we can do with it. But then I I do have to dive deeper on it, right? And it would be a similar thing if you had an agent writing code and you're not familiar with it and you're like, well, something's clearly not right here without even seeing the code, right? Something performance related, behavior related, you you're something's not right. Um, if you're trying to go back and forth with the agent and it's like not making progress on things, it's probably time that you instead of just saying like make it better, make no mistakes, it's probably time to dive in and understand what's going on.

Now, if you think about that kind of exercise, right? Like, okay, got to roll up our sleeves. We're going to dive into it. Um, developer who wrote the code probably knows more detail, right? Um, in the case of an agent, agent has maybe more context for what it was doing. Um, or it's probably faster for it to go get context. If you're like, just go like study this part of the code, tell me how it works. It can probably do that faster than you can to summarize it and give you an explanation. Right? I'm generalizing. I'm not saying all the time. It's a generalization. So, there's probably things that we could and should be doing if we think about this type of exercise. Like, how can we leave better like breadcrumbs for ourselves, right? How can we make sure that code is easier to navigate in situations like this, right?

I think the more like unfortunately it seems like it's not totally consistent, but the more consistent code is, you know, the easier it is um when you have like fewer sporadic design patterns just like littered across everywhere. Um things become more cohesive um patterns are more obvious. So like these are good software engineering practices. Um if you're noticing that like okay now that we've gone through this investigation and we see like oh this is slow because you know we're doing um instead of doing like a bulk insert into a database we're doing like a loop and doing multiple uh inserts doing multiple round trips right to the database. I'm just making something up. Um, you might say, "Cool.

Okay, well, we know how to fix that now, right?" And so, if you were talking with a human about this, so let's pretend I'm an engineering manager and I'm talking with someone on my team and um, now that we've gotten to the bottom of this, say they're more junior and they're not totally familiar with like why that's a problem, right? The code looks like that does the right thing for each item. Go insert it. Um, you might say, "Okay, well now if I explain this to you and why this is a performance sort of challenge here, you want to maybe group these things together, do fewer round trips, kind of do it all at once. You could do it in a transaction as well. Um, you know, so you maybe you want this kind of all or nothing behavior, but you could explain these things to this person and they go, "Oh, okay.

Like that that makes sense." And maybe it takes a couple of, you know, times to to help that sink in. And then going forward, they probably would think about this type of thing next time they're doing an insert or maybe next time they're investigating something that's a performance problem on the right path to the database. They're going, hm, like I've seen patterns like this before where we're, you know, doing a loop and doing many round trips. Maybe I could look for a pattern like that. Right now, with an agent, there's ways that we can do that, right? Like we could be making instructions better. So, okay, you notice that agents happen to have a problem writing um code like that. Okay? like maybe you should put into your instruction files like when you're um working on like a repository pattern or working with a database. Um however you want to try indicating that to your agent depending on the tools you're using.

Maybe you have instructions that that really guide agents to like avoid patterns like that and to instead follow other patterns that you'd recommend. And I have to think too that historically we're relying a lot on code reviewer reviews as humans. And so you might have the more experienced engineers on the team. Even if you tell the more junior people or the newer people to working in stuff like this, hey, that's not how you write um you know, insert logic in our codebase. Like here's why. um you still on code reviews might have to do a little bit more policing of that, right? You're looking for those types of patterns to make sure they don't show up. But like if you think about it, like forget AI for a second, right? If you were to tell people on your team these types of patterns, it would be great if once you tell them, you don't have to worry about looking for it in a code review.

But odds are because we're human, you're like, I still got to check for that. Like I know I told Billy on the team not to do it, but like I don't know if Billy fully got it. Like let's make sure. Or like maybe maybe I haven't had that conversation with Sally and I'm reviewing her code. So like I'm going to have to go look for that now, too. Those are things that ideally like once people know they don't you don't have to go police it, right? Ideally. And even better, if they didn't know about it, how cool would it be if like your code prevented them from even writing the pattern in the first place? So, I think that there's a lot of opportunities to put guard rails in place that prevent patterns that you don't want to have happen. Now, this is easier said than done because depending on what the behavior is, the pattern of the practice, writing something that can check for that is tricky, right?

So um as a .NET developer I work with things called Roslin analyzers. So when you write the code if you go to compile it you can do static analysis to figure out like uh does the code follow and meet such and such pattern? If so then I need to you know mark it as an issue. So it's not a test. It's not compile all the code go run the test suite. it's I'm trying to compile the code and I should not even compile it because there is an issue based on this pattern. So that's a really nice early signal that you know can stop bad patterns from getting in. Um some things are not well suited for a Rosland analyzer because maybe the the tech itself doesn't actually facilitate that. Roslin's super powerful but or maybe you don't even work with something that has a Rosland equivalent.

So you can do tests, right? you can say like uh maybe something more like an architectural test or a configuration test where you're looking for patterns in your codebase and then that way sure like it's not at compile time but you know running a test in your CI system to go and check for bad patterns that's still pretty early right if you don't have the same ability to run that stuff locally then maybe it's a bit of a pain in the ass cuz someone builds all this stuff they go uh you know get their pull request ready and then starts breaking like ah it's kind of crappy but it's better than it leaking in and then someone going like what the hell or now it's up on the poll request marked ready for review and then people are spending time looking for those patterns.

So, all of that stuff I just talked about has nothing to do with AI, but I think it's especially powerful that when you're working with agents and you're trying to keep the quality up that you can start putting patterns in. So to tie it back to what this person was saying in their Reddit post, they were talking about this sort of disconnect from the code and like um the obviously like I'm not an expert in this area and I kind of feel like anyone that says they are it's like it almost feels silly because it's so new and changing so fast. Like I don't know how you claim you're an expert if this stuff hasn't been like you know actively been used for that long. But anyway, uh unless you're like literally building this kind of technology.

Um, so not trying to sound or pretend to be an expert, but the from my experience because it feels a lot like my role as an engineering manager, it's like if there are patterns and practices that I think are ills suited for developers because of my experiences, then I like to codify those. I like to make sure that there's instructions to help steer agents to the better patterns that I think are better for whatever reasons, steer them away from antiatterns because I I believe that they're anti patterns from my experience. So that's the one thing upfront to do some direction. It's the same thing if I were talking with people on my team and they wanted my opinion on how some stuff is built. I would say like here's patterns I like to follow. here's ones that I try to avoid and here's why. A similar thing.

And then the other side of it is like, okay, well, if these things are that serious or we want to spend less time, you know, looking for things like this in a review, well, what things can we put in place to just prevent them, right? And that and again, like that latter part has nothing to do with AI specifically. It's just that I think it lends itself well to AI because with it generating so much code, how do you keep it on track building patterns, building software in a direction that is something that is agreeable with your your standards? So my point is that I feel that for a lot of the stuff I've been building, I have to know less about the very specifics of the code because if I codify some of these patterns that I'm looking for, then I can make sure that like the agent has to follow them.

either it follows them reasonably well up front, so it's more efficient, or it's doing dumb up front, and I should improve the instructions, and then I have some things after the fact that catch it. And that's not perfect. Just like humans working in a software engineering team aren't perfect, right? You might have a code base that's growing in size and there's, you know, 10 people on the team and historically there's been 30 people that have worked on the codebase and you have a couple of patterns that are pretty common and then over time a few more bleed in, right? And like it's because different people were on the reviews and they were like, "Yeah, this works. Like it looks fine to me even though it's not, you know, the perfect pattern that's used in the codebase." And so like stuff, you know, stuff over time kind of uh I'm gonna use the word erodess in terms of like perfect quality, perfect standards.

It deviates, right? And like we keep together, right? We hold on for the most part. Um but th those are like, you know, things that we can start putting in place to try getting back on track. So my belief is that over time and I kind of talked about this one of the last videos, right? One of the last videos I was saying like uh over long time horizons I think that you know no having to know the programming language specifically is less important more about like you know picking programming languages for the um the characteristics of the language and the tech stack so that you're getting software built that lends itself well to those characteristics less about like do I know this language specifically and I an expert in this language and I think The same thing with code bases because we probably spend less and less time on this time horizon looking hypersp specifically at lines of code.

I'm not saying that that goes away entirely. I'm not saying there's no value in understanding it. Those aren't the words I'm saying. What I am saying is that like I think it's less critical that you know every single line of code. And I say that because I think that there's other things that we can do. Uh I don't have the right word for it. Like I want to call it like a harness but that's too like overloaded with you know like agentic harnesses. But I mean like guard rails. I suppose the more more sort of guard rails we put in place around how agents write code, the more that we can feel confident about it building up to the standards we want. Um, a final thought just it's kind of to tie it all together like um maybe less about code specifically and more about just like the ability to to debug your code um understand what's going on like I think it becomes even more uh helpful to have like logging and telemetry.

So historically, and obviously it's generalization, but I would say historically if you were writing code and you were pretty intimate with the code you were writing, if started behaving in a way that was a little bit weird, at least you as the person who wrote the code could sit there like thinking through the code, like reading through the code, rationalizing things, and you'd have an understanding of like, oh man, like if I see this pattern, it's because it's going down this code path. Here's why. Here's here's what's happening behind the scenes like you understand that probably uh more quickly because you have familiarity with the code and if you have that less then I think that it's probably a lot more important that you have other signals that come out of your system that can help paint that picture or tell that story so that you do understand what's going on even without having intimate knowledge of the code.

Right. Um, I think about this with things like with Brand Ghost. I'm like, if something's not working, I want to make sure I have all sorts of telemetry logging that I can look at for anomalies, inconsistencies that tell me the story of how things are executing. When I think about this uh in way larger scale systems like with what I do at Microsoft, like that's critical. There's there's so many machines that are running in the data center, there are so many instances of our services and there are even more instances of all of the other services that we route traffic to. So like if there's something anomalous, something's not working, it is so important to be able to have signals to tell us what's going on, right? And again, this has nothing to do with agents. That's just a helpful thing.

My point is that if you understand less about the very specific lines of code that are going in because I feel like this is a natural direction that we're heading in, then I think it's even more critical that you have this type of information to be able to access. So those are my thoughts. Um I thought this was kind of a cool topic from Reddit, so wanted to chat through it. But if you got questions about software engineering, career development, that kind of stuff, you can leave them below in the comments. Otherwise, you can go to codecommute.com, submit stuff anonymously that way, and I'll make a video for you and everyone else on this channel. All right, take care. Thanks for watching.