The Effects of AI on the Software Development Lifecycle

Transcript is auto-generated and may contain errors.

Hey folks, we're going to talk about a couple of common themes uh from the experience dev subreddit and uh conversations I've been having whether it's at work or I guess even outside of work but related to AI and uh specifically I guess like different parts of the software development life cycle where uh where it's being used and sort of the some of the side effects around volume. So in particular, one of the one of the pain points that's coming up for folks especially like I I mean a lot of people work in teams uh in in the workplace. So when you're on teams and having a lot of code being generated now by AI, the the tax that there is on reviewing all of this code.

So, a lot of people saying like, "Hey, as more and more people are are using AI to produce code, what like what is your strategy for for being able to keep up with it, right?" Because now I think there's a handful of different dimensions here, but like if you think about maybe the volume of poll requests that go up that need that need folks to review them. um it's likely higher, right? So, you have a you more pull requests that go up. The amount of code usually I don't again I don't have like stats on this. I wonder if people do. We've measured it, but like um the amount of code that AI will write for a given feature or a bug fix is often it's often overengineered or like it's like a lot of the times we're as humans or like a as we're adding features or bug fixes like it's not just code editions.

like a lot of the time there's parts of it that are ripped out and it seems like more often than not AI is like adding lots of code uh very seems very rarely that it's removing. So you have that and then you have this third layer which is like um I don't even know the right way to say but like this uh the fact that someone else it's not even the uh sort of like the author who's putting up the poll request someone else wrote it. So there's like this this extra level of disconnection from the code that's going up. So, I mean, there's there's lots of different parts of the software development life cycle, but this one in particular, uh, I've just listed like three different things that I think um, sort of make more work during code reviews and pull requests and um, from a couple of different angles like one of which is purely volume.

So, how are people addressing this, right? Like what's the strategy here? Um, and I guess like one one way to one way to look at this is toss more AI at it, right? Just have AI review the AI work. Um, which may actually be part of it, but it's it's an interesting problem, right? And I don't know like when I wanted to talk about this, my goal wasn't to say like here's the solution for these things. Um, but just to get us kind of thinking about this part of the software development life cycle and other parts as well because when I was talking about this on Monday night's live stream for context because these videos go up at different times, right? It's uh it's Thursday now, so I've had a little bit of time to think about this more, but when I was streaming on Monday night, we were talking about this a little bit.

And if you're interested in the streams, they're on uh the Dev Leader podcast on YouTube. Um and they're like on LinkedIn and Twitter and stuff like I stream them to to all those platforms. But part of that conversation we we shifted over to this topic and one of the things I was saying was along the lines of like if we think about the software development life cycle um I was framing it like a pipeline and it's a obviously it's a gross oversimplification of like of what actually happens right it's uh it's a little bit more complicated than just saying like oh it's a pipeline and then like trying to draw it out and it's it's nice and smooth and simple. Um this we're we're humans. This is real life. There's a lot going on.

So in this oversimplified model I was saying if you think about it like a pipeline and you think about you know in this pipeline you have uh things that feed into it whether those are uh bug reports from customers if that's uh direct customer requests if that's uh engineers that are you know wanting tech debt to get uh addressed if that's um you know product owners working with customers to file things in the backlog like whatever it right? You have things that feed into this pipeline and then uh throughout like the rest of the the body of this pipeline is essentially like uh most of the development life cycle where you're taking these asks transforming them into I mean there's a triaging part right you're taking the as transforming them into uh into deliverables into code uh the code has to get reviewed uh there's then like things like you know uh build systems automation whatever else releasing um and then there's like feedback loops and stuff as well.

But if we think about just this pipeline of things flowing through to have call it like productivity right like what is our throughput in terms of what we're taking through this pipeline and outputting um I I think that when we try to apply AI and we look at the different spots in this pipeline my sort of a what do you call it like a thesis I don't really know um my claim is I that how we're applying AI is not uniform across this pipeline in terms of productivity. And so because of that, if you were to, it's obviously again gross oversimplification, but if you assumed like today you have this software development uh life cycle pipeline, assume today it's operating at you know uh 100% efficiency, whatever that looks like.

If you start applying AI to uh to enhance different parts of this pipeline, what ends up happening is that you you have more throughput at different parts, but you also have uh because it's not uniform, you end up having more like choke points, right? So, for example, if you have this pipeline and I said, "Cool. The only thing we're going to do is at the beginning of the pipeline. This person slamming on their brakes before getting on the highway. That's some dumb stuff if I've ever seen it. Um, god. If you only go to the beginning of the pipeline and you apply AI and you're able to get a higher volume of of things feeding into the pipeline, you're actually the net effect is like you're not actually getting more things done, right? Your productivity, your overall throughput, the volume of you know the volume of value being delivered through the pipeline is not greater.

you've increased the amount that you're trying to feed into it, but if this is already at capacity and at 100% efficiency, you're not adding more, right? You can you're trying to queue up more things, but basically it's queuing. It's not actually getting through the pipeline. Again, I realize I'm oversimplifying things. I said it's already operating at 100% efficiency. This is obviously not real. I'm just using it to illustrate my point. So you know not uniformly applying this optimization of like more throughout the pipeline means you queue up at the beginning. Okay, let's forget that example and go to some other part, right? So assume the amount of work uh that is being uh added to the pipeline is consistent, but we're going to use AI to to churn through more of those requests, right? We're going to as developers take those requirements and transform them into code deliverables.

Okay, if we only apply AI there and scale that up, what happens is we might take more things in from the the mouth of this pipeline, right? We're able to pull through more, but what's going to happen is that what's the next step in the pipeline? And kind of like the beginning of this talk, that next step is going to have humans reviewing it and that's not scaling due to AI. Then again, you're you're going to start backlogging right at that point, right? Cool. We can crank through more uh things coming in the pipeline, but now we're getting uh choked up here because we can't review the the volume that's coming through. Um, and if you keep playing around with this model or you think about it like a pipeline, if you're not applying AI scalability uniformly to scale this uh these different parts of the pipeline, you're going to start noticing like either what feels like an inefficiency or feels like you're backlogging things.

Uh both I mean both are kind of challenging in the sense that the overall throughput of the pipeline's not being um I don't know like felt by by some of these enhancements. uh if you're talked about this in a video probably even a couple months back and it's not the first or only time it's come up but this idea of like hey we're applying AI at work like where where is the productivity right like you know companies taking the uh these big bets on be going like everything AI like well we were promised 10x improvement where is the 10x like developers you have to go find this 10x improvement now like It doesn't work that way because you need to basically enhance all parts of this pipeline. So, I'm not I'm not here to say like it's impossible or won't work or it doesn't work. I'm just saying like what I'm noticing is that when we have these conversations like, "Hey, you know, we have all these uh these poll requests piling up now.

How are we going to review them all?" I'm like, yeah. To me, this is a sign that the uh like how much different parts of the pipeline are scaling thanks to AI is not is just not uniform. That's why we're feeling different parts where it's backlogging. Um I mean you know I was I was hinting at it not saying it's the solution but you know one such solution is like okay so why don't we apply AI to the next step of the pipeline like why don't we say cool AI to help generate code AI to help review uh the the funny example is obviously AI writes the code entirely AI reviews it entirely like what what what's actually happening like we don't know it's a blackbox now. Um, I mean, I don't I don't think that's the answer.

I think that's part of the answer is using AI to help with it, but again, it's uh it's not that trivial because if you've seen AI code reviews, uh, it's it creates a lot of volume, right? Like, you have to triage the comments that it's putting like you would if a human was, right? Right? If if you have colleagues that leave tons of comments on a poll request, it it doesn't mean that they're dumb because they put a comment that's not accurate, like they're trying to the goal is to try and be helpful, right? Like, hey, I have a question about this or I think this pattern might not be right or can you clarify this, whatever. Um, you might read some of those comments and you're like, "Yep, that's very helpful." You might read others and you're going like, "No, that's not relevant here." Or whatever else.

But you have this. This guy's flying. Holy crap. Good thing I checked before switching lanes. See you later, pal. Um, it's not it's not wrong or bad that there's, you know, lots of comments. It's just that it's more it's more work. It's more work to go through. So, we have this same challenge with AI. So, I don't think it's just add AI to the reviews. I think that can be helpful. I think hopefully over time it becomes better at leaving more meaningful comments just like you would hope that people that are ramped up on a team get better at leaving more applicable comments. Um but like there there's got to be more to it, right? I think that that part of the pipeline needs to be improved. Um when I talk about fan out problems, this is one of the things I was talking about recently.

Like if you think about uh on a team where people are doing code reviews, if you're you might have team policies like this or maybe just your practice is whatever uh you know not just having one person sign off on a pull request or a code review. Maybe the goal is to have a couple, right? Or maybe one person signs off, but you're always sending it to a couple or more people to review. If everyone on the team does this, it it's literally by definition a problem that fans out. And so, um, when you have something like this, my my thought is that like in order to reduce the overhead that's now multiplied by the number of people it goes to, like you need to do more upfront to reduce that overhead. So, if you're using AI to generate code and there's a lot of volume of it, I think there's even more on you to be reviewing it up front, right?

How can you do a better job than just dumping reviews on people where there's like a 100 files changed and you're like, "Yeah, just the don't worry, it was AI." Like, it's just how AI is. Like people still have to read that. So, how like can you better direct people to things? Um, you know, can you spend more time upfront doing a review on code that like honestly you didn't write, so you probably should be reviewing it. uh like what does that look like? How can like is that a a factor in trying to make this less of a a pain in the ass and a scalability issue? I don't know the answer to this. I'm just saying like I I think we have to get better at this.

Um, one of the pain points if we think about this pipeline kind of uh model again, one of the pain points I've noticed is people loading tons of stuff up to the front of this pipeline and um so like the examples that I give are like if you use AI to go read things and say hey AI given a set of data whether that's code whether that's um uh like like when I say code I mean like actual logic whether that's like you know uh version uh like package versions whether that's um you know reading through code to look for uh what could be a security vulnerability what like whatever it is um look through look through the production logs to like I actually do this for brand ghost like look through production logs and and uh create issues when you build AI kind of tooling up front like this.

The the reality is that kind of like with writing code, it can produce a lot, right? So it might look through a set of data and go like, okay, like I'm going to go open up, you know, a 100 work items. And so the the problem with doing this kind of thing is like it doesn't actually help deliver more value through the pipeline. You create more things to feed into it, but like that that itself is not more value. And so I've been noticing more and more things especially like at work obviously like more things coming up like this where like hey we turned on AI to do X and now now I'm getting bombarded by tons of like things where I have to go triage them and guess what part of the pipeline I'm noticing is not having a good uh a good net benefit from AI.

It's like triaging and prioritizing. I'm not saying that nothing exists for that, by the way. I'm saying I don't notice um the same scalability benefit is triaging and prioritizing um at least to the point where it's keeping up with the amount of stuff that's getting created like the work to do. So um it's nuts. It's like it's just it's so interest like when I think about this pipeline kind of model uh to me it's so interesting that you know I think AI is super powerful. I think it's super helpful, but it's really interesting to me when we start applying it at different uh I don't know different uh what's the word I want different amplitudes at these different stages the the net effect that we realize you know doesn't translate directly.

Now the cool part on top of that is like obviously my model is overly simplified um for many reasons but one of which I was saying is like hey if you assume it's at 100% efficiency it's not right it's obviously not like we're not no one no team nothing is operating at 100% efficiency so um as I pull up to crossfit here one of the things I encourage you to think about is like if you think about where your team like needs more throughput, more efficiency, whatever it happens to be in terms of this pipeline model. Man, there's no parking spots. Um, like that might be a better spot to try focusing on with AI before trying to jam AI into parts of your workflow that you're like, "Cool, if we have more throughput here, it's not going to get the benefit, right?" like for me now I'm going to be a huge bottleneck if I can't prioritize effectively and that's a problem.

So I wouldn't say like cool if I could make developers uh I don't know write 10x more code. I'm like I haven't solved the code view problem which is which is going to be a problem and I haven't it's my wife. Um and then the triaging part there I need to fix the or need improvements for the triaging part to make that more efficient. So these are things that you should think about too for your teams. But thanks for watching. See you in the next one.