Is AI Actually Making Your Teams More Productive?

Transcript is auto-generated and may contain errors.

Hey folks, I'm just leaving the office here and I figured wanted to talk about some AI stuff, but not not the how I'm using co-pilot and claude and stuff, but maybe like a different kind of angle. And this is something that's come up, I guess, a little bit more recently, like I mean, it's not new necessarily, but I think I'm seeing more conversations around this recently. and that's like around uh efficiency, right? So, I think what I'm going to talk through or some thoughts here is not uh unique to Oh, there's a previous employee. He's looking down. Damn it. Um I haven't seen him in a couple weeks, but that's the first time. Um so, yeah, I wanted to talk about efficiency and usage and expectations and things like that. And it's not a new thing. And I'm sure many of you have been have been talking about this, thinking about this, hearing about this.

And uh I think for some reason it's a little bit more like recent for me. And uh I was actually trying to think about this like why why does that seem like it's a more recent thing? And I think I got to get sunglasses. I think part of it is like at home the efficiency is very much felt and I think at work it hasn't been so much of a question about like how are we measuring efficiency more just like hey how do we make sure that that folks are are finding you know effective ways to use it making sure that they're not getting blocked like you know it's like new new technology and how to make sure people feel effective, right? So, I I haven't been having conversations in sort of like any real capacity around productivity boost and stuff like all of it is is really around like social media and and those types of interactions, but I just feel like I'm seeing more of it.

And so, I figured it'd be interesting to talk through. Um, I want to have the disclaimer too that like obviously it's difficult to talk about things and I don't know depending on if you know me from like work or whatever else like this is not a reflection of like you know specific conversations. It's more just like this is my uh my observations and a lot of it like most of my influence on this kind of stuff is either my own personal experience outside of work uh and honestly like my Google news feed uh YouTube uh and and LinkedIn right um so with that said uh productivity right how do we how do we think about productivity because this is I feel like a problem or a challenge. Challenge is a better word. It's been a challenge in software engineering for a long time where like how do you measure how productive a team is?

And there's different ways to do it, right? Is I think some of the obvious terrible ones that we're all aware of are like lines of code, right? I can write more lines of code, write more lines of code faster, therefore more productive. Well, we know that's a metric, right? It's the same as saying like um I don't know like measuring number of uh bugs closed, right? Um okay, you want to measure the number of bugs closed because that's going to mean how, you know, productive a team is at fixing things. Well, what happens if you're just making more bugs and closing them, right? Like the number of number of them goes up. Does that mean does that mean you're doing better? Like no. You can you can gain the system, right, by by inflating metrics because if you're being measured on something, then you you're going to influence it.

So, it's the same thing like with if you have a QA team, the number of bugs found, right? Like, oh, they're doing a good job because they're finding all these bugs. Okay. Well, do the thing that makes the number of bugs go up that are reported. Are all bugs equal? Some bugs are just people's opinions about things. They're like, I don't like how this works. It's a bug. So, my point is that how we measure productivity. These are just some examples of like obvious ways that are not good. And with that said, have we found good ways of measuring productivity on software engineering teams? And some would argue yes, right? I know there's some camps of people that would say, you know, if you follow, and I'm not saying it's specific to like agile, but there's, you know, maybe you're doing something like you're following canban or something else and you're able to say, look, like we have uh let's let's just for the sake of conversation, we have user stories.

So, we have things that we have identified are our uh value deliverables and we size them and scope them particular ways. It's, you know, relatively consistent for the most part, right? Not everything's perfect. And so based on how we measure our throughput of those things, we would argue that if that throughput goes up, we are being more productive. Okay. So like maybe there are some types of development uh methodologies, systems that people use that that you could make a a better argument for that. Why is the traffic already starting? What is going on here? One sec. This guy's blocking me in and I got to get over. Um, one more. Come on. Come on. You're killing me, Smalls. Let's go. Okay. Um, how bad is this? Oh, it must be just here. It says it's only going to be 36 minutes to get home. In the past every time it's been like over an hour, so it must just be here.

Anyway, um so yeah, like you could argue that some camps of people say like, "Hey, follow these uh development methodologies and you know, we have a pretty good way to to measure progress." So that's cool. And I think if that's great if you have some things like that that work for you, awesome. Um and I think there's lots of people that don't. And I don't think that there's a universal system that works because even I I would make a I don't know I feel like I I could say this quite confidently that there are probably lots of people that do really like something like canban and they would make an argument and say like canban isn't the perfect solution in all of these cases. I know there's going to be people that will always say something is the best for everything but I think there's lots of rational people that would say hey just because it works in our situation does not mean it would work for everyone.

So measuring developer productivity I I do not believe is something that is uh is a like solved problem. I think that it's a difficult thing to do and I think that it's difficult to do in sort of a generic sense. I think it's a lot more realistic that on a per team basis that you could do this kind of thing. Can we get one more over? Yes. Someone had the same idea as me, but they're slower. Um, so I I I do think like on a smaller scale that's not like super generic, you can start to do this a little bit more effectively. I think what's happening is that we have a lot of companies that are not doing it uh for from the perspective of AI. They're not trying to measure on specific cases. They're going, "Well, we needed we needed AI because that's what the world told us." By the way, I don't disagree.

I that AI is going to be helpful for productivity, but um you know, we need AI. everyone has to do it. And instead of having like a plan, instead of going like we're going to introduce it and here's our plan for making sure that we're like actually getting effectiveness, it's we need AI and like buckle up everyone because like you better be doing it and I guess we'll figure out what that means later. And I think that a lot more companies are getting to like well it's later what what's happening? And it's really interesting. I think um I think there's a mix of a whole bunch of stuff going on on one side or one part of this. I think that there are still people and it's okay. It's not like a problem. I'm not here to like make anyone feel bad about it. I think there are still plenty of people that either don't use AI or don't they're not familiar with how to use it effectively.

And like again, I'm not not trying to make anyone feel bad about it. I think that's that's a reality. What's going on? These people got to move. Give me one sec. I just want to like make sure I can merge lanes over here without getting in an accident cuz this is silly. One more. This is how we get that 36 minute unlock. It's not waiting with all those cars. Okay. Um, so there, yeah, there's a group of people that still don't have like what I would call like an AI unlock, which is, "Holy I tried doing something and I actually felt that this was going way more effectively with AI." And that doesn't have to be that you wrote a million lines of code with Claude. It doesn't mean that like C-Pilot refactored everything or you sent off something to cursor and it built a whole feature.

doesn't have to look like that. Sometimes the unlocks for you could be like, I needed help with a document and all of a sudden like I kind of tried something out here and and holy that made putting this document together way better or I needed to analyze data and like I think I know what I want to do and I think I know what data I need but like but holy crap, how am I going to sift through all this and make sense of it? And then you use AI in some way and you're like, "Oh my god, that just took a problem that was obviously going to take me way more time or I didn't understand it fully." And I had this moment of of literally like being like, "Holy this this was super powerful." So, some people haven't experienced that yet, and that's okay.

And I think that I would just encourage people that if you're if you're like, I don't really know where to start, like don't try not to shy away from it and try not to feel bad about that. That's absolutely okay because like literally everything we all have to start somewhere with it. And I think that in particular as software developers, you know, there's it's a group of people that is probably, you know, quite intellectual. There's a lot of us that pride ourselves in being intelligent and and suddenly not knowing something or not being familiar can feel wildly uncomfortable, right? We don't want to admit it. It's embarrassing or uh or we get resentful of it. We want to shun it. And I would just say like, you know, try try not to lean into that. Resist that a little bit if you can and like take some time and give yourself like what's the word I want?

like uh like make some space for yourself like to be kind to yourself. Like it's okay if if you're like I just don't know how to do this yet. That's okay. So I think there's a group of people like this. I think there's a group of people that are um you know they're like okay like I'm in the group like let's say the exact opposite end of the spectrum here and they're like I got I got the unlocks. I got my groove here. I know uh the problems uh that I want to solve with AI and how to solve them and I can do it effectively and like in at least in isolation give me problems to solve and I can use AI to solve them significantly faster than I ever could have before. Okay, so let's say the opposite end. And then and then there's obviously people in the middle, right, that are you know I would say maybe the most people And I don't even know if I can say most people.

Maybe most people are in that group that like they haven't found the unlock yet, but let's say there's a large group of people that are somewhere in the middle where it's like, hey, look, there's at least some things where this definitely helps. And I think all of this is positive. Personally, I think it's all in the right direction, right? We have tooling to use. If it was AI or if it was something else, we have tooling to use that gives us the opportunity to be more productive. Cool. I think where this starts to fall down is assumptions about expectations around this. Okay. So, it's like again the some of the stuff I'm going to say here is either like I don't know maybe it's like extremely obvious and you're like why are we talking about it?

But this idea where, you know, let's exaggerate it a little bit, where companies are like, "Cool, if AI is going to be such a productivity unlock, let's just get rid of people because we know that using AI is going to give us like a 10x on our developers that we keep." So like, if we do the math, we can just replace people. And like I think most of us know that's a a really shitty idea, yet some companies are still doing that kind of thing. And I think there's uh been some good evidence where like companies that have tried doing that are like that was probably the wrong move and other companies are are you know in the middle of burning to the ground or they're like why isn't this working? Um so I don't I don't think that it's anywhere near this uh the formulaic thing that you can do right now.

I don't know. Maybe maybe some people really disagree with that. That's cool. I just don't think it's at a point where you can do that. So, you got people that are we made the business decision, right? And like uh I talked about this in another video recently, like when you have people in leadership positions in a business making big decisions, that's effectively like that's a gamble, right? That's a calculated risk. You are gambling. And the hope is that you are on the right side of that, right? And you have good ROI from it. It's not a guarantee though. And I think what's happening is that some of these companies are so convinced that there was no risk or super low risk and such a huge promise of some whatever some outrageous productivity multiplier and they're going, "Well, where is it? First of all, where is it?" And because we don't see it, like something's obviously wrong.

Like we got to like some someone fix it, right? We we bought the thing. It's not doing the thing. Someone someone fixed the problem. And they're not realizing like maybe the problem is that we had a a shitty strategy. I'm not saying that because AI is bad or it's not good or it's not going to help or it doesn't have the possibility to do that. I'm saying the strategy was dog to begin with because it was like we just got to go allin and it's got to work and we're going to see productivity go up and to the right. Well, if you weren't able to measure it in the first place accurately, what the hell makes you think that you're going to measure it after accurately?

I I I don't know like am I is that a crazy thing to say that if you're a company that you don't even know how to measure productivity of your developers right if you don't know how to do that today and you go well we're going to 10x productivity or do some math here to trim workforce because we can you know multiply productivity ity if you don't even know how to measure it. What makes you think that you're going to have an experiment that works? And I picked a word there, experiment, because that's what it would be is like you need to be able to measure something, make a change, and measure it again. you're if you're not able to even have a baseline, like what makes you think that you're going to be able to magically start measuring it later? And again, I'm not I'm not here to say that making a decision like that cannot result in a productivity gain.

All that I'm saying is if you don't know how to measure it, like what are you expecting? It like to me it it seems stupid. Like I think stupid is the word that I would use. It seems stupid that if you don't know how to measure it and then you're going to make a big change and expect that you you have a measurable result like if what are you measuring? And so I think that some people are getting confused somehow with productivity versus costs versus like output of something versus value delivered, right? Um I don't know. I I think that people they're not measuring things and then they're blown away when they're not able to measure things after. So what's the answer to this right? Like is AI improving productivity? Can we measure that?

I think one of the things that I'm observing a lot more of is that especially when you have uh or like organizations right that are in a position like this where they're like well we're doing the AI thing and like this you know this decision we made has to pay off it has to right kind of not willing to accept that it is a gamble or there's risk with it but like no it has to and I will keep adding the disclaimer. I think that it it will actually be a very good thing. But anyway, well, it has to work. How do we make it work? And then, well, we have to incentivize people to make it work, which by the way, like if it was a good thing, like, wouldn't that be happening more naturally in the first place and not have to be forced?

But anyway, now we need to incentivize people. Show us the AI. Show us that it's so good. Show us that we made the right decision. Right? It's on you now. We made We made the leadership decision. It's on you. You have to make the AI do the AI promise. Okay. So, now everyone is incentivized to go do it, right? Let's let's make it happen. Well, if you if you do AI things, that's going to put you ahead of your peers, right? That's you want to you really want to demonstrate that you're doing a good job. Well, everything's AI now, so you better be doing it with AI. Show us that you're AI, right? The industry needs AI native developers. Everything's AI. If you're not demonstrating that you and AI are one, you're I'm exaggerating a little bit, right? But that's I I feel like that's kind of what's being pushed.

And so everyone is incentivized to demonstrate that they are doing the best and coolest AI things. And I think the side effect of that and it has nothing to do with AI specifically. It's just the example that's being focused on is that when you incentivize people this way, you end up having a lot of individuals. What does that mean? Right? That's a maybe it sounds like a weird thing to say. Um I don't I don't think about software engineering organizations as uh as individuals. Yes, there are individuals that you know compose of a software engineering organization but I think that a software engineering organization should be made up of cohesive units that work in the same direction on targeted things. So you have areas that you you own that you drive. You have metrics that you own and drive, but everything ideally is aligned in the same direction.

Maybe focus on different parts, but aligned in the same direction to deliver value to customers. This is my belief. So when you end up having the majority of people that are incentivized as individuals more than they are incentivized to deliver as part of a cohesive unit to an organization and thus deliver value to customers, I think that you create a very big problem. And like I said, I don't think this is this is unique to AI. I think that there are companies that did this before AI and they can do this with their performance and compensation structures, right? Like forget AI. People want to do well in their careers. They want to get paid more. They want to get to the next level. They want the next opportunity, right? How do you do that? Well, some companies unfortunately and it's even some managers specifically frame things in a way that you are more incentivized as an individual, right?

Show us that you can do this. Show us that you are the person you delivered the impact. You did the thing. And then the side effect of this is that when people are hyper motivated that they specifically as an individual need to be you know the main contributor the star of the show. You have a situation where an entire team organization company is hyper motivated as individuals and not a team. And this person really needs to switch lanes here. That's right. you without the signal. Absolute dummy. Um so if you are hyper incentivized for certain metrics on whatever and you're operating as an individual the problem I think is that when we start trying to measure productivity could you look at an individual and then say yes this individual is being more productive I think the answer is certainly like why not oh buddy you're that is not safe.

Chris is weaving in and out of traffic and cutting people off. At least I went around someone that was going under the speed limit in the left lane when they moved over. That person not doing that. Um, so yeah, I think you know when you have a bunch of individuals that are incentivized this way, could you measure their performance and say, well, look, like Billy over here has been able to 10x his productivity and Sally has, you know, done all of these things that she never could have with AI and like or and again, you don't have to pick AI for this example, but that's what we're talking about. So look how good they've done as individuals, right? And if you take them in individual context, great. Like now you have a case for promotion or compensation or whatever it is. But this is not how teams work.

This is not how organizations work and it's not how companies succeed. So like what what are you thinking or what are companies thinking when they try to motivate people this way and incentivize people this way? Because what are you actually trying to do? Do you want do you want your company to do better? Do you want your organization and your business to do better? Because my opinion is that if that's your goal, and I would I would hope it's your goal, then stop incentivizing people to be better individuals on their own in isolation, right? You could take someone in isolation. I'm just going to give you an example of what I mean if I'm if I've lost you, feel like this guy's just rambling and ranting. Um, take someone in isolation. that's a software engineer, software developer, building things. And you look at how many uh now that they're using AI, let's use AI.

If you don't want that framing, pick a different tool, whatever. They're now using something in this context, it's AI, and now they can they can do twice as many poll requests, right? They've closed five times as many bugs. they've written 100 uh x number of lines, right? Like we're we're back to measuring these stupid shitty metrics that literally don't translate into anything that makes sense. You could even argue like, look, they but they shipped these things. They've shipped these things. Okay, are those things actually valuable? Well, yes, they're, you know, let's make another argument. Yes, they are valuable. Yes, those are things that would help. Okay, cool. Were those the highest priority things? The answer is probably not, right? The answer is probably not if you have individuals doing this. And I'm not saying that it's not possible. So, I think that's very easy for people to hear what I'm saying.

and being heated about it. I'm not saying it's not possible. What I am saying is that it's more statistically unlikely that when you have individuals highly motivated for their own personal well-being and growth and whatever else that they are not going to be focused on the highest priority thing that a team, organization or business needs. And why? because they're not working together on solving those problems that are the highest priority. Again, not that it's impossible. I think it's less likely. I think it's less and less likely that you end up getting that outcome and you end up having more and more people doing things, being busy, and not actually shipping value. This is when you have people saying, "Look what I've made. look what I've done. And you go, great, that's super cool. And you have more people doing that. Great. That's super cool. And then you start to realize, okay, well, we're doing these things, but like what about the value that we're trying to ship?

Like what? Back to how how do we measure our our productivity? Are you measuring the productivity because people did cool things and you see more cool things? Because by that measure then I would say yeah probably give people space to go use AI and do things and you'll probably see more cool things come up. I I think you will see that and I would say if you're not seeing that that's maybe you know more education familiarity with tooling getting people comfortable with it right you know like basically guiding people to to get that experience but I I genuinely don't think that's the goal is just to have more people doing cool things because if doing cool things is not equivalent to shipping value to customers. I don't think that you're measuring the right thing. So what is the point of my rant?

I think the point is that one uh I think that people people as in companies organizations um you know didn't know how to measure productivity before and then they think they found a secret and they're they're confused because it's not translating into some other measure maybe that is or is not productivity. But there's a misalignment there. And I think the other part to that is that it perpetuates I feel like it often perpetuates well now it's up to you to go you individuals go solve this. Like we need to see that you're doing the thing with the AI like you're the ones that have to unlock it for us. And then you end up creating more situations where uh people aren't necessarily working together. And that's my take on this stuff. I do think that you can have a lot of really good unlocks. I do think that like my personal opinion is that AI is really powerful and we can do way more than we could before, different things than we could before.

And I don't think that um I don't think that it forces us to be in this situation where we you know won't see benefits. But I think that there is a a huge discrepancy in in people's expectations because they didn't know what they were measuring in the first place. So my my recommendation is sort of this mix of like I I think it's super important to help people get familiar with tools, right? Here's different things you can use. Here's different situations you can try it out in. Making space for people to to to not know, right? And to like make sure that people are comfortable asking questions, that kind of stuff. I think that's super important. I think it's super important to showcase when people find cool things and like, hey, I tried this and it worked, right? you get to see some of these uh more inspiring things as well.

So, a mix of that I think is good. But then what I think is important is that you don't incentivize people as individuals like go off do this thing and like you're the star of the show. I don't think that's specific to AI. I think that there are many examples now we're seeing with AI where it's like that's going to be a thing. So find ways to encourage what we would have done before which is like if we want to drive positive outcomes what are the goals that we have what are what are the things that we measure as a business that we're trying to drive if you have teams that are responsible for certain metrics for certain things that are like certain types of value being delivered to customers. I'm thinking about different services different products different parts of products that different teams work on.

you're going to have your own things that you are trying to drive. And so, how do you how do you take what you're normally doing and like say, "Okay, cool." Like, we're going to I'm just making this up from like an OKR perspective, if you're familiar with that. You set objectives and have key results that you're measuring against and you say, "Cool." Like, usually when we do this, like we're trying to achieve I'm just making something up. We're going to try to achieve like a 10% increase on this. We're going to go from uh you know some measure to of like 200 to uh to 240, right? We're going to get a or 210 do math. Um so we're going to see a 10% increase on this. And now like you know what we actually want to challenge ourselves because now we have AI. Um we want to we want to see if as a team as a group that we can become more effective.

We want to see if we can do more with this. Like do more with the same, right? But the it's not the same because you have AI. So we can push ourselves more. Let's let's try to do a 15% increase. A 20% increase. I had the 20% in my mind, which is why I said 40 already. So we're going to try to hit 240 this time. We're going to we're going to push the limit on what we can do. And so instead of having individuals where you're like, "Show us what cool things you can do with AI," you're like, "Hey team, we have a challenge for you." And that's we're going to try pushing a little bit harder to see if we can get creative with how we use AI to drive the things that we give a about. Now, I don't want you to hear that and think, "Well, that's just going to make more stress.

Like, everyone's just going to add more uh more things we have to go do." I'm not saying that's the solution for it, but I'm saying I would rather see people move in that direction of let's take a team aligned on some goal and then say let's let's try setting a slightly more aggressive goal to see if we can do better. Like that's how you get improvement. So yeah, I don't know. That's my framing of it. Very curious to hear from other people what your experiences are. Um I realized that was very ranty and probably the side effect of that is like well is that what's happening at your work? No, I think the thing that is happening for me like I was saying is that there's a lot more of like helping educate people on different ways you can do things. Um certainly we have and I don't think this is unique to us.

there's this experience of like lots of people trying different things and you'll see um sort of like the same experiment repeated in different ways and some of that's really good because people get to explore and I think we're seeing more like okay cool like now that we're seeing these patterns come up how do we like word I want to use is refactor like how do we pull this kind of stuff together and converge refactor our efforts converge them on something so I think we see a lot of Um, but the overwhelming amount of I see online around like just companies being like baffled that they didn't get their 10x increase and then like blaming people or incentivizing people the wrong way is just like it's uh it's reaching a certain point for me. So curious to hear from you. If you got questions, leave them below in the comments.

And of course, even if they're not related to this video, um you want software engineering questions, career questions, please ask them. Or you can go to codeccommute.com. You can submit stuff anonymously. And I hope to see you in the next one.