Code Coverage Gates Or Other Ways To Get Devs To Test

Transcript is auto-generated and may contain errors.

Hey folks, I'm just leaving work here. We're going to go to experience devs. This one was a topic about uh getting developers to write tests without specific code coverage requirements. And I didn't actually read this whole post. So, I just kind of thought this would be interesting to talk through. Um, because I think it's interesting where you have worlds where, you know, code coverage enforcement is like, you know, it's the reason that people ever write tests. Like, basically, they don't give a except they're like, well, the only way to get my code through is to make this number match. Um, and then you have, you know, other places where it's like people people want to write tests, they see value in them, and then like how do you navigate this kind of space in the middle? I don't know what I'm going to do about this car beeping, man.

It's like I want to I want to get rid of it. I can't stand it. So, um I will mention this as I start talking through uh kind of right at the beginning here that want you to keep in mind like what does that look like for you, right? Like are you at a place where you're like your only motivation to write tests is because like that's the thing kind of gaining your poll request and if you want to get it through, you got to do it. Or is it something where you're writing them because you've seen value in them? Or are you at a place where you're like, I want to write tests, but the code is the code base is so shitty that I can't uh I feel like I can't even get my code tested. Be very curious to hear from you in the comments.

So, keep that in mind as I'm talking through this. Um, so yeah, let's kick things off. Um, I I guess to start, I I like the idea, hear me out, I like the idea of of having code coverage as a as a metric. And like I say that because I think that there's interesting data that you can get from that. Uh, but I don't love in practice code coverage as a uh the code coverage metric as a as a block. And like in theory, like in theory, it sounds cool, right? And you could do things like um you know, code coverage can uh within a threshold you need to to meet it, but it can only go up, right? So, for example, you're I'm just going to make it up. Like if you wanted to have code coverage, your code base at 80%. You're like, cool.

It's a new code base, though. So it's at um 0% to start with and then now it's at 20, now it's at 30. Like basically it has to keep going up with some buffer until you meet like the target minimum, right? So like minimum once it's established is like it's got to be at least 80% or 90%, whatever. Like in theory I I'm like, hey, that's a cool idea. Like it that would get people to, you know, always make sure that they're writing tests. But what I find in practice happens is that you get code coverage that is kind of like right around this point, whatever target you set, and then inevitably you're kind of fighting the system where it's like someone puts up a change and it's like, oh, like the target's 80%, you're at 79 25%. and you're like, "Well, okay." Like, I tested all my code that I added.

Like, it's it's well covered and like I feel good about the tests. It's like, nope. Like, you got to add more. And so, then then you're sitting there like, "Okay, like what kind of do I have to add here?" And then then you add some like silly tests like that you don't really think add value, but you're like, "Fine, it checks the box." And then over time. Wow, that car is the same as me. We got to catch him. Hello, friend. Well, that was shortlived. Um, they're behind me now. Cool. So yeah, inevitably that kind of like starts training you to like not give a because you're like, I want to write tests for my stuff cuz I value them, but at the end of the day, you spend more time just like fighting the system to to get like a number, right? And then you're you're constantly adding these tests where you're like, I don't really care about what I'm adding.

I just got to meet this number. The other thing that happens, again, inevitably what I've seen is that code coverage tools, there's inconsistencies. And so again, you might you might be running things locally, feel like you have good coverage, and then you push things up and you have different coverage, and it's frustrating. Um, you might have situations where like the tool is incorrectly reporting that like you don't have enough coverage and like you clearly do because I don't know, you have different ways to prove that and then again you're just kind of fighting against it. So once you reach this threshold and you're near it, instead of it being like incentivizing to get people to add code uh like add test for their code, sorry, that it's really just like doing the opposite. Holy moly. I guess like the sun is bright here, so people go like 20 m an hour below the speed limit.

Like if it sounds like I'm driving really fast, I've just reached the speed limit right now. Just as a heads up, I'm actually going below it still. That's how slow people were going. Um anyway, so like I like I said, I like the idea of of it, but I don't in see it really adding value in practice, which is like really unfortunate because I'm like, hey, that would be wouldn't that be such a convenient tool that you go run this, it gives you a number, you got to meet the threshold, it's incentivizing. It's just not from what I've lived through. Like it it really doesn't feel incentivizing. It feels like a pain in the ass. So I like having the number. I think the number is helpful. I think if you we had a trend of it, it would be helpful. I think that when I'm writing tests personally because I don't people will freak out.

Sorry, but I don't uh I don't live and breathe TDD. Um there are some things absolutely that I do that TDD feels very natural for me and plenty of other things I do that I don't find it natural. And I'm not going to sit here and debate you on TDD being the best thing in the world or not. Um, I'm just telling you that's how I live my life. And so, uh, one thing that I like using is when I'm trying to, uh, build confidence in the thing that I've put together, I love running my tests and then using code coverage and looking at like, oh crap, like I totally missed this file or like, do I even care this doesn't have tests? Right? like my DTO, my data transfer object doesn't have tests on it. Oh no, like I don't care, right? But at least now I have uh you know a representation of that that I can understand and have clarity on.

So I like code coverage. I like having data, but in practice I don't think gating on it has been um super valuable. In fact um more of a net negative to be honest. So, to this person's point, how do you get people to write tests? Right? We're not going to enforce it. How do you get people to do it then? How do I get over this lane? Nice. Okay. So in my previous life before before Microsoft um I was at a company where you know being there right near the beginning when it was just a few of us like the the code I mean we didn't have tests like we were shipping a product there was there was absolutely no test And you kind of learn the hard way. Um, not only like, hey, like we should add test because there's value, like how to write tests, what you actually are testing.

I'm not sitting here saying like I'm I'm perfect at it, uh, by any means, but the like we had to learn the hard way. Like it's just the reality. We were shipping things, right? And it was a desktop software. So we'd go to to ship stuff and inevitably what would happen is we'd go to cut a release and there's no tests, right? So we have to build some confidence. That gone was overdue. We had to build some confidence in what we're building. So like, okay, like go go click through the app, go use it, right? And without fail, of course, what would happen is that we'd go clicking through the app and we're like, well, this shit's broken, right? Like this doesn't work. That doesn't work. And it's like, how did we we just went and added all these features and like, you know, like nothing even works.

Like what are we doing here? So then, okay, now you got to go fix all these things and you're adding fixes and then, okay, well, let's cut another release and then we'd go to we got to test that too, right? And then of course, you're compounding this fact that you're like trying to rush it because you're like, man, we got to we got to ship this thing. Like we keep keep finding more bugs and we keep trying to fix them fast so that we can go ship it. And it's a vicious cycle, right? Like I I realize as I'm saying it, you're probably like, "No that's not going to work." Yeah. Um doesn't work too well. So then it was like, "Okay, well, we don't want to keep breaking ourselves. Like, let's start adding tests." So we'd add tests. Were we writing good tests? Like absolutely not.

Were we writing tests in sorry, writing code in a way that was testable? Absolutely not. Right? like we had so many patterns that were just counter to being testable in any capacity, right? Like tons of inheritance, tons of, you know, static global variables. It was it was chaos. And so the the reality is like not only were we not good at writing tests or understanding what needs to be tested or how to test like we're also just not good at writing code in a way that can be tested. And a lot of this we had to learn the hard way. So I'm saying this because like I I think that sometimes people don't realize the value of tests because they legitimately have not been in a situation where uh they've kind of felt the the pain of not of not having tests, right? Like of fixing something and then that causes a break in something else which causes a break in something else.

Like you keep fixing and breaking because you literally don't have anything to help with your regression visibility. I got to move over a few lanes here and I got to think harder. So, give me one sec. I'm coming over, buddy. Let me in. Cool. Um, yeah. So, like to me, like how do you how do you convince someone that writing tests has value? I think there's, you know, there's plenty of reasons why tests have value, but how do you convince someone who's like against spending more time cuz they perceive it as just more work? How do you convince them that there's value in writing tests? Right? Um I realize like I'm creating a scenario that may not exist for you. You're like, "Well, I don't know anyone that that thinks that way." So, sorry.

This is maybe a little bit contrived for some of you, but um I've definitely worked with people who are like it's just a waste of time to do that or um they they perceive it as a waste of time because every time they have to go write tests, they're being told, "You need to go add test. You need to go add test." They're like, "Well, I have to go rewrite this. a Bentley, right? Because they they wrote all this code. They didn't write any tests and then someone's like, "You can't submit this until you add tests." And now they're starting they're like, "Well, I I literally can't use this code. I have to go test it. And to go test it, I have to go rewrite it because I can't add tests on this." So they're sort of reinforcing for themselves like this is just a waste of time.

It's always extra work and then they do it and they don't find themselves in a situation where they like again feeling the pain of not having to uh of not having the test story. So for me, um, if we just kind of generalize this, how do you get buy in on things, right? So you have to kind of talk to people about things in ways that that they value, right? So if you if you listen to if you're getting push back from people about writing tests, right? like listen to the things that they're they're complaining about, right? Just to to give you examples, right? Like I want to write tests. I really do believe in them, but like it's so much work because uh you know writing writing these tests uh in this codebase like is terrible, right? Like we have these crazy test files to maintain.

The setup is ridiculous. Okay, that's like that's maybe one category of things. You have the example that I gave where it's like I I always have to go back and you know do double the work because I'm writing tests after. Okay, that's another category. Like listen to what people are saying about them. The tests are always broken. Right? There's another good example. Like we have so many tests. The tests are so brittle. They're always broken. Like I don't want to write more tests. Like I don't see value in doing this. I think if you listen to what people are saying, then you can think about sort of your how how to I don't want to say structure an argument, but like structure your perspective um that you're sharing with them in a way that they're going to value more. And you have to do that by addressing the problems, right?

So it's to me it's an easier way easier cell if someone's like is always broken. I wish we didn't have regressions. You'd be like, hey man, maybe it's a good idea to add some tests like oh like tell me more about that. Right? So it's it's really only a problem when people are resisting it and generally that's because it's more work or they see some problem with it. So I think you know step one is identifying what challenges are people facing with it. What kind of friction is there? kind of like what are the people that what sorry what are the reasons that people resist or resent this kind of thing, right? If um I have a a direction I want to go with this and I'm trying to like jump there, but if you if you have groups of people that are kind of like I don't see the value in this and it's just more work, then it's going to be really difficult to get any buy in from people on this.

And that's what this person was asking about in the first place on this Reddit thread, which was like, if you don't have something forcing it, how do you get people bought in? Like, what are the things you do? And some of the things that they listed were like they seemed like they were different ways to to get a metric, right? like maybe not necessarily code coverage, but they were like, "Oh, like you know, um you changed a file and there I'm just making this up to they didn't say this specifically, but you changed a file and there was no corresponding change to like a file with the same name ending in test, right?" Like, so therefore, you didn't add test coverage for for that file. I got to get over there, buddy. Let me in. No. Okay, you go ahead, Ford Fusion.

So, like to me that's just another way of like kind of gaming it because you might say, well, sure that's a that's a good trigger to catch things, but like back to my DTO example. So, like, okay, I didn't add tests for my DTO. Like, do you really give a Are we really enforcing tests on data transfer objects because now you're automatically gating for that, right? Like, okay, so maybe I'm going to work around that by not putting my DTO's into other files. Now you can't match on that. I'm going to go, you know, I'm just working around the problem. I got to game the system because it's not adding value. So, some of their examples were like this, and I'm like, I I don't love the idea of gating things that way because instead of the value ad, you're going to have people that get frustrated and just actively work around it.

Like, it it seemingly always backfires when it's a gating mechanism. Now, again, what do I think would be helpful? like use the same idea and then give me a little report that says like you didn't add tests for these, right? Like I'm I'm okay with that. That's cool. And then I can look at that and be like, "Oh, you're right." Like I missed this I missed this service class or I'm looking and I'm like, "Check, check, check. It's only DTO's here." And then I move on, right? Like I think when you uh enforce and you gate then I think I start to notice some challenges with that. Now the reason that works for me is because I actually want to write tests. So uh my point is that the some of the examples they were giving were like gating mechanisms and what I want to propose here is like how do you change the narrative to like this is a thing you want to do.

So that to me right goes back to asking and surveying and understanding your team like where um where they have any challenges. And if basically if there's no challenges in the first place, you're just trying to to start this off cuz you're like I think it's a good idea to to write tests. We don't really do that on the team. Like if no one if you if you pitch it, if you're like, hey, I think we should start adding tests with our code. Um, if you pitch it and people aren't against it, then great, like, go ahead, start. Uh, I think it's more of an issue if you're pitching it and people are resisting it or they don't perceive um, it might be less of resisting actively and more that it's not a priority, right? So, okay, you want us to add tests, like, no problem.

Like, I'm just going to commit this and push it up and then I'll follow up with the tests, right? like sure I'll do that and then in reality like I don't actually prioritize the test so something more more important comes up and then I I don't end up writing tests. Um so again do you go to well we need a gating mechanism so that doesn't happen. I'm like I just I just don't think so. I think you want to show people that there is value in adding tests and I think that you can show people that if you write code in particular ways that you can test it quite easily. Right? So I mentioned already while talking that I do TDD for some things but I don't I don't think about how I develop as TDD based. I don't think that way. It's just not how my brain works when I go to build software.

I don't like doing it. Um, there are some things where I'm like, I really need to lean on that. When I use AI tools, a lot of the time I will say like, use TDD. And I do that because I'm like, I don't trust you. I don't trust you to be creative with this. Whereas for myself, should I trust myself? Probably not. Um, realistically, but it's just not how I operate. I need to be creative. I need to kind of feel out and explore the code in a way that works for me. Um, and then I then I want to add tests. I want to see it working how I expect and then add test. So how do you convince people that there's value in doing it right? One of the ways I mentioned is like you might be in a situation like in your your team you have a service or a product.

If people are like we have so many issues, right? Like maybe you already have tests and that's not really the big issue. The big issue is that you have shitty tests, right? People are just putting them in because they were told that they need to have test so they just do it and they don't really give a But like how do you get people to care? And I really think that it using like post incident reviews to kind of like use it as that as a checkpoint. Um, I think I think that can work really well. Um, to give you an example, like my world before at a startup that kind of grew into this this uh medium-siz business was like it's desktop software, not a live service. And so when things broke, a lot of the time the conversation was like, oh, like we didn't there was no test coverage on that.

So like that's I mean it was easy to break unknowingly because there's no test coverage for my life at Microsoft especially being on the deployment team before um the current team I'm on now. It's the conversation completely shifted. It wasn't even did you have tests on this? You should have tests on it. The conversation was actually about like when you're deploying these changes, did you flight them properly? Same same idea, right? Like did you use the tool that gives you confidence early before breaking It's just that the tool you use, depending on what tool it is, can give you that signal earlier and earlier and earlier, right? Like if you wrote no tests and you never even ran your service change and then you go start rolling it out, your earliest signal is going to be the first time that service gets used. That might be way later than you want and it might have way bigger impact than you want.

So okay, if you flight it, can you control how big of an impact that has? Sure. But like you could go one step ear like before that like did you run regression tests in your CI/CD pipeline like did did you even run it you know locally on your machine to make sure right like basically just keep going back further and further. I think they call that shifting left. And can you get an earlier signal? And so my point is that you can have these conversations after people have felt the pain. It's not it feels kind of counterintuitive like don't you want people to not even have to experience that? like, yes, of course. But I I think that it I think that it's an easier um conversation when something has unfortunately broken, right? And you're talking about it after and you're trying to get to the root cause and understand things and having a positive non-blaming conversation about this, you can go say like, "Hey, like, you know, why why wasn't this caught sooner?

Can we look at that?" Oh, well, we we flighted the changes. Great. Okay. But like, did you have tests on this? Was there anything that was running in CI/CD that you could have used to catch this? Right. And I think that over time, it's it's a longer term kind of thing, but over time, you kind of show people that these types of issues, not everything in the world, right? that many classes of issues if you wrote regression tests for them you could have uh a lot of coverage and feel more confident right so one of my meta points here is that I think that when you talk to people about the pain points that they're experiencing uh and in in sort of two capacities one is like after there's been an issue like that's painful to have to go deal with that can you kind

of use that as a a mechanism to say like hey if we had tests don't you think that would help us catch this right uh and then we can avoid this type of painful situation that's one part the other part is like if you're having that conversation or even just pitching the idea of more tests different types of tests and there's friction listen to the pain points why where is that friction coming from okay now go address those So, I think when you can get some buy in, and by the way, I should mention too that um just because I said some buy in, I think what can help is like you don't have to convince everyone all at once, right? If you're working in a team where it's like you don't have tests, you're like, "We should really do this. I don't know how I'm going to convince all these people we should." Can you get some buy in?

Like, could you get enough buy in where like you're allowed to start contributing tests without people freaking out? Cool. Okay. So, you're the person who writes some tests. No one else does. No one else cares. Okay. Can you can you demonstrate by working with you know one other person that they're like oh like I see the value in this like you know I'll do this too like can you start to build some momentum because that can also work as a a way to showcase to others like see the value see how this helped us and to share a quick example with you um we had where I used to work we had this situ I've told this story in different ways before on this channel, but we had a situation where um my team was kind of brought in to help uh another team. There's no stop sign.

Please keep driving. And uh so we were contributing in in their part of the code to be able to help meet some deadlines. And what was happening was that we would go make changes and then like the next sprint when we're all working on the next areas, what would happen was that that other team would be very frustrated that we broke a bunch of their right? So they'd be getting frustrated. We're basically slowing down their sprint. And then like right after that, we were like, "Okay, well we have all these bugs we have to go fix." like it basically would keep flip-flopping. And the reason it was happening was because we were breaking that wasn't covered with tests. So when you have a group that's not testing anything and you're making changes in the code, like we could be adding tests for all the stuff we're adding and feel very good about it, but like if they don't have tests, then we're going to have an issue.

Now, I don't think we were even doing a good job of testing our stuff at that time. So, um I think one of the ways that we were breaking this vicious cycle was like, "Hey, like this is painful, right? So, we're going to add tests. Now, you can't break our Like, we're adding functionality into your area. You're not going to break us. You're not going to break us because we have tests that are written in a way that help give us confidence to know that anyone should be able to touch code around this, right? you're going to break the expected behavior and you're going to get an early signal. And if you're like, well, I don't understand why this test is breaking. What the hell's going on? At least it's another point in time that you have to go ask someone to clarify like what's going on with this test.

And it's like, yes, you're breaking the expected behavior. There's another AMG GT There's that's two today. That's cool. And so, yeah, we could use that as an example to be able to showcase and say, "Hey, look, like if you don't want to run into these circumstances where other people are breaking your add tests, right?" And so, ironically or unironically, I don't know, both. Um, we had this kind of this uh experience. I don't know like it kind of shaped my philosophy on things I guess like I I have this philosophy that I would love to have my code base in a way that I can't say anyone. I don't mean like literally people I don't know and don't trust and shouldn't be touching my code, but for people I work with, right? Anyone should be able to come in and touch the code.

I'm not saying that we shouldn't review it or we shouldn't like agree on the changes, but anyone should be able to come in and work on it and feel confident that they're not breaking anything, right? Like I aspire to have a code base that I can go change and feel confident I'm not breaking stuff and anyone else can, right? you don't even know this codebase and you're you want to add a new feature or change something. Cool. I would love for my codebase to tell you you're messing up. Like you broke critical behavior. I can't guarantee that your shit's going to work. I can't predict the future of what you're going to add, but I can tell you you're breaking stuff that already exists. So like I I really, you know, aspire to have code bases like that.

And that's been shaped out of my earlier experiences where it's like being on the other end of that is so terrible because all you're trying to do is help and all that you're doing is frustrating the people you're trying to help because you're breaking their Um, so anyway, I think that you can you can use that as, you know, if you get some momentum, you can demonstrate the value of that to others and say like, look, you know, look how much time we're not wasting anymore because we're able to do that. And uh as I get closer to home here, that's kind of one of the last thoughts that I wanted to see if I have I don't know some suggestions or anything else around. But one of the critical things is this idea that like oh writing tests takes more time. And even um even aside from the people that are like oh I have to go rewrite my code to be testable.

say you don't have to rewrite your code. Adding tests is literally more code that needs to be written. So, doesn't it by definition then take more time, right? Okay, all the TDD people be quiet um because you're already writing tests by definition. But if you have to go add tests, it's literally more things to to put together. So, isn't it isn't that a an argument that people can make that's like say someone in management who's like do more and you're like oh but we need tests and they're like yeah but the tests are just extra like just ship the features right the the argument that tests are just more work um I think is something that you know people I I don't just want to say just non-technical people but I think especially non-technical people uh people that are in maybe more management or leadership

roles especially if you're working with people that are more junior or less confident it becomes like what seems like an easy argument to say like well the test that's just more work to do so like we're cutting that as a pri priority to go do. And so over time, you acrew this debt for sure. Like this is clearly debt. Um you acrew all this debt where you don't have test coverage. And so like forming this argument about doing it sooner, I think I genuinely think it's very tricky for people that don't have experience in it to be able to make a case for it, right? Because you're talking to people who maybe haven't felt the pain. So my my kind of thought around this is just going back to what I said earlier. It's like if you're trying to convince someone that you know your proposal adds value, right?

You have a manager that's like, "No, that's just extra work. We don't have time for that." Trying to convince them. Well, they're saying it's a time issue, right? They're saying it's just more time. And so you kind of if you speak the language that they're where their concern is, it's like, okay, you're concerned about time, right? Okay. Well, I'm telling you that yes, maybe maybe it's a little bit more time to write that, but I'm also telling you that to fix this or to not to fix it to to test this now, right, might take a little bit of extra time, but for me to go fix this later is not only going to take more time when it breaks because inevitably breaks. So, for me to go fix this later is not only going to take more time, but it's going to cost you. You see, they're going to cost you Oh, my wife's there.

It's going to cost you um not only potentially financially, it's going to cost potentially reputation because you have pissed off customers, right? And then depending if you have a live service, product, how does that fix look? It's just, you know, pushing out a change to the website, no one noticed, whatever. It's fine. Is it people need to literally go redownload a new patch? You have to wait for the next release. Like if you can add tests sooner to build confidence in what you are shipping then you are ideally paying less significantly less long-term in terms of fixes. So it might be a hard sell upfront and I get that. So if you're trying to approach it and explain things that way and it's not working, I encourage you to literally try documenting um the time that it takes to go fix bugs after, right?

Like do these Ps have these conversations and it's like yes, we don't have tests and like if we would have added tests up front, this is the kind of thing we could have avoided. Uh and especially impactful on repeat offender areas that are seemingly more brittle. By the way, this is a good um connector for things like tech debt. Like, yes, we have we have tests on there, but they're brittle tests or they're shitty tests or we can't test it and this thing keeps breaking all the time and we've been saying that we really want to refactor this part of the code or rewrite this part of the code, right? It's too brittle. So, I think that you can also tie tech conversations in in a similar way. Anyway, those are some thoughts. Um, I know that was all over the place, but hopefully something helpful in there.

Uh, if you got questions, leave them below in the comments. Otherwise, you can go to codecet.com, submit questions that way, uh, totally anonymously, and I am happy to try making a video response for you. And I'll see you in the next one. Take care.