End To End Tests Are CRUSHING Our Development Agility

Transcript is auto-generated and may contain errors.

Hey folks, we're going to go to the experienced dev subreddit. This one is going to be about testing. We never talk about testing, do we? This one is someone who's I guess kind of frustrated with how things are set up, where they're working, and it says that everyone only does like end to end testing. And so their sort of complaint with this is that um often find like the tests are very brittle. So you have this test automation that's running and it's great like there's test coverage and they're end to end. So they exercise a lot of things. Excellent. But um it's brittle. Things are always broken. So automation runs and automation fails, right? And then someone looks and it's like, "Oh, well, it's mostly right, but like, you know, something's something's off by a pixel or or whatever it happens to be, right? The the thing took a snapshot and like the rendering time was like slightly off and so it skews things like whatever the reason is.

um they're saying that they have basically only end to end tests and the side effect of that is that uh there's a lot of uh very brittle tests and so I didn't read the full post to be honest with you. So, I don't know if they were looking for some specific advice, but I got excited and I was like, "Hey, we should talk about testing cuz um I don't think we talk about it a lot." And actually, um I do care a lot about testing. And I find that um I I don't know like when it comes to testing, I feel like my I don't have hot takes on testing. Personally, I feel like I don't have like a very polarizing approach to testing, but I also still feel like compared to what I hear from other people, I'm like, is like are there other people that are just I hear them and that's polarizing to me or am I the polarizing one?

I don't know. Um, so I figured we'll talk through some of it. Um, and it could be kind of interesting because I do have some experience where there were uh fortunately not only end to end tests on everything, but there were some endto-end tests on stuff and uh it was a bit of a pain in the butt. And so in this particular case for this person, they're talking about like playright tests. So things like running in a browser, right? Uh for for web testing. And so, um, in my professional experience, I do not have, uh, a ton of playright testing, but, um, I do use Playright now with, uh, some side stuff. Uh, but the scale of things that I do with Playright is like like my blog is has like playright tests. Um, so I have playright test for my blog. I have um, it's a Blazer blog.

It's a lot of a lot of bees. Um, it's a Blazer blog and it uses BUnit just to add in more B's. And so BUnit is cool because um, it will basically you don't even need a browser to render things. So it will, uh, do like basically render the components in code for you so that you can assert on the code. Um, so it's pretty cool because like uh if you're used to doing like UI tests and having things like click in browsers, yada yada, um, it's a nice like step before that where you can still do some like a bunch of interactions and it's pretty powerful. Um, but it's like quite deterministic whereas sometimes with browser stuff there's just uh, inevitably going to be a little bit of flakiness going on. So, um, like when it comes to web testing, not not a ton from me.

So, that's my disclaimer. So, if you're a web developer and you're like, "Oh, this bald guy on the internet's going to talk about testing and I'm going to learn something about web testing." No, no, you're not. Not from me. But, uh, I'll talk about um some more general stuff. So, let me take a step back into how I look at testing and why I don't think it's a hot take. and you can be the judge of that. So, when I think about testing, um, you'll hear sometimes people talk about like a testing pyramid and you should have like a mix of things and like there's unit tests on the bottom and then like up the pyramid you have different types of tests. Um, so regardless of the shape, like whether it's a pyramid or you want to do an inverse pyramid or whatever, um, I do strongly believe that you need uh an assortment of tests, different types of tests, because they serve different purposes.

They give you they have different strengths and weaknesses. Oh, come on, bus. Why would you do that to me? this bus just switch lanes to go significantly slower in front of me. Um, so they have different strengths and different weaknesses. They serve different purposes. And I personally feel like if you take one thing and say we only need this one thing, it's the best thing. Then a lot of the time I'm like it's it's probably not it's probably not a great argument. like there's probably a lot of holes to poke in that argument. So it I'm not saying that like maybe for the thing that you're doing that one type of testing might be more beneficial. In fact, I think that that probably is the case for a lot of things where if you could only write one test or one type of test, sorry, that there probably is something that would be more beneficial.

But I do think that having a mix of things lets you pick and choose how you build confidence. And so one of my takeaways when it comes to testing is that like my philosophy is that testing is all about building confidence in things. It's about building confidence in what you're shipping. And so that could be, you know, you made a bug fix and you want to build confidence. Well, what does that mean? Well, do you you want to build confidence that the bug is fixed and like the logic is now corrected? I think that you could write tests for that. Do you want to write tests to ensure that the bug doesn't regress? I think there's types of tests for that. Depending on what the bug is um you know in in what kind of code base, whatever. I think there are different types of tests that are better or worse suited for that kind of thing.

But to me, it's all about building confidence. So, I think, sorry, I got to switch to the fast lane here. Um, I think why I I like that framing is that Whoa. Whoa. What are you doing? That's not a Whoa. Come on, people. Let's not be stupid. That's three people in a row. You can't see how I'm drive like where this spot is on the road. I have to cross this big line of cars that are trying to go to the next exit. There's no one in my lane except there were three cars that just tried to pull out like right in front of me one after the other. Um, one of them did pull over and actually I guess had to go pull over on the shoulder, but like that was three people in a row that almost got rear ended but didn't. So, um, I like talking about confidence when it comes to testing because to me that's the goal and that's that's my take on it.

You can have your own arguments, whatever. I'm just explaining my perspective on it. So, for me, it's all about building confidence. That is the goal of testing. So, for like to give you an extreme example, right? If if that was the philosophy and confidence is the goal, you might have a you know team of people or a person who has made a new feature or they fixed a bug and for them to have confidence is like literally they looked at the code they said it compiles and they're like I'm confident in this and like it's not for me to say oh you didn't add the unit test. Oh, you didn't add integration tests. You didn't go run the full suite of endto-end tests. How could you possibly be confident? Like, it's not for me to say. It's about a team or in this case, if it's an individual, if they have confidence in their changes or in what they're shipping, right?

So, this single person might not be adding certain types of tests. They might not add any tests and they're very confident and it's not for me to say like that's wrong or right that doesn't make me confident personally but if that's how they operate great I guess for me to build confidence in things like to give you an example when I have to touch the front-end code in brand ghost for me very much like I as not a front-end developer I'm like I really want to go run the tests to make sure I didn't break anything right like it's fine If I'm making a change and I want to add some type of tests on that, like I will do that, but like I also need tests to make sure I'm not breaking other people's stuff. I need that in the front end. I need that for my own backend code, especially around like when I'm doing uh stuff for like data access.

So, I really like running tests against my SQL databases because I have had enough um issues or I've tweaked something something subtle and then like you know something so subtle in a query or in something that's using the data from a query and uh boop like everything just breaks and I'm like how the heck could that have broken? And it's because I did, you know, something completely ridiculous and I could have just caught it by by running a test against a SQL database, like a not the live production one, but like standing up a like a test one. Okay, let's get this lane change going. That's how we do it. Um so like confidence levels can look different right and so depending on the team the what's being built the state of the product um could be so many different factors you will have different

uh like requirements or expectations for what you need to be confident right I've talked about like where I used to work was in digital forensics and sometimes we'd have to go fix bugs for customers customers and they were literally impossible for us to reproduce. We just we simply could not do it. And a lot of the time in situations like that, it's because the customer would be have like a hard drive that had a corrupt file system. It's forensics, right? They're they're looking through file systems and on on hard disks. So disc would be corrupt, file system would be corrupt. And we're like, cool. like you can't obviously ship us that hard drive but um actually in some cases we did get hard drives but most of the time no right and so okay you can't give us the hard drive you can't give us

an image of the hard drive okay can you tell us like like do you have enough information to know like what's corrupt on the disc or with the file system okay like we got logs yep and the log is like a crash log that doesn't even have a stack trace And so like how do we go figure out what's going on? So anyway, so with having very limited information and I'm kind of sometimes exaggerating with the stack trace, but let's say we have a stack trace and we're like, okay, we can see what part of the code is throwing an exception. We don't have anything that like gets us to reproduce that. But if we walk through the code and we're like, okay, like these are the conditions that would result in in getting us there, like basically all that we can do if we have nothing physical to reproduce it with is like to create tests to say, "Hey, look," and you could do like a red green approach, right?

like you write the test and you say yes, if you have these conditions being passed in here, this will throw the exception or result in, you know, this this code path. And so that's us trying to simulate the behavior. If we fix it, hey, look, now it does this other thing. Great. So now we have a test built in. And someone might say like depending on the situation, well that's dumb. like you know never test implementation details and like whatever you did for that test is implementation details and I would go back to my argument earlier I would say hey look we're trying to build confidence so like pardon my language I don't give a if it's implementation details because I want to have some kind of confidence that I'm able to go back to a customer and say look we've done our best here

we're as confident as we can be and we believe that your issue is solved Now if you said to me should you go test all the implementation details and only the implementation details I'd say probably not but also like that's not exactly what would give me confidence personally. So I believe that you need a set of tests of different types generally speaking and I think that ultimately the balance of like this assortment of tests is going to come down to how you want to build confidence. Okay, that's like the most generic way I can put my sort of framing around this. And so to me, like I don't think that's a hot take because basically what I'm saying is like do the that makes sense to you. And like I I kind of feel like the word that we often use for that is like something like pragmatic.

Like I'm not about to prescribe you the you know the one simple trick that doctors don't want you to hear about that cures all of your testing issues. Just like be pragmatic about what you're trying to do. And the gauge for that is confidence. So, with my rambling rant out of the way on that, um, I have lived through all sorts of different, uh, I don't know, like life cycles of testing where, you know, going from like, hey, we're shipping a product that an intern builds on his desktop and copies files to a share that we upload and customers download with no tests. I've gone from that to hey it's the same thing the intern's still doing it but we have some tests um you know basically going from absolutely nothing to to having something and on you know fresh code bases to like legacy

where there wasn't tests and how do we add them in um I kind of called out like not a lot of web testing but uh do have some experience back to this Reddit topic with end to end. So I want to I think I might have told this story on code commute before but it's relevant here. So um I had where I was working before Microsoft. I'd been there for 8 years before I moved to Microsoft and so I was there early and so also uh fortunately and unfortunately responsible for a lot of what you would call legacy code that was still there, right? And I would very much like point at it and say like that's not good code. I wrote that. It's for me to be able to say it's not good.

And so when I had moved on to different parts of the product and there were other teams and newer team members that were sort of working in those areas, uh they kind of had to they were living with some, you know, some nasty code that that really wasn't tested well. And that's again like I I take responsibility for that. So when they're trying to clean it up, so they have all this terrible code that you know I wrote and they're like, "Well, we got to make this better." The problem with it when it's not tested is like, "How do you clean it up and not break everything if you have no idea how to know if you've broken something?" Right? It's kind of like covering your eyes and like uh mashing keys on the keyboard and hoping you didn't screw anything up. You're going to screw something up and it's like it's not your fault.

It's like it's my faults for not having tests in place. So, we wanted to try coming up with a creative way to solve this problem. And we knew that we couldn't solve or cover um maybe I'll use the word we couldn't assert like every behavior that existed because how we were going to do this was with uh endtoend UI automation. And so for a little bit of context, this was a product that you could uh basically point at uh like let's just say like to keep it simple, you'd point at like a hard drive and you'd basically run a search engine on this hard drive looking for things. So you might say, well, why the heck would you run endtoend UI tests on that? Great question. Um because we did have tests that would exercise the engine thankfully.

And so these are big regression suites that would run these search engines on on data and make sure that essentially we're always finding the same things or as the team is improving things we're finding new or more um rich sets of data. Okay. So have regression suites like that that exercises the engine and a lot of plugins, but it's not exercising any of the user interface. And so this was really the part that the the team wanted to be able to improve. They can't touch it without risking breaking all of it. So we were saying, hey look, if we can start by putting some end toend UI tests in place, if we can get to the point where you basically say start this search, we don't have to actually I'm saying end to end. We don't actually have to go run the search for you. Why?

Because there's literally tests that already do that. If we can get to the point where there's basically the output of the user interface, which is a contract that says here is how you will go search things. If we can get you to that point, you're already proving the search can take that contract and go find things. You're already doing that. So, as long as we can get you that contract, we're basically covering all the way up to that point reliably. So, let's write some tests like that. So we did um and so it wasn't uh with playright this is because we had a desktop application and so we were using I think it was called white it was like the white framework and it was modeled after something else and I can't remember what unfortunately but it was cool um you know it it would

let us basically write tests where we could say like you know look for this button like kind of like you might do with any other UI automation, but we could do it with our desktop software and it was cool. So, we did this. Um, now the tricky part is because if I'm telling you that this team wants to go refactor a bunch of stuff and rewrite a bunch of it is like at what level in the code are they doing that? Because if they're going to they want to go change the user interface like well hey wait you can't change the user interface cuz our tests are all pointing at very specific things in the user interface. We're saying go look for a button name this and like click it. So don't touch that. Right? That's not the part you're going to refactor. The part you're going to refactor is the stuff below that.

Of course it's not always that cleancut. So we wrote a bunch of these UI tests that would click through and for the most part I would say like that was that was successful. I would say that worked well. So it gave them enough confidence over these happy happy paths and sort of common edge cases. Uh so like hey a user might do this when they shouldn't like let's make sure we we capture that. There's always going to be like crazy edge cases but let's make sure we got common scenarios covered. That way they could start changing the guts underneath and make sure that at least the common cases weren't breaking. So again, back to confidence. Is this 100% confidence? Absolutely not. Is this at least knowing that you didn't break like I'm going to make up a stat here, but like you're not breaking like the 80% of use case path.

Like cool. Okay. Hey, if you have coverage like that, maybe that gives you enough confidence maybe. So, we could start that way. And so, that's what happened. And then people could start rewriting some of the code underneath. That way, as they're rewriting the code underneath, they can make that testable. They can add tests. And the idea was that once they have all of that and they want to go tackle uh the UI on top of that, that's where you might have to start saying, "Cool, you're touching the UI. At least you're not going to break the stuff underneath. That's now well tested, right?" So, at what point do you start swapping out some of these UI tests for uh maybe more more generalized ones? because maybe some of the UI tests, like if you're looking for a specific button um and you need to change that button, that's like a bit of an implementation detail about the UI.

So, how do you start changing these so that you have the coverage you need so you can make updates safely? Now, like I said, I think this ended up working quite well because people could refactor underneath. But I think this is where, you know, the the happy happy story starts to turn a little bit sad because what ended up happening was that people were like, "Hey, look, we can write these UI tests and they cover so much." And so when I talk about testing, I often use two words and try not to use them to mean the same thing. Sometimes they do mean the same thing, but I try to give them different meaning. exercise and assert. Okay, so to me a UI test that's covering a lot of ground like clicking through a user interface almost doing like an end to end type of thing.

To me that is exercising a lot. There is a lot of stuff that is executing underneath that being interacted with. Right? The more things that you are exercising, I feel like the less you should be asserting on or the less strict What is with people pulling off the road today? Holy Crazy. um the more that you're exercising, the less that you're going to assert on or the less strict your assertions are. And I say that because the more things that you're exercising, if you are very strict in your assertions and or asserting a lot of things, this is where things become brittle because it means that you're checking so many things across such a wide surface area that if genuinely you made a change that was affecting one part of the test and it breaks like you know 90% of it downstream doesn't even get exercise.

Now, did you just lose all of the coverage on that because that was really the only thing that was exercising that code? It It couples too much together. So, I use these terms because it's like just because something is being exercised does not mean that it's being asserted on for correctness. If you go to the exact opposite end of the spectrum, you have something like a like what I would call a true unit test. It's on a very small piece of code that's not exercising a lot, right? Like not much. It's like ideally it's like something like a method and a scenario through a method. And like what is it asserting? Maybe some very very specific things. when I talk about scenarios where I am testing implementation details, it's a very very narrow uh bit of code that's being exercised and a very hypersp specific uh like detail that I am testing.

So I'm asserting something very specific. And so when I again testing pyramid wise or thinking about um end to end tests like versus unit tests to me those are at opposite ends. You you can absolutely assert on tons of stuff if you want in your end toend UI tests. I just don't think it's going to be great. So we ended up having this experience where people saw like, hey, look, we can write these end to-end tests and it can exercise a lot. So if it can exercise a lot, you know what that means? We don't have to write as many of these stupid other tests because if we never wrote our code to be testable, we still don't need to. We don't need to write testable code. We can just put these UI tests on top. Now we don't have to spend all this time rewriting our code to be testable or change anything about what we're doing.

By the way, I'm speaking facitiously not to make fun of anyone that was part of this. I'm sort of just doing it to exaggerate um the circumstances. And really what ended up happening was a ton of UI tests got built, a ton of them, right? And they're exercising a ton of code. This is to so far not bad, right? Then the problem is well we needed to assert on a bunch of specific things. Okay. So these become more brittle now because they're asserting a lot and exercising a lot at the same time. Now what happens is you have a big test that might take say a test takes like a couple of minutes to go through something or even if it's on the order of like many seconds and you're running a bunch of them it's slow. Like something like a unit test is like milliseconds.

So you have something that's big and bulky that is giving you the signal that something's busted and because it's touching a lot, it might not even be an assertion that's breaking. It might just be that the test itself is brutal and can't even complete. But even if it is an assertion, it's like, you know, uh was looking for 10 things in the in the list view and got 11. Okay. Um, so you know what's broken like the symptom up to you now. Go figure out what's wrong. So the challenge ends up being when you only have that one level of test. It doesn't give you the granularity to go pinpoint when there's issues. Right? There's nothing else in your testing stack to be able to say, "Hey, cool." Like, you know, um, sure, this UI test broke, but, you know, we also see there's a thousand unit tests and only this one unit test failed.

Like, maybe it's maybe it's close to that thing. Um, obviously, it's not a perfect uh example cuz you're not guaranteed to have like code coverage that works that way. My point is that without having things overlapping in terms of different types of tests, I think that you can run into a lot of uh just a lot of issues in like debugability um tests being flaky and brittle like this person on Reddit was saying. So I often encourage people to think about like again how do you build confidence if you have these different testing layers? What types of things are you checking? Do you have an agreement with your team around like you know if you have endto-end tests like really how much are you asserting right in my opinion the more things that you're exercising the less you know uh specific you want to be

about your assertions um and maybe you have different opinions on that right and like that's cool cuz I am not I'm not here to tell you what's right or wrong I'm trying to help you think about frameworks for the stuff that work for you. Like this is the fun part. If you completely disagree with the things I'm saying, that's totally fine. Um my goal is just to get you thinking about things. So to give you one as I get closer to home, like total counter example to all of this because I think this is the one that maybe is a u more of a hot take. Once upon a time um I was running a team and we were focused on mobile acquisitions. So my team is the team that built the software that would take the data off of your phone. So um after having lived through some painful times where we didn't have tests, it was like nope, we're building tests.

We're going to build them, right? Um I'm air quotes right. Uh so on this team, we actually took an approach where we unit tested like basically everything. And when I say unit test, I mean like every every like thing that we were writing was basically like a lot of testing implementation details like to an extreme. So, I'm saying this because I know like that really bothers people, right? Um, shouldn't test implementation details, test behaviors. We tested the out of implementation details. Every boundary was mocked and we test the out of it. And interestingly, this actually worked very well for us. And I'm not sitting here saying, "Look, there's proof like test implementation details. It works. Everyone should do it." Nope. I actually don't recommend that.

Uh it's funny because like in hindsight very interesting experiment because it did work um well for us but I think one of the reasons it worked well is because we were a very small team and everyone that was on the team was basically you know living and breathing in that code base. So why did that maybe work well for us? Well, I think for us it meant that at least based on the type of software we had, when something was broken, this was the cool part. When something was broken, we could pinpoint exactly where it was in the code. And how we did that was it was the spot that didn't have tests on it. So, we needed one more element to this which I left out, which is telemetry. So, let me explain this very briefly. um we had instrumented a lot of telemetry and we had a ton of unit tests.

And so what would happen is that we could see if there were um either exceptions or error cases being hit in telemetry and we had enough information from that to go hey look like let me look at the stack trace or like we know it's going down this error path. Um so like we know where that is in the code. Let's go look. And sure enough, like I know it sounds like I'm like exaggerating this, and I guess to a degree there's there must be a bit of exaggeration, but the way it felt was that we could literally open up our analytics dashboard, look for like error cases or exceptions, go cool, like what's at the top of the list? Okay, stack trace from here. Go look at the code. And we'd say, okay, how does that happen? Cool. Let me go look at the test for it.

Oh, we we just don't have a test case that went over this. There's like 30 other tests here. And here's one variant that we didn't do. Cool. Like, let's go add a test. And you could red green this, too, right? Let's go add the test that proves the the failing behavior. Fix it. Test goes green. Cool. Ship it. And we could literally sit there refreshing a dashboard looking for errors like that. Not that we were not that we were untest or shipping untested code on purpose. It was just like the escapes that we happened to miss. And I think why this was so cool for us was that we had like the group of us had worked on different parts of the product area uh sorry product suite let's say in different areas where you know we get these customer reports and it's like basically good luck figuring this out.

There's no there's no data that can repro this. We went from that to uh one of our like super awesome support guys would come over and he'd be like, "Hey, got a customer who uh was asking about, you know, a crash or exception or something not working in in this scenario." And we'd be like, "Oh, is it like blah blah blah?" And he'd be like, "Yep, like that's the that's what's happening." We're like, "Cool. Yeah, we uh we fixed that earlier this morning." like it was the coolest thing to be able to like actually get ahead of the support requests because we had telemetry and the way that we had tested just worked really well for us.

Now the argument would be or one of the arguments is if you had different types of tests like hey Nick you said you unit tested everything and implementation details and I think that's stupid by the way I also don't think that's the best way to do it that's what we did um you should have had functional tests and integration tests and whatever um mutation tests like have all these different things like and that would have helped that would have been even better and like yeah maybe so I'm not sitting here saying that we picked the best thing what I'm saying is that with telemetry and we were able to survive quite well with very much like a pure unit test it worked really well for us would I repeat that the exact same way no I think telemetry was a big part of it and

then I would probably have I don't know definitely some unit tests definitely having some implementation details tested because there was some there's some really weird we have to write uh when it came to some of the recovery stuff. So I would definitely test some implementation details to make sure my head was wrapped around it. Um and yeah, a lot more functional integration test for sure. But just interesting story. It worked well for us. So that's my take on some testing stuff. Um, yeah, I would highly recommend doing more than just only end to end UI tests, but if it works for you, it works for you. So, um, yeah, if you got questions, leave them below in the comments. Otherwise, go to codemute.com. You can submit stuff anonymously that way. And you can check out my other channels. I got the main Devleer channel. There's a Devleer podcast where I do live streams.

And then just I'm on social media, so find me there. Happy to hear from you. See you later.