How And Why To Test Software (And All The Reasons That We Didn't)

Transcript is auto-generated and may contain errors.

all right we're headed to the office it's Wednesday morning um I napped a little longer than I was hoping to after CrossFit um I am apparently like super run down and I didn't really realize like uh so I got to pay more attention to that but like definitely noticed uh man this guy's flying um I I remember like leaving CrossFit this morning and being like yeah like you know not feeling great I got a nap like for sure but I figured okay if I get like nap till 8:00 a.m. I got an hour of productivity before I leave for the office and stuff and like no um I woke up from that nap and I was like dude you are nowhere near ready to go here so um kind of sucks um but I got to listen to my body right I I've talked about

this before from like a productivity perspective it like I don't know it's uh I get momentum when I have productivity like I feels good and then I want to keep doing more um this guy on a bike is ruining this guy's day um the uh the reality is though like that kind of will push me to a point where like I'm not paying attention to certain things like I like admittedly have a problem with uh regulating that kind of stuff so I fully believe right now I'm kind of at a point where I'm just like you don't have more capacity and like I need to listen to my body so um I don't like it but it's like it's what it is man so yeah I didn't I got nothing done and I'll I'll explain why that bothers me okay so from 5:00 a.m. when I woke up till 9:00 a.m.

that's a 4our period the the only thing that I accomplished in that period two things like I ate a meal and I uh and I went to CrossFit and that's a 4-Hour period and that really bothers me because those two things are like I don't know like an hour and 15 minutes of stuff so like what's going on with the other hours there right so anyway enough of that I just wanted to explain kind of my state of mind right now I do feel very rested right now which is great um so as I'm headed to the office that's a good feel not like oh man I'm still like falling asleep or I'm grumpy cuz I'm tired like I feel great um so you know I should try to remind myself like uh yeah you didn't get anything done but but you don't feel like

a bag of crap right now and probably going to have a pretty good day so Silver Lining I guess um yeah so I wanted to talk I have two topics I want to go through today um one was just from perusing LinkedIn that's going to be more career focused uh so I'm going to talk about that one in the afternoon video when I drive home and uh unless something comes up at work where I'm like oh man this would be a cool thing to like reflect on and and and talk about but the other one I wanted to talk about was testing and I know that uh probably some people just heard testing and they're like great next video um but hear me out cuz I think it's kind of interesting um especially like literally especially if you don't like testing it's probably helpful um

so I wanted to chat through this and I'm going to start by giving a little bit of background not too much though cuz I don't want to put you to sleep more than you're already being put to sleep by how much I ramble but uh the way that uh we the guys that I'm working with on brand ghost the way that we operate uh in general is like we do strongly believe in tests um we're not like Fanatics about like oh 100% test coverage but we've lived through periods together where like uh like seeing like live through the pain of not having tests and like seeing how brittle things are and then and then be like man we're spending more time just like you know you add a feature you fix one thing and you break two more and you're not even moving forward anymore

and that's like a really scary spot to be from like a a productivity and a ability to deliver perspective and when I've talked about like being agile before like for us it's like the focus is about delivering value so if we are able to deliver value and constantly do it and or doing it without a ton of tests or any tests we will keep doing it but what happens is that inevitably um even as much as we'd like to be coding things perfectly like especially for something like brand ghost or it's evolving right so there's like some core pieces like the concepts are core but uh I'll give you a quick example the the back end for brand ghost is pretty complicated in in how it uh can template things so if you were trying to think about you're a content creator and you want

to make a bunch of posts you might structure them in a similar way so you might have like for yourself you might say I need to have a hook on my post like a oneliner to kind of open it up then there's like a Core Body uh and that's going to be something you can figure per post and then you might have like something like a sign off right where you can say like um you know follow me here or uh you might say like uh you know here's where my newsletter is or something like that so you might have something that's kind of consistent and then you might want want to say like okay all the hashtags that I have I also want to place them at the bottom of my content and don't read into this too much if you don't like hashtags

whatever my point is that uh you might want to have your post templated like this now the most simple case of templating is that you just have a onetoone mapping there's a body of a post and you put the text in there so that's still like templated but it's just a onetoone mapping so our system supports like especially in the back and supports um basically like macro Replacements where you can have text defined by macros and before we post for you we'll go do all these Replacements so you can have um this guy's swerving all over the place holy um and now he's behind me he was like in the left he was going into oncoming traffic and now he's behind me and now he's stopping oh my goodness get me out of here um the so our our templating in the back end is

is pretty complicated but we realized even before like getting too far like even with one user we were like it's the ux for it is way too much like it's going to be so hard to explain to people how to use it um is there value in it like I think 100% yes because like 90% of what I post online follows some type of templating and macro system that's replacing the macros in the template so uh I think there's a ton of value in it I just think that the experience is very complicated so back to testing though what what's happening because of this is that like the the flow of data through our front end to back end is changing it's evolving because we're trying to say okay wait like uh which things should still be configurable which things are like there's no macros

anymore from the perspective of the front end so you can't do like intext Replacements so we're hiding that for now not a huge deal um the back end is still fully able to support it so when you think about these boundaries for where these types of things can exist the back end remained unchanged it's like hey man if you want to give me stuff that's templated I will totally do it and the front end is like we will just not even provide that type of data to the back end we won't present it to the user won't go to the back end it's like it doesn't exist so that's fine um it makes it weird when I use brand ghost because my stuff is templated so some things will break for me right now and that's okay because uh I I have all the tools

to do what I need to do uh without worrying about it but it's not going to matter for our users or it shouldn't which is part of the point here so um we don't have macros anymore uh the the fields that you need to populate they are like we still treat them as like a macro but it's one to one so there's no special template the template itself is just literally um a onetoone mapping so that's cool but then we have this other like concept of like different types of posts on social media platforms can have additional fields and one very simple example example is like on LinkedIn when you post what's called a carousel post it's a PDF okay and um it makes it look like Instagram where you can swipe through pictures but it's just a PDF so linkedin's Carousel is a document

where as Instagram's carousel there other media you can swipe through but regardless on uh on LinkedIn when you're dealing with carousels when you upload a PDF you you can give it a title so that's like a title field that just like doesn't exist on text post or image post it doesn't exist on Instagram or Facebook it's like it's a thing that you can do with LinkedIn for documents so like the way that we originally had a lot of this stuff was like if it's a field that can be populated we will try to find a way to let the user interface let users populate it but again uh going back to like the macros being complic at this idea of having these other fields was also really complicated so trim these things out so what I'm getting at is like we built this system to

have all this functionality and then very quickly we're trying to trim down people's access to it just because it's too complicated to start are we going to put it back 100% yes do we know how we're going to do that yet like 10% yes um so we don't know how to do it in an intuitive way but we already and I'm not making this up we like turned it all off and then started onboarding people and almost right away we had people going hey the the title for LinkedIn documents is a perfect example had someone going hey I want to be able to set the title and we're like oh man like we just turned all this stuff off and now they're asking for it um but anyway the point is that when you do this kind of thing your code base is like is

changing and evolving right and when you have flux in a code base it's a perfect opportunity for things to get weird and to go wrong so I talked about this a little bit before but especially in the backend code I started getting frustrated because that's where I spend my time and I'm like man I'm having a lot of stupid bugs like that they should just not happen like the kinds of bugs where they're embarrassing like I would break squl queries because I would be missing a comma like what come on man like there there's no excuse for it um but at one point like near the beginning I turned on functional tests and had like containers that would stand up and stuff and then we were doing Aspire stuff and that wasn't compatible right away so I turned them off but anyway I reached a

point and I was like nope I got to turn them on so fixed up like some very minor stuff that I thought was a Spire related and it's not even it's just I broke some other things around the same time in the testing framework so now we can write tests in the back end and we can do things like stand up a a database um one of the previous Vlogs I was talking about was I need to find a I really want to try this I just don't have time um I want to try I'll explain it very briefly but when we go to run tests in parallel uh they will stand up their own Docker instances now that's good for isolation but it's really terrible for performance especially when we get into like thousands of tests and it tries to kick off like a

100 Docker containers in parallel yes there are things you can do to limit concurrency no they're not consistent between visual studio and running them on the command line it's a total pain in the butt uh limiting concurrency would just solve this problem entirely but it's just not consistent uh unless things have changed in like the past year um so anyway one way that I can limit concurrency or work around this problem actually is like when I'm writing tests a lot of the time the IDS are randomized so because they're randomized there aren't going to be collisions on things and if I'm really paranoid about hey like this scenario like I need to make sure that I have an empty database I could you know keep it the same way for that test but otherwise I want the default Behavior to be like you share the

test database because the whole all of the patterns in the code for tests rate now are unique identifiers you're not going to get collisions um the only things that can get weird are like if you're like hey I'm querying this date range and I only I only inserted one thing there in my test but some other tests like inserted 100 other things like you just have to be conscious of that um and again if you're really paranoid and you need to have the isolation great like stand up your own container but for the majority case I wanted to share the database instance um when we did this with meal coach we had so many tests uh they were super helpful but the big problem was that it was having performance problems because of what I said and run the test in parallel it would go

stand up like literally it would go try to launch like a 100 Docker containers on my computer and I'm like nope like my computer would just stop and then the test wouldn't finish like everything would break because the docker containers are all messed up and um yeah it's just like then it then you can't run the tests other you then you go run them sequentially and now they go take forever because still has to go do all the work so anyway um I want to fix that up but I it's like we don't have enough tests where that's an issue yet so it's more of a curiosity but now what we're hitting is like some stuff in the front end where like we we absolutely need tests on it and um it's not because like someone working in the front end is like bad at

coding or anything like that it's just back to this idea of flux in the code base and when things are changing that much in terms of like the assumptions we make about data flow and how users are going to interact with things what types of data can exist like when those assumptions change like without tests it makes it really tricky to know you have the right Behavior so becomes kind of interesting because well we're going to want to get these things hooked up to our build system which is a whole other conversation um man this guy in front of me we're in the fast lane this guy in front of me keeps hitting the brakes but we were going around a corner and I could see there's literally no one in front of him so I don't know what he's breaking for hopefully he's exiting

here yeah nice there we go I feel better um so we're we have like uh test containers an xunit in the back end um I don't know what we're going to be using in the front end I think before we were using jest like J um by the way we have like a nextjs front end it's not Blaze or anything like that uh for the guys that I'm working with uh I think it's just probably probably find it easier to work in react and stuff which is totally fine um I can navigate it but I need to have like the patterns and stuff set up like from scratch I'm garbage with uh front end stuff so just I don't spend time doing it so I don't know what they want to use for front end testing Tech probably just unless they've come up with something

in their travel since then since a meal coach times and uh the other thing I can't remember what it was called oh it's called packed like p a c um we we talked about it like early on with meal coach but we never got around to doing it and um I might butcher this explanation but the idea behind packed is that we were supposed to be able to have like these functional tests that literally glue our front and back in together which is pretty cool so you should be able to I can't remember exactly how it works so I don't want to try to explain it and get it wrong but I will tell you what I think it is with the uh disclaimer that it's probably 100% not what I'm saying so I think that what it had to do was that you can

run your test separately so you don't run like it's not like you run the front end and the back end at the same time and they talk to each other um I think it was more like you run a tool that will basically create the uh like the Json payloads and stuff that you'll be sending and receiving and stuff and that way as long as you run the tool it generates those things and then you have tests on either side as long as that communication's met then you're good to go I think I think that's what it is now when I run like on meal coach and I'll probably do the same thing shortly for brand ghost um there's I've explained before different videos and stuff like I really like unit testing like really granular stuff with mocks and like treating everything as very isolated

systems because it helps me prove to myself the code I wrote is doing what I expect that Paradigm I can carry over over and if I have to mock out external systems I don't use fakes I don't use stubs I don't use anything else except mocks for this kind of thing because I want full control in a test of a third- party system because I want to say like when I call this thing what happens if it throws an exception like how am I going to respond to that if you're trying to use a real thing like can you make can you forcefully make it throw an exception and then see how you react to that like probably not um so instead of like just pretending I'm like look I'm going to have full control over this thing people will say oh well how do

you know you're making the right assumptions about the happy path and stuff and like I can't but I can make a lot of assumptions about when things are going to break right so the the time where that really screws you over is if you have something like a third party system and they change their behavior Behavior like over time and they don't announce it right so if you have these types of tests and you're mocking them out you have data that's coming back and it's always been this way for years and then all of a sudden they make a change and the real system is now it's not Json anymore it's XML as an example it's a really bad example but like the the API calls is still the same thing right like then you get an XML response back and it breaks like that's

the kind of scenario that we're not covered by purely from doing that but then okay it it breaks and then you go add a a test for it so whatever but that's not I'm trying to say like that's how I like covering things but it's not I'm not trying to say that it's the best and especially when we have things like test containers now um I think there are many opportunities to be able to stand up tests run them quickly uh like totally functional tests with more complicated systems like launch whole Docker containers right and it's still fast and repeatable so I still write code in a way that can be tested and mocked out just because that's how I write code and I actually spend more time from a testing perspective on like these higher level like functional tests because I find that now

with the tools and stuff we have available I can get a lot more coverage it's a lot more coarse right so if something breaks in the functional test like okay now it's between so many layers and modules of code but at least like at least it could all get exercised which is very valuable and at this stage and I've talked about this I've made videos on this on YouTube uh in more of like a tutorial fashion where I explained with meal coach literally the evolution of how we had to test and what what came up was like the functional test and just offering tons of really coarse coverage was so helpful because like like a lot of code was being exercised and then along the way like there's so many opportunities for things to break over time that it would just happen to like you

know by by just having a wider net you would see more things it meant that when they broke you'd have to be like okay well what the heck I actually broke but at least you know so I went on a bit of a tangent there but some of the tests that I want to start writing with brand ghost are literally from calling the web API so what I've been doing for the most part is like so far in brand ghost is doing like a level below that so forget that I'm have to have to make web requests like whether it's a get or a post or what the Json payload looks like that's like a pretty thin layer of of brand ghost in the back end at least and then below that then we start to have the business logic so I've been writing like

functional tests on that that'll exercise that it will write to the database it'll read stuff back so it's still really good coverage like through the snack but like as we start adding and changing features like what's not getting any coverage right now is at the API level um we' had we've only had a couple of things like this so far where we're like oh crap we didn't like we didn't talk about renaming something or whatever I got to explain what just happened so we were merging onto the highway and there was like six cars in front of me and uh we were probably going 20 and no one was trying to move over in all the other lanes so uh I don't want to get stuck behind people like that like you're it's it's actually dangerous you're merging onto a highway there's no other cars

around yet and you're going like 20 and we're about to approach a bunch of spots where people are merging on like you're just you can't be merging onto a highway at 20 M an hour it it doesn't work it's not safe um someone's going to come around a corner flying and you have six cars going 20 M an hour for no reason no I'm not getting stuck behind that um so at the API level for us like we haven't had too many issues between the front and and the back end but I I would like to get ahead of that I think um so what we were able to do with meal coach and I'll do the same with brand ghost is that the functional test instead of just having the database stood up they will literally launch an instance of the backend so they'll

have a backend running which includes the database and then they can actually make web requests to it and that tests like the entire thing top to bottom which is super cool uh to the point where like the calling part has to parse like a Json response right so it's not just you called it and like check the database or something it's like you called it you can check the database you can uh check the the Json response you get like you you can it's the full deal and again like that's a pretty coarse level test cuz you're not going to write like a million assertions on something like that um but it covers a lot of modules and again I think that's really powerful so as I'm coming up to getting off the highway here I just wanted to kind of maybe summarize some of

the like The Meta points that I was hoping kind of came through and I realize like always it's all over the place but um code flux like change in code uh the more frequent it is and this is a very common thing in the beginning of products uh especially right when they're still maturing and forming and trying to figure out what you're building that's an opportunity for a lot of code flux and as a result um a lot of scenarios breaking um I don't think that it's like like we should it's unexpected like we should probably expect that kind of thing it's a lot of churn so that's one thing um the other thing is like finding I my opinion on this kind of thing is like finding when it makes sense to be writing tests so I kind of talked earlier about we were

doing pretty good without tests like adding tons of value without tests and moving fast and like being very productive right and so like should we have added test like maybe but also I was telling you that we were changing a lot of the actual Behavior so if we would have spent I'm just making this up I don't have numbers just to give you an example but if we spent twice as long because we were adding like testing infrastructure and getting these tests and stuff in place and we delivered half of what we did and then we just went and turned off half the features in the UI like that's literally a waste of time I'm not trying to say tests are bad please don't hear it that way but like we would have we would have spent way more time and then actually just turned

it off and no one even would have used it so it's not even like we helped ourselves so I think for us it's about being pragmatic about when we start investing into testing and I say that again as someone that really values testing it's just that I also really value shipping value so um I think for us something that I like to I I say this kind of phrase and I know it sounds kind of weird but uh I'll often say like living through the pain is a good indicator of where you need to be investing into things so one of the takeaways here is like as we started to progress with brand ghost like we're living through some of the pain where we're like man like why are we having these simple bugs like this is so embarrassing and it's like yeah like we're

probably at the point now where some things are solidifying and they really need tests on them because they're starting to become core if they were still in flux and changing rapidly it would be like a pretty difficult um decision to say like how much do we want to invest into that but some parts are like it's critical if it's not working it's the whole app is busted and useless right so um I think that's one takeaway that might be it words of wisdom there's something else I wanted to say about living through the pain um it's unrelated to the testing thing but I like I like talking about this because I think it's interesting but um in terms of like Automation and stuff I think um sometimes people like as programmers we really like to automate things so like hey we did it once like

let's just automate it uh but uh I think until you until you've kind of struggled with something and you're like man this sucks to do like I think it's important to live through a period where you're like yep like I don't know um the way we build things the way we uh mitigate incidents the way we solve and recover services like all all that kind of stuff like yeah in theory you want it to be super automatic but like what do you even need to automate it's an open-ended question I mean like if you have actually felt the pain of where things are crappy how do you know where how to spend your time effectively so I I like saying like Hey look it's not that we're going to stop but like we got to live through it a little bit to understand like why

it's crappy understand where the pain points are and that also tells us the priority because if there's a hundred things to do to make it better what was the most painful part like maybe we should start there if the most painful part is also one of the things it's maybe potentially easier or shorter to go make a fix for get that covered get that automated get that out of the way right like I think it's a good thing for helping prioritize anyway um yeah I like saying you got to kind of live through some of the pain to find out where to spend the time on things but hope that was interesting uh this afternoon if I don't come up with a better topic I am going to be discussing uh interviews because my video with Jade Wilson who was visiting from the UK Microsoft

employee she's a very big content creator on LinkedIn um we did a little interview about um interviews and super cool so uh shout out to her for for putting that together I really enjoyed it she uh was helping us and Ryan Murphy like promote our course that we just launched so it was uh very kind of her uh I appreciated it and I really enjoyed the conversation so on LinkedIn Ryan shared the post that Jade had put up and uh another fellow content creator that we really like um actually like Ryan and and this individual were kind of going back and forth and having a disagreement which is I think it's good to have disagreements and they're not like these types of things like I know these two guys like each other so I think that's super cool to be able to have a debate

like that um and I wanted to kind of chime in on my perspective on it because uh I think that some of the wording that was used is conflating things so I'll talk about that more in the afternoon unless something totally juicy at work comes up and I'm like hey this is a great meta thing to talk about but we'll see so I will see you later