IT BACKFIRED! Saving Dates And Times As UTC Wrecked Me

Transcript is auto-generated and may contain errors.

what's up it's uh Monday morning I'm just leaving CrossFit I'm in this uh crappy X1 rental because our X5 finally has uh the air conditioning unit come in they said it might have taken up to a year which just kind of ridiculous but it came in it was only a couple months perfect time for winter but uh this SUV sucks um it's an X1 I thought x1's were relatively newish um everything about this thing is God awful um sorry I'm really out of breath the uh there's no reverse camera so I had a big truck just parked beside me right there I had no choice but to like just start backing out and uh hoping that there's no no traffic coming because he just can't see anything either that or I just wait for him to go um but he just parked at this store

that I was parked at so pretty crappy um got a couple things I want to talk about today I'm going to reserve the uh the raner conversation for the drive to work CU it probably needs a a little bit more time um I figured I'd use the morning drive here cuz it's a little shorter back from Cross spit to talk about kind of what's been going on and just a little status update so um Step One is Catch My Breath and know this uh this weekend was good we uh in brand ghost we had this uh this bug where um our our scheduling I think I described this on one of the previous videos our scheduling of recurring posts is anchored to UTC time which is in I don't know in my entire time is a software developer I have only ever seen anchoring things

and storing them relative to UTC um and then you kind of like translate to a user's time zone right before putting it into a the user interface I've always seen it this way it's always worked and this is different because you do want it anchored to the user's time zone and the reason for that is when they switch um their time zone enters like daylight savings time or goes out of it they expect their post to go out at the same time that they normally would and because we anchor to UTC it will shift by an hour so this is currently broken in brand ghost uh in in the production site but I sent it an update to users like your their post and stuff will still go out it's not catastrophic but everything depending on your time zone uh will be off by an

hour so the UK folks were already affected by this because their time changes the week before uh users in Pakistan should not be affected at all because I believe they don't have Daylight Savings Time Lucky them and then a bunch of folks in the Americas are impacted but again it's an hour the very easy fix that I explained to you is like hey just open up the app and then just go to where your your top extremes are and people only have a few of these and basically just change the time again and that'll basically buy you a window of time until the next Daylight Savings Time change but the good news is I do have a fix for this um and I wrote a newsletter article on this but it's a little bit more complicated to roll out than just a fix and I'll

explain what I did how it's been successful um so far and then kind of what we have to do going forward because I think this is um like the newsletter I wrote about it touches on why like real world software development has all sorts of interesting constraints that you don't get exposed to especially if you're just doing like lead code style things um don't get me wrong like time and place for that especially if you're practicing for interview rviews um it's not what I would recommend people spend all their time doing because you're not getting a real world exposure um now even if you're building like little projects and when I think that's way more valuable than doing lead code uh just for for practice overall but um the the closer it is to being like real world I'm not saying you need to have

paid for users that's a that's practicing business stuff um sorry these mirrors are everything about this thing man like the mirrors if I look into the mirror it's like it shows me what's beside the car but um even to my left I have a blind spot like I can't trust the mirror uh so that's why I keep like looking left and right aggressively cuz I can't see what's going on but yeah the closer stuff is to real world the more interesting constraints that you have to practice with and this time zone thing is a perfect example Because the actual fix to make the system compatible with a configurable Time Zone per user is very simple um within a couple of hours of identifying The Challenge and working through the problem I had it locally on my machine fixed and functionally tested with scenarios across different

time zones different times of the year and uh and spanning these different oper or different moments where you would gain or lose hours from daylight savings time so and that's not I'm not trying to say like it's it's because I'm so good I'm saying it's because like when you don't have constraints like how is this going to work for real users like you can solve problems much easier there's a like there's fewer challenges to look through so again it's not bad to practice this way but the more realistic the challenges are the better practice should also mention I'm completely out of water and I can feel like my mouth is like almost not working so it's going to be a struggle okay the uh the fix like I said uh I talked to the other Vel opers we didn't just want to roll it out

because the Schuler in our system is like one of the most critical pieces when theuer is not working literally no one's post go out when the scheduler is uh operating incorrectly we risk double posting or missing posts like it's uh it's arguably right now the most critical part of our entire system uh next to just the database itself and like we don't want to lose people's content which by the way is backed up daily so um we have uh historical records of it like it's there should unless unless all of azure goes down or there's a something at Microsoft or whatever that's an issue there's no issue with losing people's content so what we decided to do is instead of just saying hey let's turn on this new Schuler we said well why don't we run both we'll keep the old one running so that

no one's rupted and we'll run the new one in like a log mode and it's pretty cool because this way we can uh in our logging we can check like what did the first scheduler pick up and what did the second one pick up and every time they fire we should see that they have the exact same behavior and that's because by default no one has time zones configured so the default behavior is that everyone is anchored to UTC even with the new scheduler so it is a really cool experiment right you turn both on one in logging only mode the other one actually wired up and the results that you should see in the logs is that the schedulers have identical Behavior so we did this and we ran it for 24 hours and everything was perfect okay next step let's have Nick that's

me let's have Nick actually set up his time zone configuration man this person doesn't have their lights on I've mentioned this on videos before but people driving on the highway when it's dark out without lights is batshit crazy to me it's so dangerous it's so dangerous I can't WB my head around it but so I set on the second day I set a couple of my topic streams not all of them so the point of me explaining this is like that we're staging seeing how we can get this uh rolled out so I set up a couple of my topic streams um to be allowed through on the new Schuler and I set my time zone so the old Schuler will pick up all of my topic streams like normal but it will ignore the couple that I've switched and the new one will only

pick up the couple that I've switched on and we ran this and it worked so we were able to prove okay using the new Schuler it's not just that it's logging properly but we can truly wire it up and it will post and do the correct thing based on the time zones now there was a bit of a hiccup the first time through this and it was pretty scary because I said I configured two topic streams to Poe this way the first one did not fire and I was out at Costco with my wife and then I like looked at my phone and I was like oh it's time to check to make sure it worked didn't work so I'm sitting there going okay either my sort of like test code to like go hook this up had a bug or a worse problem and

that worse problem is that we're using time zones in squel what I mean by that is like the name of the time zone now this is something new for me so cuz like I said it's always been anchored to UTC never had a reason to work with time zone names like this sorry I'm falling apart right now that car is loud um so the crazy thing is like this all worked locally I was running MySQL and a VM on my machine works but when I checked uh and ran the query manually against production uh it's saying that there's no results for the schedule when it's supposed to be firing and it turns out that the the behavior by default for an Azure SQL instance is to not load time zones I don't know why that's the case I'm sure there's a very good reason but

basically if you tried to do any operations and do a time zone conversion the convert time zone method would just return null in SQL these guys got me boxed in you got to hurry up buddy you got to hurry up this is crazy there's a huge truck beside me oh my God sorry I need to think so I can't speak this is also the problem with I should try to explain this briefly when you're used to driving like cars that are a little bit zippier so the AMG GT is pretty Zippy um when you're used to driving a car like that like I don't drive like an idiot I don't like to speed aggressively uh I get uncomfortable but I do like to be able to accelerate and the reason that I like to be able to accelerate is one it's fun but two if

I need to get somewhere if I'm looking where I want to be on the road all that I have to do is make sure it's clear and then I know that the car is able to do it in a vehicle like this I can see an opening I can put my foot down and then I have this gamble of like can the vehicle even do it and that's not a position I want to be in I just want to be like I know the vehicle can let's go so in a moment like that where I only have a you know like 10 seconds to go do a lane change to get off the highway and I'm in the far left I don't want to be thinking about that okay so time zones Azure sequel default they're not there fortunately there's like a oneline command that

you can run that loads them up glad I found that because if that wouldn't have worked we'd be in a pretty bad spot because um the the challenge is like I could do time zone convergence based on the offset from UTC like using a number but that will not help us with daylight savings time we need the logic to be able to support a given time zone and support how it's supposed to translate based on daylight savings time so um again very fortunate that I found that command loaded them up everything started working which brings me to the next phase which is uh I think we can turn on the newer by default because we've proven that it just works which is cool um but because I am the only person with a time zone set I need to adjust all of my top extremes

to be in that time zone and that should be a good next test so I think maybe this evening we will roll that out which is cool and then beyond that now we have this uh interesting migration path right so um at this point in time any single user that registers like literally right this moment the time I'm recording this if they register and they make topic streams they will be anchored to UTC everything will be fine until daylight savings time which means the more users we're on boarding which we want to keep doing the more sort of surface area this problem has because every single user that's on boarding at this point will be anchored to UTC so what we need to do is start getting time from users when they register making that their default time zone right so if I'm in Pacific

time and I register we would want to at least have that be the default at least because anchoring stuff to the time zone I'm in will at least be explainable later if someone was like hey wait no like I was between Pacific and Eastern for work travel like sure no problem just switch your time zone now that we put it into the app to be Eastern and everything will just work but the challenge is we just don't have that setting in the user interface we just don't have that concept yet so we need to start building that out and uh and snapshotting users default time zones and that way if they're like hey look like hey uh brand ghost devs um it's posting at the wrong time and we can say hey look no like check your your configured time zone in the settings and

then they can say oh like you know when I registered I was visiting Seattle I was in Pacific time uh but I'm actually from Central so we would say yep just change your your you know uh preferred post your target posting time zone into being like Central and everything will just be fine so we need a migration strategy in the the user interface for that and then ideally some automation but I might have to do it by hand to migrate users um like posting times because the data's got to get migrated right like in the database it's literally anchored to UTC um so something we could do is like we could introduce this time zone setting for all the users and put it to UTC and then if they're like hey it's posting at the wrong time I would have the same conversation with them

where I say um like yeah it's because it's anchored to UTC like you want it to post in um in eastern like no problem just change that setting where you see UTC to being Eastern and it'll fix so we'll have to do that maybe communicate it to the users but fortunately right now the user base is small enough and tighten knit enough that um like this is embarrassing it's awkward but like we got to do it it's kind of like you you do it or you give up right like so we got to move forward and we'll do it so um if people can't deal with that like I don't know what else we can do is like we just have to be able to offer my opinion we have to be transparent right um I've already Comm like I've sent out communication of the

users telling them this weekend if you have a time change expect that you're going to be an hour off I gave them an update saying we have you know code to fix this I tried to explain it's going to be a roll out strategy and we'll make it right but um unfortunately this is life and this is a complexity that we have to deal with in the system so um it's not trivial and you know I appreciate people being patient a lot of people are on their trials the products in its early stages so there's bugs in every software I'm not trying to make excuses for it but like we're going to make it right um people have been really good about giving us feedback whether it's a typo or something they they don't even expect to happen or they want something or it's truly

just not functioning um so all that we can do is in my opinion is keep listening to customers and keep being transparent what I would hate to do is like just pretend like there's no scheduling issue and then have this influx of people being like hey like why is this posting at a weird time now uh we don't want that like the first person that brought it to us was in the UK and it's because their time change was a week before us and they said hey I'm noticing like they I think they said something didn't post and then an hour later they're like hey it posted but an hour late and then it was like wait a second time change right um that's what sparked all of this so like it's good we have really good users giving very clear feedback and we don't

want to lose that so that's why I think it's important that we remain transparent with them um I think like one of the things if I if I may say so because it feels like based on the feedback this is true I think one of the things that we are doing very well is we react very fast so for example um John vanir who runs latterly IO as a content creator um he he did like a 50 minute live stream of like I haven't watched the whole thing uh one of our developers did um he did like a 50-minute live stream where he was onboarding to Brand ghost entering like his content and he was sending me messages being like hey like there's a typo here hey like what about this what about that like this stuff is awesome like that's what we want and

um this is over the weekend and people like Dev devs are you know spending time with their family and stuff it's the weekend at their Cottages like as you might expect they're they have a life to live too um no one's doing this fulltime for now and uh and we still still at like you know around midnight Eastern we're still Landing changes Landing fixes so like when you give us feedback we are reacting to it immediately we're at least talking about it as soon as it comes in and I think that when we can demonstrate that to users they know we give a so we'll keep that up um next video on the drive to work will be a rant um if you like rants stay tuned for that I'll see you next time