Initial Impressions Using GPT 5.6 Sol For Coding

Transcript is auto-generated and may contain errors.

Hey folks, going to go through a little bit of AI discussion again this morning. Um, few things I'll talk about. One of them is going to be recent model usage for me. Uh, what that sort of uh has been looking like cuz it's I think I'm a little divided after I've spent some more time. Uh, and I think we'll try to get into some some of the stuff I was making progress on over the weekend. It's going to be related to the last video or two that I've been blabbing about anyway, but uh we'll get into it. So, um more recently, I mean, depending on when you watch this, it's going to change it a lot, but uh GPT uh 5.6 Soul, I've been using that a lot. Um my initial impressions with this were just uh just super super impressed. Um was using Opus 4.8 eight pretty extensively before that.

Like I think once I had switched over to Opus 4 um and kind of stuck on the Opus models, especially just having like a really large context window that was truly like just really really impressive. Um, and then going like finally getting over to this uh back to the GPT models on on Soul. Um, yeah, just I'm pretty pretty blown away with it. And uh I think one of the first things I was trying to explain to my wife who's like she uses chat GPT and uh I don't think pays much attention to this stuff or like cares much but like finds chat GPT very helpful. Um, I was trying to explain to her like models and and why like some of them behave different and just like at a high level and had mentioned that uh when I switched to to this sole model

that what I what I notic is compared to other ones like sometimes models will when you're coding like just maybe go off the rails a little bit and they're touching stuff like touching files where you end up reviewing the code and you're like why were you even like what was even going on here? And you know, if you ask it, it's kind of like, oh yeah, like oops, I guess I'm just being stupid and should have shouldn't have done that and like whatever. And it's just like it's wasteful and it's wrong or it's uh it's just yeah, I don't know. Sometimes creates a bit of a mess. And so my first interactions with soul that I was telling my wife was that when it was going seemingly off the rails and touching things, um it was actually like stuff that was broken like it it

happened to run tests or something and then like noticed that some of the literal test scenarios were just that it that weren't related to the code it was changing were wrong and it fixed them. So when I was reviewing code and going why was this touched, it was actually like these tests were broken or um you know there was a behavior that was like that was wrong. So like yes the test passed but like based on the underlying code like that's not actually right like some just some weird stuff. So like it caught a few bugs in test code and caught a few bugs like uh in the underlying code for unrelated features. And so for me that's probably the first example um like concrete example where I was like holy crap like it was going off the rails a little bit and it was beneficial.

And so I've been using soul for a little bit now. been a full week, maybe a little I don't know if it's been a week or two, something like that. Not we're not saying like super extensive in terms of uh it's been, you know, 10 years of using the same model, but um but yeah, I think that has been uh mostly on track. closely. Now, how much of this is the model versus the harness for this next part I'm going to talk about, I don't know. But, uh, I have noticed that, um, that some of the work that I've been giving it is taking like a pretty pretty long time. Like, it's pretty significant. And I noticed that it's doing like I think this is where the harness comes in, but I think also the model has some some interesting like sub agent capabilities if I read things properly.

But anyway, um, so using Copilot CLI for a lot of the work that I'm doing and when it's spinning up like review agents and things like that, I I notice that the time that it's taking to to work through something is is pretty extreme sometimes. um like instead of taking even for like a I don't know decent size feature like I don't know 30 minutes on something it's like several hours and so there's been a couple of times where I'm like hey wait what the heck like what's going on with this and then go look and expecting it to be stuck like something silly happened with the harness where it's spun up a sub agent that was just like dead but it's still waiting on it kind of Um, and like it's just still cranking through and then I realized like it's it's just like solving review issues like you know talked to the rubber duck or did a code review and got things to fix.

And I'm like, how how are we doing this like 3 hours into something like just to give you an example, I had uh co-pilot sessions running last night and there was nothing that I had assigned before I went to sleep that was like a monumental coding effort. And when I was turning off my alarm this morning, I noticed that on uh on GitHub, I had notifications where there was a PR that went up, you know, 50 minutes before I woke up. And I'm thinking I actually didn't get a lot of sleep last night, unfortunately. But um that would have been another like I don't know like 5 hours of work on a feature. Like what what do you what do you mean you took five hours to go deliver that? Like what were you doing for five hours? So I have some questions.

Um but I think I've noticed this kind of pattern and I think what's uh not simple for me is that uh I mean there's a lot of things not simple but in this case there's I'm running a few co-pilot sessions so I'm not like paying extreme attention to the work that's going on while it's happening. And so is it I don't know is it like just the the model responses are slower because things are overwhelmed. Is it um you know getting API errors and retrying? So like there's some overhead from that. Uh the thing that I seem to be noticing is that it's doing more review loops which is itself not like a bad thing but like only if they're useful. Um because what I have noticed in the past is that if you've used co-pilot and like I I think Claude has the same thing.

It's not exactly the code review agent, but if you have a like a rubber duck agent where it's kind of giving you almost like playing devil's advocate or like trying to look at things from a different angle, um I noticed that can like balloon the scope of things and it's not always like right, but it's it's kind of like AI convincing itself because it's so confident. I think most of us have experienced this where AI says something and you're like, "Well, that's very believable." Like, you might not know better because you don't have the the data to prove otherwise, but it's very convincing. And I think like the rubber duck agent ends up doing that to to uh to the the main agent. And so I've seen like again anecdotally it seems like it ends up kind of spinning and just like ballooning the scope like oh I should go get this next and it's like you if you just would have stopped we could have delivered this kind of thing.

Um now like the more that you're touching the more of a rabbit hole we're going down. So, I've noticed more of this kind of thing. And um yeah, I don't know. That might just be the harness. That might be a combination of the harness and and the model. Don't know. But that's been kind of frustrating. I think um one of the last videos I was talking about how I wanted to use Eve from uh Verscell. And so, um, when I'm building stuff in brand ghost, I'm usually kind of in like two two major modes. One is like I'm just prototyping stuff where whatever I'm doing is probably going to be throwaway, but I I just want to see like is something feasible. So, um, doesn't really matter what's being built, how it's being built. Let let me just see if the functionality is possible. Can I can I glue these things together?

can I use this service? Like is this a possible thing? And then the other flavor of that is that I'm doing the exact opposite which is like I'm hardening things and so um in this case with Versel and Eve um we we saw the initial opportunity where it's like hey this works like this functionally does what we want. And so from my perspective I'm like I don't need more prototype on that. like I don't have other business questions I need answered around feasibility like that seems like it's going to do what I need. Let me let me get it to a point where I'm like this is going to be operationalized. Um and so I mean for me a lot of the time that means like testing right like I need to get regression tests in place. I need to make sure that this is something I can exercise repeatedly.

And so I was doing some of that work and again I I had caught this model just spinning. Um there was a point like on on Sunday like on on Sunday I almost got nothing delivered on it despite working on it most of the day. And that's because several times what I thought were like incremental steps, I was asking it to do something um getting like myself unfortunately getting distracted doing things and then checking back and being like, "Okay, like it's sorry I'm so late on this, but like let me let me pick it up." And I'm like, "It's still going." And then when I see what it's doing, I'm like, "No, like this is completely uh like you should have stopped hours ago and we would have been okay." And the only reason you're still working is because you added more scope that should not have been done.

Um like one example is last night, um and this is for like a an evaluation hardness. Uh, it was writing a custom markdown parser and I'm like, what what's going on here? Feel like I like came home to my house and like kick down the door and there's just like I don't know, like animals walking around talking like humans or something. And I'm like, what's going on in here? And they're all like, uh, nothing. uh something ridiculous like that. It just it's crazy. I don't like I don't know what it how it got there. How did we go from like I need you to to make a couple of skill changes to you should be implementing a a hand rolled markdown parser and like of all the things why why would you build that? No. Um, so anyway, yeah, I've been having some weird kind of experiences with it, but um, part of me is like I think I just have to pay closer attention.

Um, and I think just the last note on that, I think one of the the other things that is a bit of a pain in the butt. And I again don't know if this is harness or model, but um, it's a a million token context window. And whoa, buddy, that is just happened. This person just like you you couldn't see it cuz the camera is the wrong way, but on the offramp they just slowed down. There's no one in front. They slowed down, stopped on the on-ramp and then like pulled over a little bit, not even onto the shoulder. Dead stop. Nothing in front. So, I went around them and then when I looked back, they just came back onto the road and kept driving, but they never fully pulled over. They just kind of stopped on the I don't know. Weird. Now, my car is beeping at me and there's nothing in front of me.

So, uh yeah, harness or model, I don't know, but it's a million token context window. And I'm seeing that there are times where uh this is getting stuck for a different reason. And it's not like it's stuck in a loop. It stops. Um, and it's basically like it has filled up its context window apparently um without auto compacting and everything's just like failing. So, it seems like it happens sometimes between like 65 to 75% of the context window. And when I say it's so stuck, I mean that I feel like it should have done a a compaction earlier. And if I try to compact, it still fails. And so what seems to fix it is restarting the session. And what I mean by that is like in Copilot, like actually typing slashre. So it closes co-pilot, reopens the same session, then I can type slashcompact.

So, I feel like that I don't know how like exactly what's happening, but I feel like it's probably um the harness, but at the same time, I've never experienced that with any other model. So, I don't know why there's still space left in the context window and it's failing to compact because it's like hitting the limit like at 25 to 30% remaining. Obviously, we don't want to be operating like at when it's super full, but at the same time, like you that's want to compact. Like that's the whole point of me compacting. So, that's been kind of weird. Um, because there's been times where I again I I check a co-pilot session and uh it's not that it's looping kind of expanding scope, it's just that it's dead. So, cool. Maybe I spent too much time on that specific model stuff, but uh yeah, that's been on my mind.

Um, I'm going to when I stream tonight, so on Mondays at 700 p.m. Pacific, uh, on my one YouTube channel, it's a Dev Leader podcast. It's not the main channel. It's not Code Commute either. It's Dev Leader podcast. I live stream. And, um, this time I'm going to live stream, uh, I'm just going to show like how I how I work, um, with with Copilot. And so, uh, I'm actually going to start that an hour earlier and go for two hours. But my goal with this is that there's been more and more conversations at work about like you know using loop engineering and uh you know we'll build features with AI and um kind of try to try to change like I don't know like the standard way that we're used to working to do what a lot of places are doing. and um and h and how do we how do we do that effectively, right?

How do we make sure that we're not just I don't know being uh reckless with it, I guess, is maybe reckless is the word. Um reckless, naive, I don't know. Um and so I think that's good that the conversations are happening. I think it's good that people are trying to move in this direction cuz I think this is, you know, uh the path forward, but uh in terms of some of the conversations, I'm kind of feeling like I think I'm think I've been doing this for for months. Um and because I haven't written code in in months, but I've been building tons. So, I figured I'm gonna I'm just gonna maybe show people like this is how I've been working. And I don't know, like we'll see. Um, I still feel like there's a huge spread of like how people are using AI tools.

And not that I'm saying I'm doing it right and everyone else is doing it wrong or something, but um I feel like if there's still people that aren't finding some of these opportunities, maybe it could be helpful to go through. So, I gotta fix my camera. There we go. Um, thanks for watching. If you got questions on software engineering, career development, send them on in. Uh, you can write that in the comments or go to codecame.com and I'm happy to try making video response. Take care. See you in the next one.