In this video, I talk through some of the things I am building with agents -- including some games and agent tools.
📄 Auto-Generated Transcript ▾
Transcript is auto-generated and may contain errors.
All right, folks. It says it's an hour to work. I sure hope not, but we're going to see how we do here. So, going to do a little AI recap after the weekend. Talk through some of the stuff I've been building, how that's going, things I'm learning, what's working well, what's not. And so, to kick things off for a little bit of background, I'm working on a handful of different things at the same time. Um, no, I don't necessarily think having a ton of different things to work on is a great um way to focus and make like dedicated progress, but I also think that there's uh a lot of benefit to be trying to explore with different things. So, um, the handful of projects that I'm sort of working on outside of work are number one, Brand Ghost. So, Brand Ghost is the the platform that I create for posting content to social media and then helping businesses with their with their blogs and essentially trying to automate a lot of their marketing process.
So, I spent a lot of time on that. That's the primary focus. And then the couple of other things that I have going on on the side are two games that I'm putting together. They are entirely built uh through AI. and I'll talk uh more about those in this video. And then there's a couple of other projects that I'm doing and they're paused a little bit because I was doing a bit of a um AI token or I guess AI credit like rebudget calculation. We're just over halfway through the month now. I think it's the 19th today. Uh so actually we're in the sort of last stretch here and I wanted to make sure that with my setup uh I'm not going to blow through my budget but the two or I guess the couple of other things I'll I'll talk about uh briefly are
I have uh uh how do I call it like there's it's a combination of like an open source thing and then like a hosted portal for uh agentto agent communication and then my uh SQL workbench that I have that has AI providers built into it for helping you with your your SQL. So, a lot of different topics to go through. Hopefully, this drive isn't a full hour. Um, let's get into it. So, first thing to talk about, I guess, is the AI budgeting. So, uh, I mentioned recently that as a Microsoft employee, I do get, uh, sort of like AI, uh, credit. There's too many too many words that mean almost the same thing. Uh, it's not dollars, it's not tokens, it's credits. Um, so I have AI credits for co-pilot usage. Uh, and those routinely get dialed back. So, um, I guess I didn't even realize, but this month it was cut down in half again from what it was last month.
Um, so good thing I checked. But, uh, I think what's super interesting is that last month at this time, uh, I had already sort of blown through that budget that was doubled. So I had gone to basically start using claw code. I had mentioned and if you've watched other videos and stuff like I'm not I'm not like picking sides here. I'm like oh clawed code versus copilot CLI one is absolutely better. They're they're similar but different tools. Right. I think that what I've been saying is that because I've been using C-Pilot CLI more regularly. I think a lot of my development flow is more naturally suited for that. Like I just happen to to work better with it, but that's not because I think it's a superior product necessarily just for total transparency. And so I started paying for the uh Clawude Promax plan so that I could kind of load balance that way.
I could go through the uh credits that are subsidized for me and then as well as the having my ProMax plan from Claude and just kind of spread the work out. Uh I was got going down the path of like, oh my god, am I going to have to go buy like, you know, like a huge rig and stuff? And I'm like I probably I think that's on the horizon. Uh I don't think in the immediate future. I think my wife would be pretty upset with me and especially because I think I found a balance. It's working, but I do imagine that that's going to be on the horizon. And when I do that, I'll make sure to video it and stuff and uh keep people up to date on what that's looking like. So I think if I reflect on the biggest changes since last month, right?
Biggest changes since last month are um I think one around models and two around a couple of workflow oh like I guess like one real workflow change on the model usage. Uh basically, uh I I just went to use I think I was using GBD56 for a while and then when Astra came out was using that. So obviously using Astra is going to be outrageously aggressive for token consumption. Uh high reasoning or I think I was picking the highest and then um like a the million token context window. So, I just switched over this month basically sticking to uh Soul 6.1 and I started the first few days at least with the the 900k token context window. Um I feel like they changed it within the first week or two or something. I don't know. Something's different cuz I thought the the two options were a two or sorry, a 400k context window and a a million.
And I think now it's like 260K and 900K. But anyway, uh I use the the smaller context window instead of the bigger one in medium reasoning. And I did this not because I was like, "Oh, I'm going to optimize my budget." I was like I I felt like whatever I had selected was going to be fine anyway and we'd see how it would go. But um the original reason for this is because I was having these outrageously slow um agent interactions. So I would ask and I had a couple of co-pilot windows running. So nothing unusual and then I would check in on the work and I'm like it's still not like anywhere where it needs like near where it needs to be. It's so far behind in making progress. And I think one day I was waiting like 6 hours to get like a pretty simple PR up and I was like something something's wrong.
Like it's not like I'm checking in on it. It's not oh Nick you had really dumb CI stuff going on or the agents like you know stuck going down a rabbit hole. It's literally like scrolling through the chat. It's just taking ages. So, I used another co-pilot session and said, "Go look at the slow session." And then I realized the few sessions I was running, they were all going super slow. And when I had co-pilot do a bit of an analysis on it, again, I'm not talking like super thorough, but just like where where is time being spent, it wasn't on tool calls. So it was like hey when it does go run a tool call it's completing within like you know depending on the tool we're talking like sometimes milliseconds other times like couple seconds but it was saying that some of the roundtrip time to go uh you know get the LLM to answer was taking like you know over 5 minutes or something crazy um repeatedly.
So, it was going so so slow. And so, I didn't know what to do about this. Like, I'm like, "Okay, like everything just sucks, I guess." And I I had this total coincidence where for a particular session, I needed to restart it. And when I did, I I could visually notice that it was just going faster. And I was like, "Hold on, something's weird here." Like is it because I cleared the context and like I did a a compact and then restarted like which basically I I did too many things at once so I didn't know what to isolate. And it turns out it wasn't compaction cuz I I thought oh maybe when the context windows getting too big that's why I brought it back down from a million to something smaller. It wasn't that. It wasn't the reasoning level. So, I thought maybe from being uh from high or very high down to medium that would help.
And it like those things help, but they're not the root cause. Um the and I still don't know what the root cause was. But if I restarted the session, right, restart the session, basically just have it resume. For whatever reason, all of a sudden the the actual interactions where I'm making the API call were suddenly fast again. or when I say fast, they're what they should be. They weren't outrageously slow. So, um, my point here is that I accidentally ended up optimizing some of my token usage because I was messing around trying to figure out what variable to do here. And the really weird thing is after I don't know like a week or so of having this like really shitty interaction where I'm constantly just like you know uh every little I don't know every few hours or something every couple hours I'm just restarting sessions so they they stop being slow.
It it just kind of went away. I don't like I don't know what actually changed but the API responses started being normal. So, um, the result of that is that I've kind of stuck with a smaller context window on medium reasoning, and that's been working really well from a token efficiency. Like I said, I'm I'm under half my budget, and the budget was cut in half. So, pretty cool. I am also using cloud code in parallel. And again, I'm going to move on from this topic in just a second. But the other thing that I think was helpful was that uh maybe due to the the reasoning level being cranked down, but I have fewer I have fewer sort of long running co-pilot activities. And when I say longunning, I mean like there's some work I was doing where I'd let it crank through uh a bunch of stuff for like a couple of days, right?
Unattended. Just like, "Hey, go work through this. I will check in on it after that kind of thing. Um there's just less things like that. So I'm those were absolutely wasteful and like I'm not I'm not saying that like I'm shocked. I'm saying that because that was something I was trying. I think it served its purpose. It worked. But I think from a an efficiency perspective, it's just it's simply not worth it. um you know if you have unlimited tokens and stuff then sure like do do whatever you want I guess but uh not the case. So that's all good and well on budget. Um sucks that you know things are clamping down more but at the same time I'll stay on top of that. Um the I guess that's the reason I have not been uh working on some of the other side projects I have.
So the one that is the agent to agent uh encrypted secure communication and hosted platform for that and then the the SQL database um workbench. Um, given where we are at in the month and the amount of tokens I have remaining, uh, I probably will go spin that up for a little bit. See if I can just make some progress. Um, the workbench was absolutely working. I think I was just working on some extra features for it, but I think it's actually at a point where like people could be using it. So, um I will probably shift my my focus on that one from like let's crank out more features to like let's just get usage. I think that will be a fun uh direction to pivot that to and then obviously as people are using it I can prioritize the right features for the agent to agent communication.
This is actually one that I built for myself primarily um because I keep running into this situation where I need like co-pilot uh sessions to talk to each other whether that's on the same machine or across machines. And this is going to sound funny, but I I get to this point where I have these sessions running and I'll say like, you know, I say I have uh one working on front-end code, one working on uh a backend service and another working on a different backend service. Because these pieces have to talk to each other. I have noticed that if I have my agents working across the repos, it's not that they can't get stuff done, but they seem to be just a little bit more dumb when they're touching the other repo. Uh I have this feeling that uh some of the context just isn't being loaded up the way that I expect when that's happening.
And that's fine. But a pattern that works really well for me is that I have um the agents of each session I will say like hey go review you know the PR for for this that this other agent was doing because basically you have to interface with that right and so one of the things I was doing was I'd be like go check this out and tell me what you think and then basically I'm copying and pasting stuff between the agents and then I'm like well this is dumb. So then I have them actually like leaving comments and stuff on on GitHub, but then I'm like still telling them, hey, like you need to go, you know, go review this and then go respond to the comment. Like it just I'm now doing other boring that I don't want to be doing, right? Like now I've traded writing code for like I'm copying and pasting messages between agents.
Like this is outrageous. No, thank you. So, I wanted to build um this conclave application that would allow me to have uh sort of like agent chats set up. And so that's uh that's open source. So that will just be available. People can use it. And then I'm going to offer a paid for platform for hosting it with some uh other features like a dashboard and things like that. So sort of like a hosted managed platform and then the open source variant. I want to I want to get that a little bit further because I got it to the point where it could be used but it wasn't um what's a good way to say it? Like it could operate but it didn't feel like it was um like the user experience is bad or I guess you might call it like Devx, right? As a developer the experience was kind of crappy.
So I had just got it to that stage and I was like cool I need to try using it. And then because I was tightening up my budgets, I'm like, ah, I can't really make much progress. So, I want to get that a little further. So, those are two that I'll try to pick up a little bit more in the end of this month here. And then the other two are um games. And so, I have talked about this a little bit. I've showed it on live stream a little bit. And then I just started tweeting a little bit more um for the second one. And so the two games are essentially a like a creature breeder caretaker kind of uh casual gameplay loop. And the other one is a role playing game. And so the role playing game is something that I've been sort of building and designing for for years and years and years.
Like one of the first things I ever started programming, but it's never a finished product. It's always like a sort of a playground for me to to to build an experiment. And so I wanted to use AI to kind of bring this role playing game world like to life now for real. Um, and so so far it's still serving as a bit of a playground for using AI to build things. So I'll touch on this a little bit more. Um, and just to talk about the creature breeder and the status of that. Uh, that's been going pretty well. Um the I think the the thing that this is proving out for me is that it is entirely possible for me to have this is just a pair of agents working. One's doing art and other assets. The other one's doing like the code and the gameplay loop.
It's not that neither uh are like sorry either are forbidden from touching the other. It's just how the work is automatically split up and they can file work for each other like on the on the GitHub repo and stuff like that. So if one is doing a feature and they're like, "Oh crap, I need art assets for this." Like they'll just file the the ask. And so this has been a really cool like process because um it's been like there's a lot of autonomy that's put in and I and I shifted this along the way because I wanted to get to it to be more autonomous. Um but you know in the beginning that's certainly not how it was. So a lot of stuff went in up front and um now it it operates mostly autonomously but the way this works is that I give it a general direction.
So, for example, on the gameplay mechanics, I'm like, "Hey, you know, there's this set of features and stuff that we talked about and like go work through these, right?" Like, I'm letting you make the decisions on this stuff. And so, I let it progress and then periodically, whether that's every, you know, depending if it's the weekend, maybe it's a couple of times a day, if it's during the week, maybe it's every couple of days, I have it give me a demo. And so, I go try the game out. And so this applies for the the art as well, but um basically I try it out and then I'm like, "Okay, like this part actually feels good. This is silly. This is clearly broken." And then instead of just saying like, "Hey, fix this." Like here's issue 1 2 3 fix them. Um it's do that, but also like what is the pattern here?
Like why is this kind of problem happening? Right? for example, um there was some mechanics where the the creatures themselves like they have um there's like a genetic system, but then there's also like a I'm going to use the word AI, but like not not like an LLM AI, but there's like the creature has a brain and it needs to go do things to satisfy needs. And so the creatures would basically just go to the food dispensers and keep blasting food. Like that's all they did. So, you'd go to play this game and um all that would happen is that the creatures would go use the food dispenser until it ran out of food and then that's the game. It's done. Um so it was it was not a good game, right? Like the actual experience was bad. So I had to be like, "Hey, like the creatures have to do other things, right?
Like we talked about there there's supposed to be like toys where they can go have fun or there's supposed to be things like other critters and things in the game world, flora and fauna that they can interact with and be curious. So after a little bit I was like hey like why like we talked about this why isn't that built? So then it was a matter of shifting its focus to go build out those systems. As a result, it files a lot of asks on the art agent to go make the assets and animations for that. So, um, kind of perpetuates the, uh, autonomous part here. Is this not my exit? No way. What are you, what are you doing, Google Maps? Um, and then the other thing on the art agent is I get the updates a little bit more frequently. So, every few hours I have it give me some screenshots about how it's um doing the artwork and the models.
And uh I have that one more frequent updates like that because the um the whole art side is is such like a I don't know a a thing that I'm not good at. I'm not experienced at it. I'm not skilled at it. So I need to like I need to learn more about how to direct it better. So I also need more feedback opportunities, right? So I need to also shift my thinking from like, hey, this model sucks, make it better. This texture sucks, make it better. To like, well, what is it about the model that sucks? What is it about the texture? I don't even have the vocabulary for a lot of these things to to tell it. So like it's also just more opportunity for me to kind of get better in that domain without having to learn like how to go do it myself if that makes sense.
So I need to understand it better so I can communicate my asks better and so there's more frequent updates that way. Um but that's progressing. One of the more recent things is that I only had a couple of base creatures that you could basically breed and raise and um so introducing some more bases for that. And the other thing was that I noticed it was kind of neglecting this ask I had which was that when you're breeding the creatures in this genetic system like these traits are supposed to be transferable between the creatures, right? Right? So, if you have a creature that has a a fur coat and it is blue, right? Like maybe that's the only blue creature you've ever seen and your creatures are green.
So, if you were to get the blue creature and breed it with your green one, like maybe you get uh a coat, now all of a sudden your creatures are able to have the blue coats or maybe you get like a yellow coat because they mixed or you get uh patterns on the coats because you have the two colors showing up. Like it had just neglected this concept, which is basically like one of the core game mechanics. you breed the things and then you have these uh you know uh genetic traits that are able to pass around. Otherwise, the game loop is kind of uh a lot more boring and like why would you why would you breed them if uh there isn't something that happens as a result. So, um it's now focused on that. It was showing me examples of like cool like here we can make these creatures in all these different colors and different patterns.
Um, so kind of showing me that the ARC system is capable of it and that will then have to make sure that it's wired up with the the actual gameplay side as well. So, that one's progressing. It's pretty cool. Um, the other thing I'll talk about is the RPG. Um, so for context, I have decided that like one of the things that I really wanted to do with my RPG is I wanted to have different form factors. And this is going to sound kind of weird, but like I love 3D like isometric style action role playing games, right? It's like my my favorite genre, but I also realize like that's a that's a complicated thing to build. It's a complicated thing to get that looking like how I would want it to look. Uh, but I love, you know, the systems and mechanics that go into that kind of stuff.
The thing that's a lot easier for me to build is something like a, I don't know, like a Pokemon battler, right? like I can focus on the game mechanics and just have uh you know really simple assets, but I can have a I could take something like a Pokemon battler and and build a lot of like cool game systems into it. And then I I realized like one of the things that I I want to be able to do is like not is not actually just pick one, which sounds kind of funny, but I think there's a way that I could have my game experiences across these different form factors, which means more things to go build. But I actually think that it would be a really cool experience. So, um, I'm still going to build the 3D action role playing game, but as a stepping stone, I'm going to build one of the companion apps for it first.
And that companion app is a turnbased tactical RPG. They will use the same game mechanics. And it will actually allow you to bring your characters between the two so that you can actually just as an example you'd play the action role playing game like on your computer and then if you were like I don't know on the go or whatever you could pull up your phone and you could use your characters or use the same game world and play a different variant. So it's not the same literally the same game like you just are multiplatform. it would be now you can go play with your your game world and experience like a turnbased tactical RPG. So, you know, you can find loot, you can level up characters, and then you can move those resources and stuff uh to the other game formats. And there's a couple of these that I have planned.
So, I'm starting with the the tactical RPG. And um another cool side effect of this is that I get to to play with different um like text stacks and game engines. So this is actually going to be entirely in 3JS. Uh which is pretty cool. I thought that I was going to have to do something really silly with the art and assets, but what was cool about going through this process so far, this one's already been uh at least to make it more playable. It's only been a couple of days now. Um, but in the beginning it was already telling me like, hey, no, like, you know, we'll still use Blender. We'll still get, you know, real real things put together. And the cool part about that is that I'm hoping I can reuse a lot of a lot of that, if not all of it, in the the 3D role playing game.
Um, I I thought that I was going to have a lot of waste that way, but being able to reuse the art is going to be awesome. So, I started tweeting the other day uh just showing like the first set of assets it was introducing and as expected completely ridiculous. It looked like it was ignoring my instructions to like not make the assets from scratch because the first skeleton it made was like uh I'm actually better than I could make still, but like that's not going in a game. Like there's no way. Um, and then you know within a few hours it was already providing better skeletons. And then within a few more hours it had some characters that were kind of shoddy but like hey it's a starting point.
And then when I woke up this morning, uh I had told it before going to bed, hey look, like I see that you're tuning a lot of the art and assets, but the problem is that you're doing these oneoff things like tuning like putting a lot of time and effort into like something that is like I don't know um the stitching on a glove. And I'm like th like those gloves are not going to be the final thing in the game. Like there's no there's no way. So, you're spending an inordinate amount of time on on that doesn't matter. So, I said, "What I need you to understand is that for a lot of this art, I actually need like a pipeline put together, and that pipeline needs to allow me to go from concept art because there's a lot of really good like AI art generation now to get concepts, right?
So you could go to chat GPT and say like you know make me a whatever like and it's a fantasy role playing game right so you can describe things and it's going to come up with really cool So you and you can use offline models for this too which I do. So I can generate concepts and for me that's really helpful because I can say cool like this concept like I can I can see this being like part of the game world or other things where I'm like that's that's the it's not a fit and then I can give the concepts over to the LLM and say hey now I want you to take the things from this concept and I want you to start you know making these into into game assets. So again that's easier said than done. I certainly don't know how I would tackle that uh engineering problem, but I going back a few months, I did have some of this going with inventory icons.
Same concept, right? Generate a bunch of concept art. I had like a call it like a structured templated kind of language for a lack of better word where I could have layers of different themes that I want. So, if I wanted to make swords, right, I could say like layer on the weapon uh terminology, layer on the sword terminology. Um, and then I want it to be themed to be like fiery and I want it to be like some material type as well. So, I could layer on these different things, generate a bunch of concept art, and then have it go from that concept art that I that I sort of pick and say like these ones and have it make inventory icons. And that was working like not perfectly, but pretty well. It was cool.
So I told Cop or Claude last night instead of spending so much time on on this, you know, that's not going to matter like pour your energy into this thing so that we have a repeatable concept art to game asset pipeline, right? It's not going to be fully programmatic. It needs AI like handoff in the middle. Um, but go do that. And so when I woke up this morning, I had uh screenshots that I had created of the character models uh no longer wearing like the the assets it got from like Blender Kit or wherever else. They were wearing like armor that it made. It literally took some of the concept art screenshots that I had given it from chat GBT and like it showed a couple of steps how it was taking like aspects of that and then and then it made wearable armor and like is it good?
Not not really. It's not great. It's not horrific though. Like it actually made like wearable chest plate uh which is super cool. So I now can get it to focus on like make this pipeline better, right? Like and ideally if it does it the way that I want, if it were to enhance like I don't know a lot of the foundational pieces, I could ask it to go regenerate things, right? So, if all of the things that it generated were anchored in the wrong spot or had like silly texture problems, I could say, "Hey, go, you know, go through these assets, regenerate them through the pipeline, ideally as deterministically as possible." But I get that it can't be perfect. Um, regenerate them and then I should hopefully wholesale have improvements across the assets. Um, I realize like as someone who's not an art person, um, what I am saying might sound like, you know, maybe that's it can't be done or whatever, but like that's what I'm trying to do because I want cool art.
I suck at it and I think this would be a really awesome way that I can get my creative juices flowing and have um assets that I I can look at and not like hate to look at. So, we'll see how that goes. But, um, I got to play the demo, uh, got to play the playable demo today. Um, it's 3D. I can move the camera around. Uh, the character animates when it's walking on the on the map. Um, there's no attack animations, like it kind of hesitates or something. But yeah, like you can battle two skeletons, you can get the loot, which is cool because I put a lot of time already into like these backend systems. like there's a whole loot generation thing. So yeah, it's it's working. Um I'll have both of these games being put together by Claude. And I've been saying I need to make some uh dev leader YouTube videos.
I think what I'm going to do is uh start doing like dev diaries on these and kind of talk through from the AI side of things, like what's going on. um and and kind of show how these games come together because I need some accountability for getting them finished. I think that would be super fun and that might be some good content. So, that's my AI update. Thank you for watching. And of course, if you're new to the channel and you don't know already, um this channel is generally about submitting questions that I can answer for you. So, if you have questions about software engineering, career development, leave them below in the comments. Otherwise, you can go to codecute.com. You can go to the contact page. You can submit stuff fully anonymously that way and I'm happy to make a video response trying to help you out.
So, thanks for watching and I'll see you in the next one. Take care.
Frequently Asked Questions
These Q&A summaries are AI-generated from the video transcript and may not reflect my exact wording. Watch the video for the full context.
- How do you manage your AI budgeting across Copilot and Claude, and what changes did you make this month?
- As a Microsoft employee I get AI credits for Copilot usage, but they routinely get dialed back and this month they were cut in half. To balance the budget, I started paying for Claude's ProMax plan so I could load balance the usage and the subsidized credits. I also switched from higher-context window settings to a smaller context window to improve token efficiency.
- What is Conclave and how does it help your AI agents communicate?
- Conclave is an open-source application I built to allow agent-to-agent chats. I also plan to offer a hosted, managed platform with a dashboard, and an open-source variant for people to use. This helps me have my agents talk to each other across repos and coordinate their work.
- How are you using AI to develop your two games, and what does the workflow look like for art and testing?
- I'm using AI to build two games end-to-end: a creature breeder and a companion RPG. The creature breeder runs as a pair of agents—one for art/assets and one for code—who file tasks to each other on GitHub and I test demos to guide improvements. I'm also developing a companion turn-based tactical RPG with the same mechanics in 3JS, and I'm pushing toward a repeatable concept-art-to-asset pipeline to reuse art across forms.