In this video I talk through some of the agentic development I am doing between several agents working in parallel on some hobby game development.
📄 Auto-Generated Transcript ▾
Transcript is auto-generated and may contain errors.
Hey folks, I'm just going to drive to CrossFit here. We're going to do I guess like a little AI recap because it's been a little while. Um, so I'm going to talk about just a handful of things that I'm building and how whether like the shape of how I'm using AI is changing or or not, like what's working well or what's not kind of thing. Um, and for context, if you're new to the channel, usually I go through uh user submitted questions. So if you have questions about software engineering or career development, please feel free to leave them below in the comments. You can go to codecomute.com and submit stuff anonymously that way. And uh otherwise I go to the experienced devs subreddit, talk about things from there or I will periodically do these AI recaps and just kind of share what I'm building. So I am a software engineering manager at Microsoft.
I've been uh there for about six years. uh been engineering management for about 13 years, but I still love to build things outside of work and so I'll be talking about some of those projects in particular. So, um right now how things are looking for me is that we have uh some changes that came over the past couple months as Microsoft employees for for co-pilot usage uh like token allowances. So, I basically am balanced between a Clawude Max plan, Pro Max, whatever it is, the 20X one. Um, so I'm using that and I'm using Co-Pilot and when I've talked about, you know, Claude and C-Pilot in the past, like kind of comparing and got to get around this truck. No thanks. enormous truck that was going to take the entire time the light was green to get through the intersection. Um, when I've talked about Claude and co-pilot, it's not like I tried to make it not a thing of like, oh, you know, co-pilot is better than Claude or Claude is better than co-pilot.
There I think they're comparable tools. I think that people are going to have pro their own pros and cons to both. Um, and I used Claude code a lot last year, but uh, this year it's been almost exclusively co-pilot CLI. And that's a lot of that is just again like token allowances and stuff. So going back to balance between the two is interesting. Um, I have been using Claude code more for what I would say are like less important things. And I don't mean that because like I think Claude is inferior, but um because I've been using Copilot CLI so much more this year. I'm just a lot um I don't know better better I better aligned like how I develop things to using Copilot CLI, right? It's probably the best way to say that.
And so for things that are more critical for me, so when I'm building things like brand ghost um which you know if because that has paying users that's going to be something where I want to make sure that I'm you know doing my my best kind of development and so co-pilot CLI for that. Um I want to make sure that I am putting my token allowance towards things that are uh what I would say are more critical. And then I'm kind of using my my Pro Max plan for Claude as a little bit more like um if I happen to go over budget for the week, like so be it. It's not going to be the end of the world, right? So uh my Clawude Pro Max plan right now is being put towards uh a game. So I've been been trying to balance this because uh I need like more compute locally.
Oh man, I need more compute locally so I can build games. But I've been using uh the ProMax plan for one of these games that uh has been fun to build so far. It's called Habitat uh it's called uh Evo Habitat Zero, and this is like basically a a creature breeder game. Um, and it's uh, you know, taking concepts from other games that I played as a kid. And, uh, I have two agents that build. One does a lot of the game mechanics. And I've talked I actually talked about this in a live stream not too long ago. Um, I basically tried to define the majority of the game mechanics and things like that with chat GPT while on vacation. put together a bunch of uh you know like game highle game design docs and then set cloud code on it. And so I have one agent doing the game mechanics and then another one doing uh like art and assets.
And it's been rocky. There's been times where I'm like, "Holy crap, like how do we get down this path?" Um and then there's been times where like it's going pretty smooth. And I would say like this week in particular, it's actually been going pretty smooth. I think the beginning of the week was the biggest hurdle where I was noticing my local builds. I have a machine dedicated towards running like Docker containers for uh for GitHub actions just to run them locally and save on the action costs. And like that was crashing a lot and what the heck is going on? But it turns out that um there's a couple things. one was that the Unity builds were just like taking so much memory. Um, and one of the challenges there was that I wasn't using like a Unity uh asset cache. So every time it was spending spending 40 minutes importing assets and then using like an insane amount of memory and then also running um some builds in parallel.
And those builds were from from the agent that's doing all the art. It had like been making models and textures and things like that. and it had no concept of like what is an acceptable file size for an asset. So there were some pretty simple or I would call them simple models for for things in the game. It's not a It's not an art intensive game, but um there's some simple things that were taking like half a gig of RAM u sorry not half a gig of RAM, half a gig in terms of file size for a simple model and then the side effect is that it's loading tons of these types of things when it's doing the editor build. Um it just just crazy. So anyway, that was the the hurdle at the very beginning of the week.
And then since then, uh I have it where the the art agent at least is reporting to me every few hours it will do like here's a handful of screenshots for things I've been working on. And it's it's been super cool because sometimes the art that it starts to put into the game, I'm like, "Oh man, like definitely that's a step backwards. It's uh like it's not good." And then I'll see throughout the course of the day or by the next day it's been iterating on it. So, uh, the actual model that's used has better shape. It's less like Minecraft. Uh, the textures go from being like, you know, a a picture that's been kind of stretched across a funny surface into something that looks more like a texture. So, it's cool. like it's it's uh notably improving on the work that it's doing, but what's not happening is like uh it's not like self-improving.
So, what's a good way to say this? It will iterate. So, like it will land some some models and some textures and things like that. And it's kind of like it's saying like I got as far as I could and this is good enough for now. I'll land it. And then it ends up doing follow-up commits with improvements. But what what's not self-improving I mean is its methodology. That's what I need to say. So however it's doing stuff is allowing it to iterate and that's good. But um it's also just like being pretty wasteful I would say because it would have been nice if for some of these models instead of taking like you know 2 weeks of iterating on the model and the texture like it just did it right the first time because it's not if you see the models it's not like a complex thing it's doing right.
It's really complex for me as someone who is not skilled at art and assets. It's difficult for me to do it by hand. It's also difficult for me to to sort of give an agent the proper guidance to do these things effectively because I don't know. So, what I've been doing to periodically drive some improvement is I'll take some of the screenshots that it generates. Uh, I just jump over to chat GPT because I can get some images generated, too. But I'll say, "Here's some screenshots. Here's like what I think is going well. Here's what I think is not going well. Can you write up a bunch of, you know, um, generalized concepts for improving things in the agent's flow of making art and assets?" So it does I give those over to the agent and then I also ask Chad GPT you know based
on my criticisms can you take these screenshots and like basically generate uh not concept art but like can you generate you know remake the screenshot with the improvements and it does and then I also give those over to to Claude and say hey like you know update your instructions uh to adopt these like methodologies and and and then here's like a targeted fix I want you to go do. And so one of the examples is that uh for a lot of this game it's on a space station and the space station itself is metallic, right? It's kind of like a technology. It's it's inorganic, but the creature habitats on the space station are organic. So there's like moss and greenery, and that means I need to have both of these art styles mixed together. So, the agent had made moss and it placed it, I think, really well.
So, it looks cool, but the texture of the moss is just like it looks like a picture that's stretched across a funny surface. So, one of the things that worked really well was this process going to chat GPT. And again, the agent, you know, did an iteration and it sent me the screenshots and I was immediately I was like, "Oh man, like that's not going to work." Like it it made it worse almost. And then I waited and another iteration I was like, "Wait a second." Like I think something's happening here. And uh now at this point, I think I have some screenshots where the moss looks looks pretty good. But when I'm saying it, I mean the next few hours. So, it's seemingly just like trying some things out, getting a little bit further, and just saying good enough for now. We'll come back to it.
So, again, it's really like I would love to compress how much time it's taking on that stuff into a shorter cycle, but it's uh it's working well. Like, it's making better and better art over time. So, I'm happy about that. And uh I think like especially over this weekend cuz it's a Friday morning. I'm going to have to figure out on the game mechanics, I think it's kind of hitting a spot where like it seems to think it's done building the game and it's just kind of like finding some extra little things to do. And I'm like I can tell you the game is not it's not uh done. So, uh, I'm going to have to basically press pause on that agent, uh, have a conversation after doing a bit of play through and then say like, cool, here's the the new road map. But overall, for two agents kind of, you know, cranking out art and assets and one doing code, I think it's working pretty well.
Um, so when that's going, um, I actually have another game that I want to put more attention back into. So I am going to try and take some of the learnings especially around the art and assets and and kind of uh regenerize them so that I can use them in my other game that's a role playing game. uh it is a multi- sort of format game because I want to be able to have like uh a 3D like action role playing game and then I also want to have a companion app that is allowing you to use the characters and your items and stuff in a uh like a turnbased strategy like role playing game. So like Final Fantasy tactics kind of thing. Um, so I'm gonna start with that one, I think, because I I do think that a, you know, more like closer to open world action role playing game is, for anyone who's tried making role playing games before, like sounds awesome, right?
But it's it's it's a lot of work to get right. So, I want to start with the turnbased one. Um, and set some better some better constraints. So, I have a lot of the uh sort of like itemization and stuff built because I love building that kind of stuff, but I I actually have to get to a playable state. So, I'm going to I'm going to try kicking that off over the weekend as well. And maybe that will mean that I'm chewing through my Clawude budget way faster. Uh, and that's now going to be a goal for me to be less wasteful with Claude because like I said, the art and generation that way I'm pretty sure is being super wasteful. Okay, that's game stuff. I'm almost at CrossFit. Um, otherwise, what's going on for some of the other things? Um, I think a lot of the patterns and stuff that I've been talking about are continuing to work well in my development.
So, uh, reusable instructions, right? So, I because I've been building a bunch of different things, there's lots of it's like casting a wide net for the different types of, uh, lessons to learn, the patterns and practices and bringing those back to adopt into things. So for example, I wanted to put together a um a crossplatform app and um like to to prototype something basically and if it you know happens to be usable and good then I'll proceed with it. This is brand ghost related. And so I have a repository of like templates that I built up. And it was like, hey, we have like, you know, almost uh around like four different templates that would work here to scaffold an entire app. Like which kind of like what tech stack do you want, right? Like what what are you actually after here? Let's talk about the pros and cons of each and then we can scaffold it.
And so I did. And so it worked really well in terms of getting an app up and running super quick. But I think the part that's still missing comes back to like the UI did not look good, right? So like I'm missing I'm still missing some proper guidance on like make this user interface not look terrible. And um as I come to CrossFit here, I think one of the ironic things that's kind of showing up time and time again is like the stuff that I'm actually I feel like I'm good at out of my skill set like for actually building software like architecting systems, building software. The agents that I'm using, it seems like I've been able to give them pretty good guidance to do the thing that I do well or that I think I do well. And um the stuff that agents struggle with is all of the that I suck at because I have not been able to give the agents good guidance, right?
They don't have good guidance for building UIs because I haven't built good UIs before that look good that are aesthetic that like I don't have a lot of experience building good-looking UIs. I my UI work has been windforms from like a hundred years ago. Um, so the irony. Why is this thing not focused? What do you focus on? Focus on my bald head. No. Okay. Well, sorry it's blurry. Um, yeah. Thanks for watching. If you have questions about software engineering or career development, leave them below in the comments. Otherwise, go to codekameete.com. You can submit stuff anonymously that way. And I'd recommend like if you want to see like scroll through, pick some other videos. You can see how the Q&A uh style kind of works. But I am happy to try answering things if you think they will help. Um, like I said, I've been an engineering manager for over 13 years now, writing code for like over two decades.
So, I am happy to try and help. But yeah, thanks for watching. See you in the next one.
Frequently Asked Questions
These Q&A summaries are AI-generated from the video transcript and may not reflect my exact wording. Watch the video for the full context.
- How do you balance Claude Pro Max and Copilot usage given token allowances and budgets?
- I balance Claude Pro Max and Copilot CLI by reserving Copilot for the most critical work and using Claude for less critical projects, like the game I’m building. I want to make sure that I am putting my token allowances toward things that are more critical, and if I happen to go over budget for the week, it's not the end of the world. Right now Claude Pro Max is being allocated to a game while Copilot CLI handles core development tasks.
- What challenges did you face with AI-generated art and assets for your Habitat game, and how did you address them?
- I faced memory and build issues as the AI agents generated art and assets. The Unity builds were taking a lot of memory and I wasn't using a Unity asset cache, so assets import could take about 40 minutes and use a lot of memory. The art agent produced assets with large file sizes and had no concept of an acceptable asset size. To address this, I iterated by reviewing screenshots, asking ChatGPT for improvements, updating Claude's instructions, and pausing to reset the road map.
- What patterns have you learned about using AI agents to build apps and games, including templates and UI guidance?
- I've learned that reusable instructions and a set of templates can help scaffold apps quickly, and I built a repository with four templates to choose from. The UI side, however, still needs better guidance because the UI didn't look good and I didn't have good examples to teach the agents. I found that the agents do well with architecture and coding, but my own lack of UI experience means they struggle with UI aesthetics.