Agentic Skill Reuse, LLM Costs, and My Own Development

Transcript is auto-generated and may contain errors.

Hey folks, I am headed to the office here. Going to do a little AI kind of recap catch up. Um, I have a couple questions I still have to to get to that have been submitted, but I figured I want to give them a little bit more uh thought because people actually wrote in and I I think that deserves it. I've just been uh like pretty pretty overwhelmed with work. And uh honestly, I'm being totally transparent. I just have not had like the ability to like put thought into that the way that I want. And so I'm finishing up my last all call day tomorrow, which is Saturday morning. And then I'll have a couple days off after the weekend. So I'm I'm just hoping that, you know, use that to recharge and then kind of get back in the groove. So, a little delayed on those, but I uh just want to give them proper attention.

So, I figured I'd do a little AI catch up and then um yeah, have have something to talk about. So, I think few things for me just high level and then I'll dive into them in more detail. Um still doing uh a bunch of stuff with skills uh trying to make sure that I can have them in a way where they're I don't know like reusable, sharable workflows, right? I'll talk a little bit about that from the perspective of some things at work and how that's shaping up. Uh just because I think for some people you'll you'll see and hear different things depending on what you're doing at work versus what you're seeing like on social media or from talking from other people. It's just another data point for you to hear. Um and then I'll see if I can as I'm talking through that make note of anything.

Oh, I wanted to talk about costs in tooling. I might start with that one and then I'll see if there's anything like more uh specific for examples dive into. Yeah. Okay. So, we'll start with cost and things like that. So, over the past month, I would say is where I've really started to notice this. I have had moved away from my like Claude Pro Max cuz it was like 200 bucks a month and I just wasn't using Claude and uh again disclaimer not because I think it's bad or you know because I work at Microsoft I have to use C-Pilot and it must be better. Not at all. Uh just I hadn't been using it and so I'm like I'm not paying 200 bucks a month. I had gone a couple months without using claude at all. I'm like why am I doing that? Right? So canceled.

Uh, and then when I started doing more uh, co-pilot CLI work, that's been there's been talking about in my previous videos recently, that's been uh, a nice sort of transition. It's really the first tool ever for my development experience. It's pulling me completely out of Visual Studio and as a .NET developer, I live there. So, what I've been noticing over this past month though is that I am hitting raid limits like crazy. And I would love to tell you it's because I'm just so agentic 100x developer bro. Uh, but I don't really think I'm doing anything crazy. like I don't have I'm not going to sit here and like like I don't know like brag about how much how much AI I do. Uh I actually don't think there's anything fancy going on there. So I don't know what's been happening but to me over this past month it's just been a lot of rate limiting.

And I've been confused by this because I have a budget set up and I was like, there's no way. This is like a I don't know like a week in to to March and I was like there's no way I blew through my budget and I'm being uh like limited here. Like there's just no way. And so that wasn't the case. So, I had blown through a lot of it, but I also have um like that was sort of my allocated uh part with my subscription, but I have budget on top of that. And I hadn't even touched my my extra budget. So, I was like, I don't I don't understand what's going on here. like I'm not using this in an oddly aggressive way and I'm being rate limited and I'm willing to pay money for the requests that are going through like I

just I none of it's really adding up to me and uh so I don't know after checking my budget I kind of just figured I don't know maybe something transient I just I just don't know but over the whole month this is something that was happening I have to yawn. Just holding that in. Um, it's just been happening more and more and I I can't really understand. So, um, for me, I was like, "Okay, well, you know, I've been saying this on on these videos, like I don't I don't have a problem with Claude." So I'm like, okay, like let me get when I have my PowerShell terminals running C-pilot CLI, I'm like, okay, get Claude right beside there and just, you know, use Claude. So I've been going back and forth between Copilot CLI and Claude. And then in Claude, I was hitting my rate limits, and that's because I wasn't paying for uh the higher tier plan.

So, I'd have these evenings where I'm like coding things and like I just can't make any progress. I And you're probably saying, "Well, Nick, like couldn't couldn't you just write the code?" And the answer is yes. But uh the way that some of this or a lot of it I guess all of it was being done was that I was able to do it with AI and I didn't have sort of the the mental capacity or the timing to be able to sit down and crank out code. Um so I think I was really relying on AI for a lot of this this past month to uh to make progress on things. And that's because I've been uh you know swamped with things related to work. So yeah, just hitting hitting these rate limits. And that's another thing to call out, right? Is I'm not doing this during the day.

Like I'm working. So by the time I have time to go do this, I'm like we're talking about some of the work in the evening and I'm still hitting these rate limits. So anyway, the situation was like basically not hitting my budget for co-pilot, getting rate limited, using Claude, getting rate limited, and then I was like, "Okay, well, time to up my budget for co-pilot." And uh cuz I ended up eventually hitting the budget kind of as I predicted it was going to get hit. And then I I ran out of budget again on co-pilot. I'm like, "Holy man." And like I don't know how much I'm spending on co-pilot now, but I was rolling my eyes before at spending money on like 200 bucks a month on claw and not using it. I'm like I I feel like my total co-pilot costs are are are exceeding that now.

Like what the hell? So, um, yeah, I wanted to talk about this because it's kind of like this weird I don't know over the past month, this just weird shift in not only getting rate limited, but like blowing through budget and I realize as I say both of those things, like doesn't doesn't that sort of uh indicate that like you're that you're using a lot more? And I I think the answer is like yes, I'm using I am using more and I'm not totally bothered by having to spend money on it. Like I I get it. It's actually it truly is helping me do a lot more. But this combination of like how much I guess how much the the spend went up plus being rate limited. And I'm like in my mind like pick your battles, man. Like either like shut up and take my money or like rate limit me and like make it cheaper.

That's kind of this this feeling I'm getting, right? Um so yeah, this past month has been kind of weird for that. Just um I'm not I'm saying all this I'm not really sure what to make of it yet. Like um I I I think I have to pay more attention going forward to my tool usage so that I can I can better understand where I'm spending time and uh the budget for it. Right? So, just as an example, okay, now that I'm using Co-Pilot and Claude, if I go to my Claude Pro Max plan, right? If I go explicitly say I'm paying 200 bucks a month upfront on this, am I just like avoiding all the rate limits and then I can just say, "Cool, let me dial back my my co-pilot spend." I'd be happy to do that because the tools are, like I've been saying, it's not that I think one is way better than the other or something.

It's just I've happened to gravitate towards co-pilot CLI. So, I'll figure that out. Uh be curious to hear from people. Um what's going on for them? Like how they are perceiving some of this stuff? You know, are you running into the same things where you're like, "Holy crap, I'm blowing through my budget. Are you getting rate limited on stuff or are you are you navigating this in different ways or that's just simply not not a factor?" I'd be very curious to see. Okay. Uh, next up, maybe I'll talk about my own sort of like continued skill usage, refining and things. I think, uh, it's actually not it's not even skills for me most recently. Like, it's not that I've dropped off from using skills or building skills. It's just kind of like the refinement of skills in my development process is still happening. Uh, I just think that that is I'm not on some curve right now where I'm like every moment there's something new and exciting for it.

Just kind of chipping away at that. And uh I think the next thing for me is uh like targeted instruction files. And I I don't know I because I'm kind of getting back into the groove with Claude. I assume it has this concept. I just haven't uh explored how it works yet. But for the copilot CLI, um you know, you have your instructions file or your agents MD file and you kind of have that at the root of your your repository. Uh we can also do uh like pattern matching. So when the agent is working in particular folders or particular files um you can load up like contextsensitive instructions and I hadn't been using this at all. I wasn't sure that I didn't know this kind of thing existed.

I was under the impression the only way to do this kind of thing was to like in subfolders you can have other instruction files and they kind of get like locally loaded up but I don't like I don't know what the order of I don't know like does it go up the full directory structure and look and append all of the instructions do they have a precedence like I just I just don't know and so I hadn't dabbled with that at all but I've been noticing like man I I want more stuff in my in my agents MD file, but I know there's a tradeoff. Like the more stuff you're jamming there, you're it's sort of like I have all these very specific things I need addressed and if they're always loaded up, like it's just kind of nonsense context for the agent to work with, right?

Like I'm just going to make up a random example. So like imagine I have really specific instructions for uh writing tests and I have really specific instructions for how I want SQL queries to be handled. Um and then I'm working in a spot that has like absolutely nothing to do with either of those two things. Like it it just has all of that loaded into context for really no reason. And I don't know, this video might not age well because by the time people watch it even, maybe it's like I know we already have like million million token context models to use, but maybe that's just really not an issue and we can blast stuff in and it's all great, but I feel like it is it's almost like a trade-off. Like I feel like it's not getting smarter in general or doing better work in general.

Um, so even though you're adding more specific instructions, it's almost like it's polluted just by other So, um, when I started poking around a little bit, I was like, "Oh, cool. Co-pilot can use this." And the idea is that you drop them into a common spot and then it uses these uh pattern matching is it globbing on on the path that it's working in. Uh, so this is pretty cool. Um, I'm just starting to play around with this and uh I mean even what I had something that either ran overnight. I saw it this morning. I don't know if I asked it this morning or ran it overnight, but I had something for SQL queries forSSQL files and uh agent was doing work for me and when I had checked it, one of the things in my instruction file said like at least for this repository I'm working in, there is only manual migrations at this time.

So, I'll change it later, but migrations happen manually right now. Don't need to discuss that. But, so what happens is how the repo is set up every time the agent does what it probably should, which is, oh, I'm going to make a new migration for you. And every time I have to be like, don't do that. Don't do that. And I've tried putting it into instruction files and it's not working. So I do have one of these dedicated uh instruction files now that when it's working in a SQL file, one it tells it migrations are manual. So like if you're not working in the base uh sort of migration file, you're you're in the wrong spot. And the other part that it says is because it's manual right now, you need to tell the user when you're done doing work if you touched one of these files.

And today was the first time uh I guess it had to do that. And I saw at the end of uh you know the session that it was doing it had this big exclamation mark and it was like hey like I I did some work in this SQL file. You need to go migrate this stuff. It's like holy crap I guess I guess it worked right. Like it wasn't the only piece of work it was doing. It's not like I said hey co-pilot go do uh some SQL work here. It was just part of the feature development and it had to go uh do a data migration or change I think it was either renaming a column or inserting a column. So anyway um did the right thing and followed the instructions. So I'm going to be doing some more of that.

Uh, one limitation that I'm well I mean I'm sure there's many more but like one of the limitations that I'm kind of noticing is like um that works really well for like SQL files of course um it was working well for like I I don't use entity framework as a .NET developer. I like having uh SQL and I use Dapper. So I I generally have like repository patterns in my code. And so I was like, okay, if you're working in a repository like not not a Git repository, but like the the what's it called? Like the design pattern repository. If you're doing one of those, I have requirements around caching, around tracing, around transactions, things like that. And so I said, okay, like this is actually a good opportunity. I can say if you're working in a file that has repository in the name, like here's some dedicated instructions.

Right? So I found a few of these that work well, but there's a few that really don't. And so for example, logging. I want, you know, I have specific guidance that I want around logging. For the entire lifetime of this repository, I've had a documentation folder that says here's how to do logging. I had uh a specific instruction in my agents MD file that said for logging, go read this documentation that just simply doesn't work. And maybe some of you already know this and you're like, well, yeah, duh. Uh but I was hoping it would work and it just like has never proven to ever be successful. Uh so I'm like okay I really want to do some targeted instruction file specifically for logging but there's a problem and that's like you I don't just do logging in a certain file path. It's like if you're touching the C code so yeah sure if you're working with a CS file yeah you need to go look up logging but I'm like that's basically everything.

So, should that live in the agents MD file or should it be one of these dedicated file path ones and I just have to match on everything? I'm like, this doesn't feel good. I I and I don't know how else you would define it, but I would love to have almost like contextaware additional instructions, if that makes sense. So like if you're doing something related to logging, go load up this uh this information. Um so instead of like one big agents MD file, like can I just have more targeted ones? I actually think this could be pretty cool for like reusability across repositories and stuff. Like maybe I want to have the same logging patterns and stuff across my my different projects I work on, but you know, copying and pasting an entire agents MD file just like maybe because it's so polluted with specific things already doesn't make sense to copy and paste that across repositories.

I think it would be super cool like what we have that's uh sort of getting more traction now with these plugins in marketplaces. Could we have things like plugins for dedicated globbed instructions and even for like if we had this ability to have like contextsensitive additional instructions? I would love to have that and then I can start to peacemeal sort of like a setup for myself and that way that was terrible. That way I could use my tooling to basically say like when I'm working in a new repository or whatever or switching to one I haven't been in for a while like sync these particular types of uh instructions. I think that would be super cool. Um, it makes it tricky when they're all designed to be dedicated to a given repository though, right? I might work on two completely different things, right? I might have stuff for Brand Ghost, so I'm building stuff for my business that is CBased, and I might have my own other personal projects.

And I'm like, they're they're both C. I want to follow uh some of the patterns and practices I use no matter what in C, but I don't want to copy and paste this stuff across repositories, right? That's that's never feels like a great option to me. And then how do I keep these things in sync because they are checked into repositories? I just I'm hoping the tooling around this gets better. I know like as I'm saying it out loud, I know to go build something that can do this and sync this is not hard, but I feel like I'm not seeing a standard for that um yet, but I'm sure we'll get there. So, that brings me to like for stuff at work because I I said this would be I think good to kind of hear what's going on with different teams and stuff like that.

So, a lot of where I'm finding the teams I work with are at is in more recent time a lot more like people have been using Copilot or Claude for a while now. So, that's not really new. But, I would say like more recently people are starting to find like more and more of these like productivity unlocks for themselves. was like, "Hey, wait. I can make a skill for this or I never thought that I should try and ask C-Pilot or Claude like this type of thing when I'm building software." And like, "Holy crap, that works way better than just like go implement this feature and hope it works, right?" Like uh even uh really straightforward example is like uh having two different models or two different sessions kind of uh do some analysis on something and make recommendations, right? Just two different perspectives and then using even a third to then take those two and kind of like look at both perspectives and kind of take pros and cons from both.

Okay, that that isn't writing code. That's not having the LLM write code, but it's still a really helpful thing that you could try doing if you're designing things or investigating things. So, I think there's more and more of that coming up, which is great. I love to see it. And so, what's happening now though is that people are going, "Wow, so cool. I did this and like now I can make a skill or I could share a prompt or like I just want to like share what I'm doing with other people because I'm so excited that it worked for me. Again, all awesome things, but then they're reaching this point where they're like, "Well, do I do I just copy and paste this file and like do I send it to you on on teams or like do I put it into this repository that's like, you know, like a hundred years worth of content across like multiple teams?

Like what's what's the answer?" Because it feels like it's not a good answer no matter what to go share something that's trivial, right? we know that we want to check it in so that it's tracked, it's it's easy to share. Um, you know, you can sync to it, but then like where where is that really supposed to live? Where's a good home? Especially if it's not specific to a code base. And so, like, yeah, we're kind of getting on board with these uh these marketplace systems, which is cool. We're trying that out. It's going to be new for us. But I I think the fit is really good. And so we have that to take advantage of. But that's kind of this next phase that we're getting into. And truly because we're we're hitting this pain point of like so cool to see everyone's trying to build skills and share, but there's no there's no good way to share.

the number of times I've seen someone like either try to zip up a folder or like just take this MD file so you can like see how I did it and build your own. Don't get me wrong, again, super cool that people are having these unlocks for themselves in terms of productivity, but it's uh it yeah, it feels like so backwards for like how we have to share stuff. It just yeah it seems like you know it's demanding a solution almost. So uh pretty excited to see what we can start to do with people just being able to like put stuff into a plug-in marketplace sync right getting a common spot for this stuff. Again this is not new. I mean AI stuff moves super fast. Like is it new in the last year? Yes. Um is it new as of this week? No.

So, it's not that it's a new groundbreaking thing, but I wanted to share that like this is one of the things that the teams I work with are starting to to see see as a pain point and try to address. It just hasn't been hasn't been a great option. Now the next sort of weird part related to that is I think we have like a pretty good solution to try out and it's going to be an experiment. I hope it works because it seems like a good fit. But that should be a good solution for like how you configure MCPs you want to connect to for uh your skills and things like that or like agent prompts. Feels like a good fit. What seems to be missing right now is like we don't have they call like a paved path like a really good straightforward way that if you have an MCP server like how to distribute and share that that seems to be a bit of a a gap right now.

And so it works trivially well like if you already have an MCP server hosted and running somewhere cuz it's just like you have a configuration that says here's where my um here's where my server is running, right? You you use my plugin, you'll get this MCP configured and you can just access it. So, like that part's simple, but we don't have a paved path for like you built an MCP server or your team's building an MCP server. Cool. How do you either distribute that so people run it locally or how do you go deploy that in a compliant way that everyone should be able to go use? And again, these these are not super um big challenges like in isolation, right? Like how do you share an MCP server? Well, I mean share the repository if you need to like build it locally, right? Big deal.

Or have CI/CD publish an artifact and you can go download have the MCP server to run locally. No big deal. You want to host the MCP server, no big deal, right? It's a web service. whatever. Um, but we're talking about the ability for for people to go, you know, have uh like what is the surface area we're talking about? Like we're employees at Microsoft should like the security, compliance, privacy implications of being able to do some of this stuff. It's just like we need a paved path. We need to do it so that no one has to think about it, right? You should in theory just be able to press the button and you can have something that's deploying your MCP server cuz we can guarantee the only way it works is if it's compliant and secure and things like that. But some of that stuff still has to to kind of come to fruition, I think.

So that's the next kind of step that I'm trying to look at is like again not that this problem has not been solved but in terms of how we're going to do it, how other people are already doing it around us uh in our org. Um like I'm sure other people are already doing it but did they all start trying to pioneer the way to do it in the org? Are we already reaching a point where um you know there's a wellestablished pattern? Uh I just I know I'm not going to be the first one. Uh I feel like in my in my smaller part of our our or our group like I'm trying to get that question answered early and got to see what's next for it. So, um, yeah, I think that will be kind of cool so that people feel like they if they need to go beyond having skills and they need to be able to provide data to agents like different integrations that way that that they can, right?

Or if we're doing uh things that allow us to this is like a big consideration, right? Is like are you reading data from a data source, right? That obviously has implications around like security without a doubt. Are you authenticated to be able to do that? But an even bigger sort of thing to consider is like when it's not reading data, it's writing. If you can modify, mutate, right? That has all sorts of implications, right? It's there's don't get me wrong, if giving wrong access to to people to be able to read information, not good. But uh even worse would be you go build um oh not crossing. Okay, these people are walking towards the crosswalk but turned.

Um, even worse would be, you know, same same kind of thing, but you have right access or imagine like you're running your your LLM and it it has approval to go do something and it goes and writes and you're like, wait a second, like I I didn't know that it was going to go do that, right? Drop the production database kind of thing. So, we need to just make sure that um how that stuff is built is is compliant. It's all stuff that if you're building your own, like these are solvable problems. Like, but uh when you start having a lot more people and a lot more surface area, I think that we just got to get these things nailed down. So, that's my AI talk. I uh realize it's nothing like groundbreaking. Just figured we'd talk about it. you see where I'm at on certain things and and I'll catch you later.

But yeah, thanks. I got, like I said, I got some some questions I got to get responses for. If you have questions for this video or in general, just leave them below in the comments. And otherwise, you can go to codemute.com and submit stuff anonymously. I'll see you in the next one.