Behind the Scenes: Developing with Microsoft Agent Framework

Transcript is auto-generated and may contain errors.

Hey folks, I'm just on my way to CrossFit. Going to chat through a little bit about Microsoft Agent Framework. um some of the the building I've been trying out there and um just kind of a thought stream on what I'm going to be trying to put some videos together on on my main channel, Dev Leader, because I tried doing this before, like putting some things together and the results were pretty terrible. And it's I think like entirely because of like how I did it. Uh and this approach seems to be working significantly better. So for those of you that aren't familiar, uh Microsoft agent framework is sort of this successor to something else that Microsoft has called semantic kernel. And if you are a C developer like me, then you may be familiar with these things if you've been trying to dabble in the AI space a little bit.

And so Semantic kernel was this framework where you could basically set up um like a a chat client so to speak that you have with your LLM and then you could start hooking in like tools for the LLM to use very much like we have with uh like our co-pilot CLI or clawed code like you can give it tools to use as it's uh running and executing uh you know things in the chat for you. And then Microsoft agent framework as this successor is really like multi- aent orchestration uh which is super cool. So uh just like you can do sub agents and tasks and stuff like that in in these uh these other harnesses. You can kind of do the same concepts in Microsoft agent framework.

Now, that is a pretty big unlock because I think well, I've been noticing that the more and more time I'm spending in co-pilot CLI and Claude that uh you know, you can you can go from having a couple of things that are like prompts or simple skills into like, you know, entire workflows that you're that you're doing that are like basically fully agentic or they have points where they need, you a human in the loop kind of thing. You can do a lot of powerful stuff. And if you're thinking, okay, like this this is a cool thing that works. And like depending on what you're you're doing or building, like maybe you're at a point where you're like, I wonder if I could have this done like a little bit more programmatically. Like could I have this triggered from something? Could I run this as a service?

Right? So, in in my mind at least when I start getting to those types of questions, um it goes from okay, well, I'm I'm probably not going to to be running like co-pilot CLI or cloud code like somewhere in the cloud uh to go, you know, perform tasks when it receives web requests. Like I that's not sort of the the life I want to live personally. I I would rather have something that is like a first class like you know harness that I've put together that can do some stuff for me. But that's where Microsoft Agent Framework comes in. And that's also where like all the the challenges come in. And so it's it's been really I don't know I want I'd love to say like enlightening to to suggest that I've learned so much. Um but I'm still obviously like learning through this stuff. But it's it's been yeah like eye opening I guess is a better like phrase to use here.

It's been very eye opening because with Microsoft agent framework, you get a lot of stuff like out of the box and in terms of like setting up flows of things. That's that's really cool. But I think the eye opening part for me is like just how significant it is of a of a difference to have good tools that your LLM your LLM can use and to have um a good like harness setup in general. So, let's talk about like what that means and kind of how I went wrong the first time when I tried this. Um, so if I talk about my first attempt through this, this is going back a few weeks now. Um, I mean, it's going to sound very obvious like why this went so wrong, but like hear me out, I guess. So I had uh a bunch of skills that were like orchestrated uh things for agents to do.

And so they work very well in claude code. They work very well in co-pilot CLI. And basically I have like a you know a oneliner for a skill that I can run and either of them uh does a whole bunch of steps like there is like sort of like web research. there is uh taking that information, aggregating it, you know, summarizing it and then pulling out some details. And then there's a human in the loop part. So you sign off on something and then from there uh it actually spins off orchestration across a bunch of parallel agents. Then after that when those agents are done there is a sort of another step that brings it all back together at the end. and done. But even the the orchestration of the sub agents that are done in parallel that itself is its own orchestration. But all of this is like it's just done through skills, right?

So each one of the skills has just primarily a prompt and I think a couple of them have some like some uh tools like that are PowerShell scripts and those I think are primarily just to make sure that some of the information is structured uh between some steps just to kind of streamline that a bit. But the point is this just skills and they work really really well in co-pilot CLI and claude code. And so I said, I have this genius idea. I will ask co-pilot like let me show you these skills, right? Like let me show you the skill folder and I have this Microsoft agent framework project and I basically want you to go translate that orchestration that you've been using. Go translate that into uh a Microsoft agent framework pipeline. like go build the the workflow for me. And so it did. And uh it was terrible.

It was it was really really really terrible. And so my iterations with that were like, okay, like I'm reading some of the code and I'm like I I see what it's done and in some spots I'm like it seems like it's bypassing some of the built-in Microsoft agent framework. like some of the pieces that I feel like the framework should be responsible for. Like there are some spots where when it was trying to run some agents in parallel, it's like I'm going to go build my own sort of like pipeline orchestrator and I'm like ah there's there's literally like a Microsoft agent framework thing for this. So there was a bunch of stuff like that that it seemed to be kind of cheating or working around. And I suspect part of that was really like Microsoft agent frameworks new. So it doesn't have um as clear of an idea like how it would go build all of those things given all of the all of the stuff that's available.

So I think you know first thing that went wrong was like not giving it enough documentation to pull from like I think I gave it the highle docs and assumed that it would navigate some of the website but like absolutely not. So once it had something in place, I was getting it to refactor it and kind of pointing out some of the documentation that's like, hey, like no, like use this for orchestration. And so it would, you know, start moving some stuff around. And then I'm like, oh, wait, like now, like little by little, we're starting to get better and better results. And I just kept iterating this way. Okay, so point being that I took all of these sort of vibecoded iterative steps with it and by the end it was giving me results where uh I'm I'm looking at tool calls, I'm looking at token usage, I'm looking at the actual output and I'm like this is starting to feel like it's working.

Like this is actually going pretty good. And then I said, "It's time for me like I need to go I need to go look at this code and see what's up now. Like I'm kind of getting to the point where like I I got to understand this a little bit better." And when I went looking, this is where I realized like all of the optimizations that we were doing were very much like hyper specific to the scenario. And I know I'm not showing you code like because I'm driving a car and talking about this, but what I want you to imagine is that anytime we needed an optimization for something, instead of taking it as a general approach, it basically just like thought extremely specific about the the thing that we were discussing and like added if statements for that and like scattered them and stuff.

So it wasn't it wasn't coming up with generalized patterns for like optimizing these these situations or correcting issues. It was like nope there's a you know if I need if I'm writing something like this and we talk about you know these words like if statement on them and special condition. So when I was looking through all of this, I was like, "Oh my goodness, all of this is hyper specific to like a scenario that I was trying trying to use as like a baseline to kind of go through this work. It was awful. Um, the code was so awful that I was like, I I can't I actually can't continue using this because every single optimization that we've touched, I just have to throw away. Basically, I I literally need to go back to the beginning because nothing we've done could be used. Okay. So, in round two of this, I went um like probably a more intelligent way.

And this is really for me highlighting uh something that uh someone on my stream said last night, which was like uh and they were talking to another person like in the comments in the chat, right? And they said, you know, the it's from their perspective, it's a lot less about the model. Like yes, obviously, you know, newer models are generally better, but a lot less about the specific model and a lot more about like what the harness can facilitate with it. And that was a lot more apparent on this second pass so far. So for full transparency, I haven't completed the full orchestration uh being re-implemented, but what I decided to do was take a more narrow approach. So I took one part of the orchestration. Remember when I was saying that I have an orchestration where there's a bunch of parallel agents that that run.

Um what I am doing is taking the workflow specifically that those parallel agents do in parallel and um I'm treating that as like a segment of a pipeline or a workflow to go build out. And so my idea was like I want to start with that but I need to literally start with the first stage in that. So in literally instead of jumping into like let me use Microsoft agent framework to get the orchestration cool stuff like let me start with Microsoft agent framework and I'm only going to build a single step. Okay. And to me that was important because if I can get the single step and I have confidence in it like then I can build on top. And that's why I'm realizing it sounds like kind of obvious now.

I think I was just hopeful that there's enough parallels with the orchestration stuff to what I was seeing in Claude Code and Copilot CLI that I could take all of these skills that are already the prompts I need, but anyway, it doesn't work that well. So, yeah, I focus on a single stage and that meant that I could focus on the tools that it would need to use. And another thing to call out is that I am using an older model in my code. I'm using GPT41 which is not good at uh having sort of a lot of text output at once. In fact, if you ask it to write something that's like, you know, uh the equivalent of like anything over like 1500 words, like it it won't. And if you said to it like I need you to add, you know, add more and you told it like I don't know like what in whatever body of text you have either add more at the end or insert text.

It can't do it. It re tries to rewrite the whole thing. So for me this was really about um how do I give it the tools so that it can do that right? Or if I said, "I need you to change this." Right? You you use the word um my variable ABC and like that's forbidden. So I need you to change that. And what it would do is it would rewrite the whole thing. And when it rewrites it, it's not doing a good job of preserving anything. It's literally just like doing a one shot. Let me go do it again. So when you can give it some tools, then it could do a direct search and replace cuz it can use the tool and say like, you know, look for this phrase, then replace with this phrase or you can give it an a specific insert tool, right?

So that insert tool is actually doing the job of changing the text and the agent doesn't have to go rewrite everything or it can append, right? So just giving it effective tools to work with the text that it's uh trying to produce like sort of the the artifact of the of the stage of the workflow. So, I spent time getting the one stage and then I said, "Great." Like, now that that one stage works, um, I have two other stages and, uh, that I want to link up. And it's basically like uh I don't know, this is pretty common thing I I find in like agentic workflows, but you have something that does some work and then you have maybe like a two-step kind of thing that is something that like reviews because we've all seen agents hallucinate, make up, uh or say yeah, like say blatantly wrong things or omit things, whatever.

So having some kind of review where it can have its own context, look at the the artifact that's produced and be like, "Okay, like here's my critique of all these things." And then uh you something else that uses that to go do the fix up, right? Whether that's the first agent or not, however you want to do it, but like basically a review and fix step. So that was the next part I I wired up. And this was also kind of like couple of uh enhancements to tools because I realized some of the some of the things it was trying to do to like to fix up we just couldn't do effectively. For example, um yes, there was a search and replace tool. Okay. Okay. So, had that um or there was like a a m like an edit sort of tool, but when it needed to do multiple surgical edits, what it was sometimes doing was like it would do a couple and then it would just like give up on the others.

Uh it would just Yeah, it would just stop. Uh, and I I have a feeling that's because as the the text was changing, it was not able to do the um the text insertions that it intended to and it would have to go reread the article or the the artifact and it would have to go do like all of the work again. So, uh, okay. Like we need a bulk edit tool, right? So when you have a bunch of edits you want to apply like use this tool and do them all at once and this kind of thing. But the biggest part I found in uh in this review part specifically was like how to indicate properly to something that has to do the fixups like an effective way to handle that. And what I mean is like instead of just saying like this is wrong.

So instead of saying like don't use um you know don't use uh naming that's forbidden, right? Don't use naming that's forbidden and I found this specific forbidden word or this these sets of forbidden words at these locations. So you're just giving it a ton of context on what is wrong and what needs to be fixed up where it is. And then in some cases like can it even give a a recommendation? And I found that like tuning that up and sort of like bulk editing was ultimately the biggest mover for for this kind of stuff. Uh so yeah, now I have a mini agent orchestration and uh I'm at the point where I'm going to go try wrapping that in the next layer of orchestration. But um this approach is working way better because I know from the beginning I have a tool setup uh and a workflow setup that I feel better about.

Uh because the other time I got to the end and I said I got to toss the whole thing. So yeah, I'll make some videos on how I've been trying to put some of that stuff together. Um and you can check that out on my main channel which is Dev Leader. It's all my my programming and AI tutorial. So, when that's all ready, head on over there. You can check that out. And I'll see you in the next video. Take care.