Discovering the Sweet Spot in AI Orchestration


I was in the middle of recording my latest premium video on agentic orchestration when my Claude Max subcription up and left the building. I was in the middle of my pi.dev demo when BOOM! I got the dreaded "third party tools" request error.

Note: if you don't know what pi.dev is I'll explain more at the end. I'm also putting together a video on it.

I went online and snooped around and, sure enough, Anthropic knee-capped OpenClaw and every other third party tool that used it's OAuth API Key:

Thankfully I had some extra usage to draw from, and Anthropic gave people a credit on their API accounts, but the net effect was a simple one: I was working along happily, and then my day went upside down.

Control, Do You Have It?

This is what happens when you give control over to a service, framework, or toolset: when things change, it hurts. As programmers we understand that building a malleable, maintainable, and "correct" codebase is the goal. So why do we ignore that with out tools?

The answer is simple: less control typically means more efficiency and speed. Ruby on Rails, the .NET ecosystem, Django. Let someone else decide and argue about it, we'll use what they think is best. Why not? We have work to do!

I just wrapped up my Winter Cohort (which was a ton of fun) and on our final day I asked everyone how important control was to them. Claude Code is a fine tool, but if your entire AI coding process is based on what Claude Code offers, you're one dimensional. You do everything "the Claude way", which, for now, is a good thing because they're the ones defining how this whole thing is done.

What happens when that changes? It's something that you learn the longer you're in the tech space: a given service or vendor will, eventually, disappoint you. The tool, language, or framework might change and offer less value, or maybe the founder will turn out to be a horrible person. The company might invest in things you don't like... who knows.

This is the rug-pull factor, and the ideal number is 0. The only way to approach 0, however, is to own your own toolchain. That, unfortunately, leads to a fiddle factor of 10. The higher the fiddle factor, the less work we do.

Classic NP-Hard combinatorial optimization problem: minimize the fiddle factor while maximizing productivity with proscribed tooling. You might have found a good balance in all of this, but add a parameter you didn't think about before (new model release, the owner sells to Evil Corp, malware) and the whole thing comes undone.

The Sweet Spot

Unfortunately the only answer I've found is the one that's always been true: money. You have to pay if you want more control.

Going back to my pi.dev story: if I'm willing to pay $400/mo or so, I can have a super premium experience using Opus 4.6 and Sonnet 4.6. I know that Opus is overkill for coding, but it consistently brings the best results through planning to write code.

I can reduce that figure by using Opus only for planning, however. If I take the time to write locked-down skills, detailed instructions, and then craft a solid plan, I can use Haiku to write the code since there will be very little "thinking" involved there.

In other words: if I fiddle a little bit, I can have more control, and reduce my costs. The goal is to fiddle in the right way and not get lost in the weeds by playing in the toolchain and not with it. This is where pi.dev comes in.

Hello Pi

One of the reasons I love pi.dev so much is that you can control almost everything, right through the pi interface. It's aware of it's own docs and settings, and also how to extend itself, so you can tell it precisely how you want to work and it will configure itself that way.

Pi works with the notion of "extensions", which are typescript files that fire on given events. You can mix determistic and non-deterministic stuff, test it out, tweak it, and... yeah fiddle with it until it hits your sweet spot.

Crafting your sweet spot with Claude Code amounts to you just doing what the tool wants you to do because you're paying a subscription and hoping you don't hit your rate limits.

With Pi, you don't have to hope as much. You can create a planning agent that uses a more capable model like Opus or GPT 5.4, then have a set of agents that use whatever model makes sense for the work. If you don't know what model to use, pi will choose one for you.

Yep. Pi will choose the model that makes most sense for the work the agent is doing. It's an incredible tool, but like I keep saying, it's also very fiddly.

Pi makes it so easy to create these agent fleets that you find yourself sucked in to this new world of AgenticOps as if you were forcefully pulled through a void in reality and thrust into the future of software development! Everything you thought you knew about AI coding has changed because you can now do the thing you thought about, but couldn't get Claude Code to do properly for you.

It feels like being transported from Wichita to Bangkok and realizing that the rest of the world has evolved and moved into the future...

I Know I Sound Like That

Landing back on the ground here, I do realize that a full agentic swarm isn't needed that often. For most things, letting Claude kick up its own agent teams will work. Things like refactoring, debugging, polishing and documentation aren't "big lift agent army" energy, and I would argue those things are probably the bulk of our time.

That said, you could create yourself a refactoring agent, debugging agent(s), and documentation agent. You could then orchestrate them into a "polishing" process led by a senior planning agent with Opus and...

This is your brain on AI.

Sorry, I couldn't help myself. It was too much of a challenge to see what could be generated. I like that the initial brain state is frying in the pan before AI enters the building. That kind of captures it.

How Much Control Do You Want?

This was the last question I asked my cohort. There are tradeoffs to everything, for example, you could:

  • Use Microsoft's tech stack and have all of your tooling given to you by one corporation. From the platform (.NET) to the data access (EF), to orchestration and deployment (Aspire), to your DB (SQL Server) and your IDE (Visual Studio). Everything works well together and you can get things done.
  • Be an Apple Fanperson like me, with an iPad, iPhone, 4 Home pods and anything and everything else Apple makes. They work together, I don't need to think about it.
  • Use Ruby on Rails and have the entire development stack, and process, handed to you.
  • Use Node or Python and assemble packages together exactly how you want, praying there's no supply-chain attack issue (spoiler: there always will be) but also creating the precise thing you or your client needs.

This is an individual choice, obviously. People drive minivans and Tesla's because they have other things to think about and, at the end of the day, just need a car that gets them from point A to point Zed.

Others drive Lambos and Humvees because the driving experience is, to them, far more than getting from point to point (I'll let you fill in the blanks as you will).

What's your coding preference? Do you build Lambos, or are you OK with that Model Y experience like every 5th driver on the road is?

My New Video

I made a 48 minute video that gets in to this topic but mainly explores the different levels of orchestration you use to get work done. I build an MCP server and chat interface for exploring Cassini's mission plan (what else), and I do it 5 different ways:

  • Full vibes. No orchestration at all, just being "interactive". This is great for refinement steps and debugging.
  • Agent Teams with Claude Code. This is where you let Claude do stuff for you and hope it will actually do those things (spoiler: it often doesn't and won't tell you).
  • Plugin orchestration. I used Superpowers and Oh My Claude Code and go for a walk.
  • Using pi.dev. I'm still learning this tool so don't take this as a "guide", it's me exploring. I had lots of fun and like this tool a lot, if you can't tell.
  • Rolling my own. I made a CLI for my clients to build on top of, something I call "Chappie". It's not as fast, but it is, without a doubt, the most finely-tuned experience I've ever had with agent orchestration.

I'm planning on diving deeper in to pi and tools like it, such as OpenCode (the big one in this space) and Forge. There is a lot to dig in to here, so I might just focus on pi for now.

I take requests, so let me know what you think! Also, very curious to know what level of control you like to have.

Thanks for reading!

Rob

P.S. you might notice a format change with this email. I think I'm going to move back to Kit for the newsletter stuff as Ghost is simply not evolving, which I find strange given the capabilities the team now has with AI. I'll have more to say about that later.

P.P.S yes I wrote this entirely by hand, one word at a time, like I always do. Except for that brain on AI image, which was kind of fun to generate.


It's not hype, it's happening

Get a premium membership and enjoy timely, level-headed videos on the emerging AI industry.

🥷🏽 Notes From an Imposter Programmer

I taught myself to code in 1998 and within 7 years had a client list that included Google, Microsoft, Starbucks, Ameritech, KLA-Tencor, PayPal, and Visa. In 2014 I decided that I really needed to understand core Computer Science concepts, so I dove in, using the free resources from MIT and Stanford. In 2016 I shared what I learned with The Imposter's Handbook.

Read more from 🥷🏽 Notes From an Imposter Programmer

Something I'm trying to remove from my life is watching other people give their hot takes on something and substituting their experience instead of FAFO for myself. It's entirely understandable why people do this: there's a lot to know in our industry! Keeping up with everything is nearly impossible. That said, I think there's a threshold where it's our actual duty to try something for ourselves to see what we think about it, rather than repeating some influencer on social. Given that I'm...

You can get rid of imposter syndrome for good. It's possible!

I was telling a good friend about my new coaching/workshop thing I'm trying to get off the ground (ridding yourself of Imposter Syndrome) and they asked me a great question (paraphrased): This sounds like a slam dunk for people dealing with it, but are you offering a practical solution with steps or just inspirational stuff? How can you guarantee such a thing? This was in response to my offer: I want to help you break through your personal barriers and transform into the professional you want...

I have a career coach that is worth every penny because he pushes me to do the things I want to do, but that I resist doing for one reason or another. These aren't small things either - they're big, consequential, possibly career-changing things that freak me out. Last week he suggested I do something that pushed me right out of my comfort zone and into orbit (paraphrased): Face this fear of yours directly and write the page. You're worth it, and you can do it, but only you can figure that...