Raw transcript — "Pi Coding Agent Observability: HTML Specs with Gemini 3.5 Flash and GPT Image 2"

Auto-generated captions (yt-dlp), cleaned via vtt-to-text.py. Timestamps preserved inline.

What's up engineers? Indie Devdan here. So, you're planning a new feature for your product. The first question is, what type of spec should you use? You remember the viral post on the unreasonable effectiveness of HTML put out by anthropic engineers? And you remember that OpenAI released a benchmark gapping image model that generates nearperfect images. So, what should you use? Markdown specs, HTML specs, or something else entirely? The answer here is the same as it's always been. More useful tokens outperforms fewer useful tokens. The key word here is useful. As you and I build more engineering agents and product agents, the question isn't just what's better. The question is, what's the trade-off between performance, speed, and cost of this agentic solution? I've got three Gemini 3.5 Flash PI coding agents and a PI observability dashboard that can help us answer that very question. We're going to test three unique specs,

[00:01:00] markdown, HTML, and an enhanced HTML version. And here, this is going to enable us to measure the trade-off triangle of three different spec types with the exact same prompt. You can see our agents getting to work here. Every event, every turn, every tool call is coming in with every single agent. Here, we can see everything. We can observe everything. And that means we can improve everything. I've been engineering for over 15 years now. And some rules of engineering never change. This is one of them. If you don't measure, you won't improve. In this video, we measure the tradeoff triangle of these three different types of specs using PI agent observability. Agent observability is simple and this application is just simple. You have a centralized server that you stream events to. You have a UI that reads those events and then you the engineer the developer can intake the information creating a closed loop structure where

[00:02:01] you can then improve not just your engineering agents in the terminal and in your software factory but also your outloop products. Observability is essential for understanding the performance, speed, and cost trade-offs for both your engineering work, but especially your out the loop products, right? Your product agents. Our two agents have already completed their work, and you can see the associated costs with that. Surprisingly, our markdown agent has used more tokens than our HTML agent. And our HTML agent has used a little bit more context. This may be variance within the model, but it could also be how the specs were written. themselves. Our markdown agent used a total of 29 turns. So, it could just be that the markdown agent did a much better job of understanding the code base. This is one of those things where if you don't actually look at what your agents are doing, you'll never know the key differences between any prompt that you're writing. VPL agent looks like it just finished here. This is our VLAN agent. This is our HTML plan agent.

[00:03:01] Here's our markdown planning agent. This agent used 25 terms. HTML agent used 17. And this one used 29. So what exactly did our agents just plan out? They each created their own spec. And so they're working on this application called Steelman. Let's go and just open this up. So this is where we dig into our product agent. Classic engineering agents where you're in the terminal or you're building your software factory, your ADW, so on and so forth. These are great. You and I as engineers, we're going to be using these a lot. But the next kind of very important agent that you and I are building is our product focus agent. We're going to present a bull thesis for some company, for some stock. Here we're talking about Apple as an underappreciated AI distribution winner. And this agent is going to do everything it can to counter this, right? It's going to give me the bare thesis of everything. So, we're going to start the Steelman agent here. It's going to generate a bunch of UI that aids its barecase argument. The whole idea here is to strengthen your thesis as to whether you should invest, you shouldn't invest. It's really here to poke holes in what you think is right or

[00:04:01] what you think is wrong. Agents of today are very cyclopantic, right? So, they're going to by nature, by default, tell you what you want to hear. Some of the great engineering models are a little better at this. By default, these agents want you to be happy with the results. So, they lean into everything that you're saying. Here's our executive summary coming in here on why we shouldn't bet on Apple as a AI distribution leader. Great response. Here we have generative UI components that were generated here on the left. Back to our engineering agents. What are these actually doing? These are going to create three new generative AI components for us to use. Right? So a quote, a catalyst timeline, and a valuation gauge. They're going to add that to our existing set. So this is our table bar chart that our agents can populate arbitrarily with any information. We have scorecards here. And then we have raw HTML. And the whole point here again is to steal man, the opposite side of whatever you are saying. Our starter prompt here, you know, we said Apple is an unappreciated AI distribution winner. We asked, you know, how valuable was that Mac Mini

[00:05:01] claw trend to Apple? And then it's breaking it all down here. You know, we can dig into that specific clause here. Deconstructing the open clutter claw trend. This was confirmed. However, the financial and strategic value of this to Apple must be heavily qualified. Very interesting here. Mac Mini represents less than 2% of Apple's total corporate revenue. Not a huge impact, right? Not as big as you might think. There's some irony to this which the agent points out. A lot of people miss this in betting on Apple. The claw trend proves that developers want a decentralized local open- source AI that operates independently from anyone, right? Just because Apple's delivering this doesn't mean that selling this cheap, low margin silicon box is going to be good for Apple over the long term, right? It's a one-time purchase. What Apple is looking for is like repeat revenue, right? And any company's looking for that repeat revenue. But anyway, we have a decent tool, 40 references. So, it didn't just make this stuff up out of its ass. It really did research via tool calling and then it made references and put together a proper bare thesis for us. Okay, so I

[00:06:00] built this just to really showcase the idea. Well, first off, cuz this is actually valuable, but second off because I wanted to showcase this idea of a product focused agent. If we hop back to our agent observability, we can see that our steelman agent has executed. Okay, so you can see everything going on inside of our product focus agent. Right now, we're in the swim lane view. We can go ahead and switch to our single view and let me click our Steelman Asian and we can see a great breakdown of everything going on here. Uh we got that cracked Gemini 3.5 flash tokens per second speed. So when you're looking at selecting the model for your product, you're going to want something like this. We have a cost breakdown. We have tokens per second breakdown and we understand the context of the agent that we're running. You can see here very clearly Gemini 3.5 flash running. We can see everything that happened. All right, so bunch of research happening here. There's the start. Here's the end. And you can see all the artifacts generated, right? And of course, we can click into anything, see the exact tool call arguments. We can see everything that built our generative UI components, assistant

[00:07:00] message, we can see the results and all the thinking, so on and so forth. You know, classic agent observability. If you don't measure what your agents are doing, you cannot improve them. This is just a fact. Um, a lot of vibes out there, a lot of vibe coding going on out there that will get you to a certain level of very, you know, lowle performance. But if you want to go all the way, if you want to truly agentic engineer systems, especially in your products and services, but also for engineering work, it's very valuable to understand the tradeoffs between the trade-off trifecta, performance, speed, and cost. Just a quick note on Gemini 3.5 Flash, having a really great time using this model. It is uh I would say near state-of-the-art. It really surprised me in a lot of cases because specifically for that cost if you're thinking about building out product focus agents the performance speed cost trade-off is really really great and I know everyone's complaining about the uh you know Gemini 3.5 flash price came in definitely higher than people were expecting. It's not so much a flash model anymore. I think for the

[00:08:01] performance you're getting out of this thing it's pretty fantastic. I'd like to think about cost per intelligence versus cost per token. What you really want is the intelligence and the performance. And if it's worth it, I'm certainly willing to pay more. Obviously, we'd all love every model, all the best models to be dirt cheap. But that's just not the case. Got to play with the cards we're dealt. So, this is agent observability. This is the core idea. If you stop the video now, I think you get the idea. If you don't measure, you can't improve. And this is especially important once you start building products where your agents are executing thousands and tens of thousands and hundreds of thousands of times over the course of some period, right? a day, a week, so on and so forth. If you can run a 3.5 Flash or a Deepseek V4 Pro and you can save and still get the results and still call all the right tools, you should absolutely do that. Okay? Like there's just no debate there. Like you should absolutely understand that and make sure that uh you have that in place in your production system. So that's the idea, right? That's the core idea here. You

[00:09:00] know, the architecture is exactly what you would think. You stream events to some server. The server also stores this into DB. So, you know, we can refresh this, get everything back. Couple nice things I built into the system. You know, I have a function mode where we can really get things super compressed. And then I have a form mode to be more visually aesthetically pleasing. We have single lane mode here in our observability system. Let's go to that steelman agent. So, we can really get some more detailed results, right? That one run took 79 events. Total 1 minute duration. You can see the cost so on and so forth. And then we have race mode which I think is really interesting. So, this is a left to right kind of race situation where you can see the events side by side. You can see every set of events chunked by turn, right? And then we can dial into these. And again, it's just about understanding what our agent is doing and when it's doing it. And of course, we can stack. So let's say we want to erase our planning agent and we want to see our HTML and V plan. And we we'll go ahead and dive into the results of these agents in just a second here. Again, agent observability is so important because in the start event, if I click into this, you can see we have

[00:10:00] all the details of this agent on boot up. I'll just pull this out. You want to know what your agent system prompt is, right? And a lot of times if you're building production agents or you're, you know, customizing your system prompt, you're going to be templating strings into your system prompt. So you're going to want to see what the real results are. Have you actually seen the full system prompt of your agent? Like do you know what it looks like? Even the pi coding agent, which I'm using a lot more now, as you know, if you've been watching the channel, this is a decent size prompt. Of course, it has all the skills that I'm adding a decent chunk of context is just showing off all the skills. You can see your session file, session ID. You're going to want to know all the tools that your agent actually has loaded. You can see our communication tools from a previous video where we talked about flat pietoi agent communication not sub aent delegation but you can see that I also have sub aent delegation ready via another pi coding agent extension and then I have this session digging capability all the classic tools and then you know tool snippets prompt guidelines and then all my skills here. You can really see here that by loading a bunch of these skills, it's also kind of bloating up my system prompt. Just

[00:11:00] important small pieces of information like that, you only understand and you only know if you're looking and if you're inspecting your agent. So, PI agent observability very powerful. And how you observe and ingest that information is also very important. Let's see what's going on in turn five here for each agent. Right? Looks like they're roughly on the same track. Making tool calls, understanding the codebase to build the spec. We're on style CSS and we're on the app. Everyone understands engineering agents. This is like table stakes. This becomes exponentially more important when you're building out product focused agents. You know, let's do a couple things here. Let's run another prompt here and then let's understand the differences between the spec markdown prompt, right? classic markdown prompt are HTML plan which is using more tokens to generate HTML which you can understand better your team can understand better and your agents can understand better and then an expanded what I like to call Vspec or V plan

[00:12:00] where we add visuals so we'll go ahead and open these up in a second but first let's kick off question here for our product um what I want to see here as I'm thinking about purchasing Apple and I want to get a good bare thesis breakdown is show me Apple's revenue venue a Mac Mini versus all their other product lines. I'll just say visually just so it's super clear. Of course, we can just dial into this. Let me go ahead and show you the live stream of this steel man, right? Doing research using tokens inside the product calling the tools that you set up for your product. There it is. Emitting the artifact. Let's go and see what that artifact was. There we go. We got a nice pie chart. And you know, Mac Mini sells basically zero. Was the Mac Mini craze really that crazy? Uh, no it wasn't. And you know, really interesting. You know, you can hear and listen to all the hype, but even with viral events like this, it's not that important to Apple's total topline revenue. Now, of course, things can change. These can grow. The Mac Mini claw craze, it certainly is a signal,

[00:13:01] and I'm sure Apple is betting on a larger trend of AI in some shape or form. They certainly need to act on that a little bit more. But just looking at the raw numbers, right? We can see our core references here, right? Apple's first quarter results, right? I mean, we can click into this. This is like legitimate sourcing, right? January 29th, you know, we're getting the latest information here thanks to our agents and our tools. Having a bear, a counterpositional agent is super super powerful for getting that opposite perspective. Anyway, back to our agents. So, we can see again, we're just continuing to trail and understand the tokens per second, which my god, Gemini Flash is a beast here. My tokens per second calculation is a little off here. I had to do some inference on this because PI coding agent doesn't come with this out of the box at least that I know of. Comment down below if I miss something. Yeah, just great performance here all around. There's another really powerful tool that I want to show you in addition to the PI agent observability system and that is GBT image 2.0. The Anthropic team put out this great post on the unreasonable effectiveness of HTML. It's

[00:14:01] about using HTML with specs, right? Give your agents more tokens and they'll perform better. That's the top line for this, right? more useful tokens. I've been using and leveraging that idea in a different way, specifically with images. Let's go and just dive into our plans here. So, we have our markdown spec here. We have our HTML spec and we have our HTML vspec. And you'll notice something here. The VSpec contains images. All right? So, you can see very, very quickly where this is all going. And so, you know, classic markdown spec, tried and true, simple, best for preserving tokens and having a text focused result. Okay, so this is my classic spec planning prompt. I have templated my engineering into these sections, right? Task, objective, problem, solution, relevant files, blah blah blah. Classic stuff here. We're building these three new types of generative AI that our steelman agent can respond with. So, very important stuff there. Let's go and see another version. Okay, so let's talk about HTML, right? What does that HTML look like? Just want to get the reference to this and I'm just going to fire Chrome on this. Here is my HTML version of that.

[00:15:01] Okay, and so relatively bare bones. As you can see, I'm not using a ton of tokens to use every HTML tag under the sun. But you can see here, we're getting a more visual rich portrayal of very importantly here, our UIs. With the HTML spec, my agent is starting to dial into what the HTML components look like. Here's our quote component it mocked out. Here's our timeline component. And then here is our valuation gauge that it mocked out. And then a couple more notes. So, I could be doing a lot more here with the HTML spec. you can push this really really far and be more prescriptive. Basically, I just have my same markdown spec in HTML which gives it a few additional capabilities to communicate the information in a more accurate way at the cost of tokens. Interestingly, if we go back to our swim lane and we drop steel man and we drop VLAN and just compare our markdown and HTML spec, now we can see very clearly markdown actually used more tokens this run. It used 170 events versus 100

[00:16:02] events and it used about the same context. Okay. So, what happened here? Is this just variance? Is this our agent being more focused on the actual plan and doing more research? Assistant doesn't have to think about outputting in markdown. It's unclear. But now we can see right now we can understand that there is some difference here. And obviously the right thing to do here is to turn this into an eval. run it over and over and over again to look for the average performance results. And you know, hint hint, that's what comes next after you have observability. After you're building out your product focus agents, you're then going to want to scale your repeat work into eval. But step one is just measure. So that's what we're doing here. Okay. So I'm going to do Chrome on our HTML V version. Open this up. Very similar version except we have images embedded. We're using that powerful GBT image 2 model. And per plan now you can embed images to enrich your planning experience not just for your agent. Our agent is going to read this.

[00:17:00] It is going to execute this. We've updated our build prompt to say if there are any images inside the plan you must read them. Image tokens are very useful for your agent especially very powerful multimodal models like Gemini 3.5 flash. If we jump down to MMLU and all these multimodal benchmarks, this model is going to be really, really great with these Vspecs, these visual specs, these HTML specs, these image specs, Gemini wins on multimodal. We know for a fact that not only will this improve your agents, but the more I plan with Vspecs, visual specs, it's so helpful to see interfaces to spec with and images to spec with, right? I can clearly see exactly what my model's thinking in a visual way. And this is all possible. I think it was nearly impossible before the GPT image 2 model. Now this is a reality. Let me just make this super clear. All my plans now are VSpecs. They're visual specs. Now HTML is an enhancement to that. Sometimes I use HTML. Sometimes I just use markdown that reference images. But I am always planning with images now. And of course

[00:18:01] this is more expensive. You know something that our observability tool did completely miss is that it's not capturing the cost of generating each image. I mean this actually costs1 or $2 or $3 total to generate all the images. But you can see here that this experience of a gentic engineering with this type of plan is superior. I mean it's just better. I'm quickly understanding the plan. Okay, nice. So thanks to the HTML spec portion of this. We also have a free form breakdown part of this plan where we're actually writing out what these components will look like. So same deal, right? We're creating steelman generative AI components. Here's a quote. Here's a timeline component. And here's a valuation marker. So, I didn't get it perfect, but you can see here it's even a little bit dynamic. We can play this a little bit. And this is where HTML specs really shine is that you can get some like demo proof of concepts for specific parts of your work. Even if you're doing backend work, DevOps work, product work, and of course UI work. It's nice to have HTML prototypes rendered into your plan.

[00:19:02] I'm using visual prompts a lot more and I'm throwing in some HTML where it makes sense right especially for front-end work HTML specs very powerful but a lot of the times text is enough in agentic engineering if you're doing it right you're probably stuck in one of two places right there are two constraints of agentic engineering it's planning and reviewing these types of visual specs these HTML specs they really really help with the planning side of things and it just comes down to the fact of do you know the performance speed cost tradeoff off of using each one of these different plan types with each different prompt you're comparing against and skills and agentic workflows and software factories and so on and so forth. It all comes back to your ability to truly understand the internals and the uh trace of how your agent got to the result and what it cost them to get that result. Agent observability is at the center of this. I myself need this for engineering but specifically for product work that I have upcoming that I'm working on. I'm

[00:20:01] getting into that really important phase where I need to compare the performance of different models. Not just on can they do the job, but how quickly and how much does it cost. This is the trade-off trifecta. You need to understand the tradeoffs, right? Engineering is all about making the right trade-offs for your work, for your products, for your customers, for your users, for your company. Again, just one more time, you know, there's a million observability tools. I'm not trying to sell you on mine. I'm just trying to communicate the importance and the idea here. If you don't measure it, you cannot improve it. Uh if you like this video, if you like this idea, like, subscribe, join the journey. With a single PI coding agent extension that we can pull up right here, extension PI observability and with a single user interface, I'm able to now capture all the events of every one of my agents that executes. And again, you know, refocusing on the PI coding agent, this tool allows us to build and use extensible software. It allows us to build our own custom Asian harnesses and customize them just like

[00:21:00] we need to. This really helps us get at that idea of understanding the uh tokconomics of the work we're doing. In our previous video, we talked about the single most important thing for senior engineers to be focused on. And this was one of the key ideas. The first step here is to use more tokens. The next step is to generate value from the tokens you've created. And then the final step is to actually arbitrage your tokens is to capture the revenue, right? capture the value that you're generating usually in the form of of cash, right? Revenue from your product. This is the agentic value chain, right? This is tokconomics, right? You want to move up this chain. You don't want to just be stuck at the bottom here token maxing. You want to move upwards. And the only way to do this, right? We talked about this last week and we're doubling down on this idea. Everything is connected here. This channel is a chain of steps to become the greatest agentic engineer possible. We need agent observability to help us get to this place where we can arbitrage our tokens. It's not enough to be able to just run a bunch of agents, right? That's cool. Great place to start. Terrible place to end. We need

[00:22:00] valuable agents and then we need to capture the value of the agents. Step one, use the tokens. Step two, generate value from the tokens. Step three, capture the revenue of the value generated from your tokens. I have this on now in every single agent by just plugging in this extension where we're building slices of composable customization thanks to the PI agent harness. I'm stacking up these extension as I mentioned, you know, and this is true. Uh maybe you can tell from watching, you know, videos week after week. I'm building a new agent harness every single week. I'm building a new slice of composibility that I'm adding. Some are hitters, some are losers, right? If I show you my terminal here, right? I've got my PI communications network here that has been super super valuable, right? Paying agents via com system. Say hi. This has been ultra valuable. Any agent can coordinate with any other agent and this is just on all the time, right? So you can see there my agents talking to each other. This is a flat agent communication. I'll link this video. You can see the responses came back. And this is just, you know, one additional advantage I'm stacking up for

[00:23:00] my agents and I want to share it all here with you on the channel. So links to the PI observability codebase is going to be in the description for you. We didn't go into too much of the prompt details because it's here for you. If you look at docloud skills, you'll see HTML spec, you'll see HTML vspec, visual spec, and I have my markdown vspec prompt that I use to generate markdown prompts with images. So, still generating useful, valuable tokens inside of the plan, inside this codebase. We'll have the extension here. And of course, you'll also have access to the Steelman web UI. So, you can just test this out. Um, you know, it's a starting place. I highly recommend you think of every piece of code you see, every open source project, everything you're intaking as an option. With the option, you can take it, you can adapt it with your agents. Take it, make it your own, improve it, do your thing with it. I'm going to do the exact same thing. I'm going to specialize the PI observability UI and agents and push them uh to really serve the exact use cases I need solved. If we look at the future of where this is going, I think a lot of things are just going to be the same, but a little better and a little

[00:24:00] better and a little better. Specifically, when we talk about specs, it's clear that multimodal specs, giving your agents text, image, audio, video as part of your plans is going to be more and more valuable. And the leader here is going to be the Gemini models, right? They just have the advantage on multimodal. I mean, they have YouTube to train on for God's sake, right? The advantage is going to remain there. It is pretty clear that uh these models are starting to specialize and really start carving out their own areas of expertise. you know, Gemini 3.5 flash top tier model. Pretty impressive that OpenAI is doing a GBT 5.5, you know, pretty cracked engineering model and they're still putting out insane image models. And of course, Gemini put out that new Omni model. So, you know, lots going on, lots more to see. This is our bread and butter here, guys. We ingest every model. We ingest every aentic engineering idea on this channel and then we synthesize to make it useful and valuable. None of this stuff matters if you can't build it into your work for you, your team, your company, and

[00:25:01] ultimately your customers. If you want to win, you got to measure this stuff, especially if you want to get your agents into products. You want to balance the trade-off trifecta, performance, speed, and cost. You don't just want to spend tokens. You want to spend tokens that are valuable, and then arbitrage that value. All right, you know where to find me every single Monday. Stay focused and keep building.