When people do code development, we’ve made an unconscious trade-off. The person writing the code is the knowledge base. We may have a few tests and docstrings or readmes / wiki’s, but let’s be realistic. In most organizations, prior to large scale LLM deployments, these were best effort, usually up to date at version 1.0, and then subsequently rot. Training on a codebase consists of either inspecting the code yourself, or sitting down with engineers who’ve worked on it.

With agentic development, we need to make a conscious decision to build a system that can manage the knowledge of the agent across all its work. This starts with the history of decisions an agent makes: what decisions it made, how it made them, and the composition of the agent.

What data do you want for your agents writing code?

Let’s start with a simple example. You have code in a git repository. The commit history represents the changes that have been made to a repo over time. Specifically, history has the files that changed at each commit hash, and a message of varying quality.

With a coding agent, you have the representation of the agent at a point in time (the model used, instructions, tools accessible, etc.), and the sessions or history of the agent working on the code.

One of the first things we need to do is think about the metadata to track. For each commit our agent checks in, we minimally would want:

• The git commit SHA the agent made

• The version or identifier of the agent at a particular point in time

• The actual log of the agent session that built the code

Git generally has some constraints about the payload sizes that we could check in to a repo, and logs are a hard proposition because these can grow very quickly with the amount of work agents can do.

However, for commits, we could enforce that we add metadata about the agent that we could use to look up agent components as needed. This is a simple but good starting point.

We also need to think about the observability of the system the agent is building. Code is the current ‘plan’ of the system. We also want the ‘as works’ version of the system so our agent is better at understanding how its actions impact the resulting system.

Creating an agent lineage system: from agent to code to deployment

Like many ‘agentic’ things, this example is really focusing on the lineage of an agent. We do similar in data management today. In row level lineage, we add identifiers to trace a row of data throughout a system, including transformations and aggregations. This way, if something goes wrong, we can trace back to a source system or step. However, this type of lineage is the most expensive. We may choose to only do column level or dataset level lineage if we can sufficiently see data suppliers, inputs, transformation steps, output, and customers at one of these levels for fixes.

Let’s bring this to a more realistic coding agent example:

We need to trace the agent’s lineage from ‘what’s running’ backward to the specific agent code that was used for code commits. Minimally, we need to leave enough breadcrumbs, or identifiers, so we can tie each stage of the system together.

From code to deployment, this may be metadata for a container that contains the specific commit SHA that the container was built from, or a git release tag.

Working back, we may need to store the agent’s SHA as a part of the PR or other code history. And for each agent session, we need the version or commit SHA representing the configuration and design of that agent at that specific point in time.

This is somewhat easy to do in this particular example because we’re talking about 1 code commit. But part of the point of agentic development is how we can do this at scale. This is where we need a system of record. Luckily, we’ve been doing this for a while in data engineering; we need to bring some of these principles to agent building also.

Build an agent warehouse for observability and scale

An agent warehouse is simply a data warehouse but for the purposes of managing agents, and in this case, the artifacts they create and manage. You could use any other type of datastore. For this, we’ll keep it simple as a database where we can store all of our information, including unstructured logs.

We use writers that sink our metadata and transaction data from each of our primary sources. These could be things like:

• Skills and MCPs for our agent to write out telemetry to a specific location, or a logs collector

• Git hooks and syncs to store our code

• Using Cloud Observability suites to get our deployment logs and data

Once we have all this data in a common location, we can start applying transformations to this to bring together different attributes and build our lineage for specific commit SHAs or agents. This is highly useful:

• First, by having a full span, we can see the impact of the agent’s behavior on the end system. This allows us to understand the downstream implications of agent behavior.

• Next, by understanding the behavior, we can further tune our code agents. This may be something like adding or modifying agent skills, or even fine tuning a model based on our specific code usage.

• And last, by entering the virtuous cycle of improvement, we can refine the agents to ultimately write and ship better code in the process.

For different code bases or agents, your transformations may vary, but they will likely follow a similar flow where you build lineage or spans, and then analyze the information. Here, using an LLM to rate spans and provide critical feedback is a good pattern to start with.

The limits of git when storing agent data

When you’re getting started, I actually think it’s pretty easy to just package your agent information and skills into the code repository you’re working with. For a 0 to 1 type pattern or prototype, this keeps things simple.

But the issue with this approach becomes scale. While you want all agents to have a common understanding of how to work with a specific repo, you also have agent specific information for how an agent should act across all code bases. In this example, you’ll have some coded aspects of the agent, and also knowledge the agent(s) accumulate over time. You have to design a system (such as the logical design above) to plan for this.

Implement agent lineage

I started working on a localized system for this using a session journal skill. My colleague Guillaume also has a great example of analysis on local Antigravity logs. I encourage you to see how these approaches are treating summarization with LLMs.

I’ll be publishing more code examples on these types of patterns in the coming weeks, but take the time to learn data engineering basics in the meantime as managing state in agents is increasingly one of the biggest bottlenecks to scale.

Happy Tuesday!

Share

When I originally come up with the ‘What I built this ….’ concept, it was about sharing things I built that I thought might be useful with the community make AI and cloud computing things easier.

This week, it’s a bit different of an article. I’ve been really into local and open models. I think they have a lot of potential and may represent the future of where AI workloads run, in particular at the edge. But I also ran into a lot of struggles in a recent folly, and I’ve chosen to document them here for everyone as learnings I’ve had.

The goal: rebuild the Samsung Tiny Recursive Model pattern with Gemma 4

Say what you want about the goal, but I’ve been fascinated by the Samsung SAIL team’s paper about how a 7 million parameter model can outperform an LLM at problem solving tasks.

So I set out to try to build a similar pattern, but using a 'smaller’ Gemma 4 model with it instead of a custom trained model. In this case, Gemma 4 2B.

The code based development for this actually wasn’t too bad, there are a few gotchas that I’ll highlight, but a lot of my pain came in re-discovering things in local and open AI that became large blockers to success. You can find my code here.

The RAMpocalypse is real

If you’re on Mac, LM Studio is by far the easiest app I’ve found to get up and running with local models. It abstracts a lot of the packaging and runtime choices you need to make.

Getting started with Gemma 4 is equally easy. You download the model, then load it into memory to get started.

Here’s the hard part. All of my machines are either 16GB unified memory (aka Macs) or 16GB RAM mini PC’s. I have one gaming computer with a Radeon 9070 we’ll talk about here at the end, but the point was to be able to run models in my office, and everything except said gaming PC is there.

I quickly discovered that any model north of about 8GB size will make all of these computers come to a crawl. At one point, between my coding IDE, loading a model, and Chrome, I had a 48GB swap that MacOS was valiantly trying to manage, but just couldn’t do it.

Development is simply difficult with a 16GB device, and it’s highly limiting to scaling in communities. I haven’t touched fine tuning much, but at one point it took 90 minutes for a single tuning pass.

Insight: Anything serious with models requires an outsized investment. People who are running things locally today have already invested and have capacity to grow. Those who want to learn will be constrained by the physical limitations of hardware.

Learning: I’m planning to do a lot more in Q3 about how to learn with local models given the limitations I’m discovering to make it more accessible for everyone.

AI Runtimes and configurations remind me of the early Hadoop days

Let’s assume we’ve gotten the hardware problem fixed. The next question is how you’re going to run models, and what configurations matter the most.

Ever heard of ROCm? I hadn’t until going down this path recently. The runtime community is improving rapidly, but the experience is incredibly uneven across hardware types and suites. In a non-NVIDIA stack, the variability and compatibility of software to hardware to models is all over the place (see recent results from people trying to use the newest Intel B70, a value 32GB but with a lot of software compatibility issues).

Want to run a model with MLX on MacOS? The model you download is a different type than GGUF used elsewhere. Want to deploy whatever you built? You’ll have to make sure you understand where it’s going to be run in order for it to be compiled properly.

These are just two choices. If you consider iGPUs and CPU model runtimes, you’re probably going to need llama.cpp but need to learn all the settings.

Coming back to the header, this reminds me when Hadoop was young. You had to know the specific version of file types you used, trade offs of file types, the runtime that was used to build them, and what use case you were actually doing to pick the runtime to use for your workload.

Insight: This market is immature, but luckily it’s evolving quickly. Understand where you’re going to run all this in advance, pick common components wherever possible, and be acutely aware of the tradeoffs. Use LLM’s for research.

Learning: Similar to above, there’s just not much out there for someone to get started or go from 101 to 201. That said, the community answers are really good when you know what you’re looking for, but if you want to try this at home, pick the most popular hardware and software to make your life easy (do as I say, not as I did).

Evaluation is still the big question

Hardware and software learnings aside, I’m coming back to the hardest thing when doing this work I’ve found, which is a good eval dataset and training / test data to support it.

In the recursive models example, I’m using HumanEval, but the model tuning is struggling with the iterations required given hardware constraints, and I find myself asking more philosophical questions about what a ‘good eval’ actually looks like.

At one point, when I did a test, the model changes went from 0% to 100% passed on a sample of evals, but I’ve been peeling that back to make sure there’s no overfit on each pass to a question or test set.

Learning: I’m planning to spend a significant amount of my time in the coming weeks getting in depth with some of the leading and popular evaluation frameworks to build my own understanding of how they work, what they are actually measuring, and the input datasets. For people and organizations building agents and models, this is the critical piece.

Meanwhile, DiffusionGemma is released

While I was doing all of this, Google (my employer) just released DiffusionGemma 4, which is aware of leading and trailing tokens in a prediction. This might have been the best approach to start with over my recursive one, but I’m going to keep going and see how far I can do this.

The next steps in my local AI journey

Now you could probably also ask ‘does this person know what they are even doing?’. It’s probably a fair question. I’m a believer in learning by trying, failing, and trying again. I've found my questions and searches are getting better, and I’m getting precise about issues. But I’m less than 1% in on this journey, and while it’s been incredibly frustrating, it’s also surprisingly rewarding.

Remember that gaming PC I mentioned? Well now I’m looking to get a Radeon R9700. 32GB of vRAM seems to be a good buffer size active parameter models and models below about 16GB in size, and I’m curious how it turns out.

Could I do all of this in the cloud? Yeah, but that’s not really living most developers experiences now, is it?

Hit me up with things you’ve found useful learning about all this so we can all upskill in this process.

Share

So why is tracing so important now with AI?

An example of two requests

Let’s say there is an AI agent (a coding agent, support agent, doesn’t really matter) that intakes a request (intent) from a user, has an internal agentic loop, and ultimately should satisfy the request (intent).

This is a very simple example of the flow, where the agent is performing work. Let’s assume that you’ll have multiple users or requests that match intents. Ideally, the flow of each session looks like this:

This would be an ideal state where similar or matched intents, when clustered together, yield similar or matched outcomes when the agent does work.

However, when you factor in model non-determinism, various tool calls and definitions, and other things like specific context or data the model has access to, this ends up looking something more like:

So how do you figure out what went wrong? We could observe some things our agent may have performed like tool calls out to other API’s or databases, but we need to look at the trajectory and the reasoning chain that yielded a particular good or bad outcome, analyze them, and then modify our agent or other components.

Why does tracing help?

In the example above, we need to see the internal and external work the agent performed, the timelines the work was performed in, and in what order the work was done for a particular intent or outcome. Tracing gives us the ability to string all of these together. Even better, we can enable tracing that gives us the full prompts and responses for each turn of the agent so we can see, in detail, what actually happened and when.

In 2019, this was a nice to have, but this is becoming a requirement to budget for and implement when working with agents. Not only does it help for building better agents, but it improves the observability for things like regulations when we need to show why agents perform a particular task and what they did.

Tracing applied to our example

Let’s assume we’ve added tracing to our agent example. You can do this with OpenTelemetry and built in extensions in most agent frameworks (Google ADK example here).

Now, let’s assume we have two users that asked a question about their bill last month, and why it was so high.

Using tracing, we now have access to more data points. For example, we could see that for one customer, the agent may have properly found the customer’s information, but the bill for last month had not been processed yet. As a result, the agent only had access to information that was two months old, and the resulting outcome wasn’t correct due to data access.

In another example, we have a coding agent. In one flow, the coding agent had access to a repository but chose to use grep to search for the changes required instead of using an AST or tree based tool to find functions. By having access to the internal thoughts and steps, we can determine if, in this example, we need to further tune our MCP tool definitions or even our skill so the agent is better equipped for what to use and when.

And, we can use LLMs to improve LLM quality by providing all of this information to agents with specific tasks.

Implementing tracing

In the observability stack, access to real time traces, metrics, and logs are table-stakes for agentic development and runtimes.

Look in the frameworks you’re using to start seeing how you can implement the default tracing abilities. Keep in mind that a substantial portion of agentic system cost is actually on the observability and control around the agent, not the agent itself. Budget accordingly.

And last, make sure you’re thinking about data security. When you are exporting traces, in most cases you will need the full logs to figure out what happened. But that doesn’t mean you should necessarily give everyone or an improvement agent access to personally identifiable data, and you should be aware what you can use for training and improvement in your customer agreements and regulations (ex GDPR).

Hope you enjoyed this weeks post!

Share

So my goal this past week was to build a service that could take requests centrally, route them out to local models, and pull them back to the user. Why? There’s a ton of examples out there calling a local model directly, but when you have a fleet of models and agents / users that can call them, you need some way to manage the chaos and deployments. Smartrouter provides logging and monitoring for these calls, and we can extend it to support ‘local AI’.

So how did I build it? First, research about how I should run local AI.

Considerations for running local AI

This is by no means an exhaustive list but incorporate some of the things I’ve thought about and read up on as I’m getting up to speed on local AI, keeping in mind the goal is concurrency and increasing the number of requests a cluster could serve for a particular model type.

First thing I’ve been reading up on is how to run local AI in clusters. This isn’t so much an activity like database sharding, more like how you can run multiple services. And there’s complexity here where if you try to run a large model, you need to take into account how to shard a model across things like different GPU’s based on vRAM size. In the example of concurrency, we can scale up here with faster GPU’s or storage like Lustre for model loading. But if we’re constrained by something like cost (so no GPU usage), or limited to the devices we have (I have 2 AMD Ryzen 5 NUC’s), that doesn’t really work. I’m looking to scale out.

So, in this example, what I really want is a fleet or pool of workers that take items, work them, and put them back. And there are a number of great existing examples of using queues and workers to complete tasks. So I decided to implement this pattern in smartrouter for local routing.

How the local cluster works in smartrouter

Smartrouter allows you to register machine runners into a ‘local cluster’, which is a logical service representation. This contains a queue for worker requests, the models available to serve, and any additional queueing or execution limitations.

Next, you can use the existing router architecture to route requests through smartrouter to the local clusters. This includes dynamic routing.

Here’s an example of a request that does a few things: first, it routes to a local model. Next, the router actually changes which model the request gets sent to based on the routing rules. Then the runner picks up the request from the queue, works it, and publishes it.

A runner in smartrouter is like a machine agent. It registers itself with smartrouter and uses llama.cpp to pull down the model to run. I’ve created an example using Kubernetes pods that also allow you to run pure CPU or GPU inference, and I’m working on a Mac and Linux version next (the Mac code exists but I haven’t been able to give it a thorough working through yet).

We can also test smartrouter local agents by creating a key just like any other app registered and routing to the appropriate ‘model’:

curl -s -X POST "https://gemini-smart-router-your-url-hash-uc.a.run.app/v1/models/gemma2:2b:generateContent" \
      -H "x-goog-api-key: gr_local_cluster_verify_key" \
      -H "X-Client-App-ID: app-local-verify" \
      -H "Content-Type: application/json" \
      -d '{"contents": [{"parts": [{"text": "Write a 3-word quote about coding."}]}], "generationConfig": {"maxOutputTokens": 10}}'

And get a result back:

Target Endpoint: https://gemini-smart-router-your-url-hash-uc.a.run.app
------------------------------------------------------------
💬 Gemma: "Code, debug, deploy." 
💬 Gemma: **Code, create, solve.** 
💬 Gemma: "Code it, fix it, thrive." 
💬 Gemma: **"Code, Create, Connect."** 
💬 Gemma: "Code makes reality." 
💬 Gemma: **Lines, logic, build.** 
💬 Gemma: **Code. Create. Conquer.** 
💬 Gemma: **Write code, build dreams.** 

Feats and pain using AI for development

Feat: Once we defined the desired architecture, LLM’s are pretty good at writing kubernetes specs in particular and running a playbook. When we got to this point, the process was a lot easier.

Feat: The rework cycle is dramatically shorter. I explored some earlier concepts about clusters and runners, and while they worked, the data model needed some heavy revisions to support scaling. LLM’s make this work significantly faster, to the point that you sometimes have to make sure you didn’t throw too much out the window.

Pain: Definitions really matter. In the first iteration of my local implementation, I called the machine agent an ‘agent’. And that created a number of comprehension problems for the LLM where changes would be made both on the frontend of smartrouter and in the machine agent, which I obviously did not want. After a few rounds of this, I went back to the drawing board and defined very clear and different terms in the application’s terminology. This helped the LLM generate better context for each of my reuqests going forward.

Pain: LLM’s still like to tell you it worked even though it didn’t. When I do iterative development, I like to use test driven design, but even that has flaws with the human. The tests look fine, but LLM’s I’ve found have a tendency to try to mock services to pass tests rather than run the request to the ground. I also ran into a few instances where a ‘200’ was not in fact a success. My learning here is to continue to build out my agent skills on testing so its clear what my expectations are, in this case no mocks in this code and real logs to prove it.

Next steps

Download and try smartrouter here: https://github.com/davenportjw/smartrouter

Let me know what you find / excites you / frustrates you!

I’m planning to extend smartrouter next week to support more dynamic routing requests and implement more analytic capabilities for routing. Cost in particular is a vector I hear a lot about, and mining current information for future learning is top of mind in a number of areas. Look for more on this in the next few weeks.

Also, I’m exploring cool concepts others have in this space, for example Matthew Tracy’s implementation of a router here: https://github.com/MatthewTracy/yagami

There’s a lot of good stuff here.

Share

The internet era question for skill building quickly became a question of ‘how do I Google to learn this?’. Those that are good at searching for the answer built a system where they can quickly sift through different materials, links in this case, to find the information they need. One of my favorite interview questions was asking someone how they search for something they don’t know, and seeing the thought process for how people reach the desired answer.

When you combine this with good rote understanding, you had a superpower for executing work. But in the AI era, how much of this is still relevant?

The key question now is ‘do I need to learn this’?

LLM’s can return responses back to us faster than anything to date, and in increasingly high reliability. I’d argue that the main question now isn’t ‘how do I find the information’ but ‘do I need to learn this information’? Being able to situationally apply this continues to show how high performers continue to outperform with AI, and it builds on the first two skillsets of architectural understanding, and the ability to know how to find the right information.

Take an example where you’ve asked an LLM to write a new feature for an application based on a prompt. There are hard skills we’re all continuing to learn about prompt and context engineering. But the real challenge comes in managing the response.

In the example prompt, we may ask the LLM to create a new login feature for a website. The LLM can build any number of pieces of this including authentication patterns, deployment patterns, and things like jwt tokens to make sure a user can call other services. But let’s assume we’ve never worked with login and auth before. What of this do we need to learn, if anything? This is the new challenge when we can create faster than ever.

My learning plan

Understanding what to learn is quickly becoming a new and desirable skillset, in particular when you have do this rapidly. Here are my thoughts on how I do this:

1. Being a consultant at heart, I’ve always valued the T shape of knowledge. If I’m doing something with an LLM, I usually try to minimally understand the high level concepts. So in this example above, I’ll read up on basic auth patterns or ask the agent to explain its coding behavior to me.

2. Next, I usually ask the question ‘do I need to do this once, or will I need to do this repeatedly’. This question helps me prioritize if I need to move on to increase my knowledge depth.

3. If I need to increase my depth, I’ll typically keep asking the agent or move back to ‘Googling that’ to find some informative articles. I leverage adjacent skills in my ‘T’ to increase my knowledge depth. For something like authentication, I will cross apply domain knowledge on testing to create different conditions.

4. If I need to increase by breadth in a topic that I don’t know but should, I find myself using more NotebookLM, or Youtube videos on in the background to read up. Some of these are fun topics like doing home work where I need to get better at breadth in order to improve my ability to prompt. The generalized T here does still go a long way in providing incremental value to LLM usage.

Applying this in your work

With AI I think it’s important to highlight that, just like ‘Googling that’, there’s no single strategy that is the best, although there are probably 5 to 10 that work well. A lot of this comes down to practice and repetition. In AI this can feel exhausting with the amount of information you have to go through in much shorter windows. It also feels somewhat evolutionarily opposed: we still need 30 days to build habits, so use it but remember this is a marathon, not a sprint.

Hope this is useful! Happy Tuesday!

Share

• How to route calls based on more of a central governance model than letting any developer pick any model for their app

• How you could dynamically route based on some complexity map

• or How to control some level of app throttle (including dealing with 429’s from specific models)

So, I built the smartrouter app over the past two weeks, and I’m intending to extend it further.

What is smartrouter and what does it do?

Very simply, smartrouter is compliant with a subset of the Gemini SDK. It allows you to specify a custom endpoint and model type. Then, the call is routed to smartrouter, which runs on Google Cloud Run. It’s just a container with some services though, so you could run this elsewhere if you wanted.

Here’s the architecture:

For Gemini models, you can specify the Google Cloud region/location you want to run smartrouter in. Then you dynamically detect all of the available models in that region, the multi region of your region, and globally. You can turn models on and off for users to use.

For requests, these are routed via a gemini-dynamic model. Based on your app you configure, the requests are sent to the models you choose. All requests and metrics are logged via Google Cloud Monitoring and Logging so you can see usage based metrics and track costs in near real time.

Features coming soon

I’m a little rusty on my AWS and Azure skills, so I’m planning to extend this to other clouds and model types as somewhat of a forcing function for me to learn about the updates that have happened since I last used them a few years back. This includes extending the endpoint capabilities to other SDK types.

I also want to explore more dynamic router methods and I’m planning to make this more advanced and configurable.

I’ll respond to community requests as they come up and are within reason :).

How I used AI to do the work

Since Antigravity was updated, I've mostly used it to help with the code work. I’ll work with Antigravity on the feature I want, let it do some coding, and then validate or correct the work. A few quick things I’ve found work well:

1. I keep sessions pretty short. I define a small feature, work on it, get the tests done, and then open a new session / conversation.

2. If I need a planning discussion, the only artifact I’m looking for is a task list to handoff to subsequent sessions to keep the context window small.

3. There were a few features that the agent just struggled with. When this happens, I’ll usually try a few different approaches. Then I will go back to the old methods, which include Google search. I get a few pages, give these to the agent, and then ask it to correct the work. This pattern works pretty well.

4. As I go along, I ask the agent to update itself. This was part of the inspiration for the session journal, which I intend to extend.

Next steps

The link to get started with smartrouter is: https://github.com/davenportjw/smartrouter

Try it out, find bugs or enjoy it, and let me know your thoughts!

Share

1. Someone is told they should ‘try AI’ or has some mandate without a goal.

2. The person installs an AI tool or tries AI in a very specific flow they are accustomed to. AI struggles with a very directive prompt lacking context to the objective, and usually fails.

3. The person assumes that AI is ultimately not helpful and goes back to known ways of doing work.

Honestly, this makes a lot of sense, particularly in the absence of being able to conduct transfer learning from someone who is an expert at prompting and using AI. I was also guilty of this behavior, but I have a few friends at work and a friend personally who were what I consider ‘all in’ on AI early, and they quickly shaped my perspective of what is possible when you use AI tools effectively.

Today, let’s explore what ‘all in’ is, how you find and amplify leaders in your company who are all in, and hopefully give you some starting points based on what AI tools I’m currently using.

What does being ‘all in’ mean?

In my simple definition, being All in on AI simply means you default to using generative AI or LLMs as your partner or tool first. This doesn’t mean you’ve given up control. If anything, this definition, if maximized, maximizes the ability of you, as the individual to exploit as much creativity as possible.

Day 1 with AI

An easy way to start here is to install a general purpose app or tool of your choice (or allowed corporately), and then design your first prompt to fully ‘offload’ a task to the AI agent or model.

Your goal as you start using AI is to read and learn about the prompting strategies that work or don’t. A prompt that instructs a model to get the latest 50 results from A table and make a table from that is probably not going to be as successful as a prompt where you provide context, so what the goal is (why are you getting 50 results), and the other relevant information to accomplish the goal. The example above is representative of how I typically see people start with AI in the opening example. Many, if not most, AI tools will struggle as they lack the context to understand the request.

Moving from directive to intent in prompt strategy is a key learning. If you are using AI to build a machine learning model, consider what would happen if you gave an AI tool the prompt, context, and tools needed to build the model on its own. It will be a struggle the first few times but you’ll learn about the agent and model preferences, and ultimately be able to scale yourself far faster. You’re building your own knowledge base about how to use AI and refactoring your work.

Next, AI is ultimately a team sport. Learnings should be shared, and team member should use the same tools. Each team working on common goals should have a repo for sharing skills, tools, and prompts that are useful with the coding or AI tooling. This commonality streamlines generation and debugging where everyone can help.

Day 2+

I believe most organizations have pockets of AI usage today, even in organizations that might consider themselves behind. There are other questions about ‘shadow AI’ usage that I address in an earlier post to remediate (remember most people are well intentioned, governance needs to enable experimentation with reasonable guardrails).

The question of ‘scaling’ is something I typically give to organization leaders, and it’s not a complex list:

1. Share your AI successes and learnings. In an increasing number of companies, I find the leadership being the most extensive AI users. As leaders are more comfortable sharing successes, failures, and learnings, this will give others the confidence to ‘try AI out’

2. If you aren’t using AI, use it. Of the ‘transformation’ paradigms, I’ve found AI is one where leaders need to lead from the front.

3. Amplify the champions. Find and celebrate those in the org who are using AI effectively. Learn about what is making AI work for them, and enable governance around the flows users have found work well.

4. Become experts in your business and process. I write about understanding your corporate moat, and why value stream mapping matters in earlier posts. In reality, AI is intertwined with being an expert at your business to keep an advantage. Getting started and becoming an expert mix technical, process, and business capabilities.

My current AI flows

(As of May 2026). Most people know I work at Google so my stack is going to be weighted toward Google tools, but it would be easy enough to swap these out for a number of other options. I view these more about the flow than the tool itself, although some certainly help more than others.

Day to day AI

My two most used AI products outside of coding tools are as follows:

1. AI Inbox in Gmail. This is surprisingly good at taking my inbox and making a to-do list out of it that I can check off for the day. I try to keep an ‘inbox minimal’ approach; AI Inbox helps summarize a few threads (sometimes this is where its a little more questionable), but the to-do lists are pretty solid and span across Gmail and other Workspace tools.

2. Gemini in Google Chat. Working in connection with AI inbox, Gemini in Chat also just helps with the drudgery of punch lists to get done.

3. Wildcard: Gemini in Slides and Docs. When these work, they’re awesome. When they don’t…. well, you know. I’ve found Slides to be good with AI at basic flow charts and Docs to be good at using Gemini to rephrase and summarize with the recent updates, which does help with my conciseness.

Planning and Coding

I’ll highlight my primary workflows here and also if you wanted to swap out tools, what I’d probably start with.

Planning

1. AI Studio or Gemini app. Usually I find that these do a pretty good job of giving me a list of things that I need to consider or do as a part of starting a project. Most of the time, at least in coding and software engineering, I probably know 75%+ of what needs to happen so I’m mostly using AI to fill in the gaps. These tools are pretty good here and fast enough that I’m not waiting.

2. NotebookLM. If there’s something I’m doing (I have one of these projects I’m starting) where I need to get a good amount of context, I’ll use NotebookLM along with my good old ability to use Search. I find NotebookLM and Gemini Deep Research to be a bit of a sword. They can scan 50,100,200 articles in a manner of minutes, but sometimes they miss the nuance in creativity that I was searching for. That said, they can also summarize a lot of commonalities at a rate far faster than I could. Learning: Use these tools with a purpose.

Coding

I’m pretty boring here. I use Antigravity IDE mostly. For whatever reason, the code at left and agent at right view just works for me, in particular with problems that require agent + human co-work.

I usually have anywhere from 2-4 threads going at once. I try to cap complex work at 1-2 workflows requiring my attention, otherwise I find the context switching for me personally just isn’t worth it. On the AI adoption levels I’m probably somewhere between a 4 and 6.

I have no problem YOLO’ing, but I find that Gemini in particular can struggle with nuance of time based decision making and implementation (ex. the repo from April 2026 is the one I really wanted even though everything else looked close).

I’m hoping to explore more this year full agent handoffs in more complex codebases, which is partially requiring me to build a few codebases. Agents in particular I think you can automate from 0 to 1, but the rails for 1 to N or N to N development aren’t there yet, nor is the token consumption model that supports this.

I’ve never been a heavy CLI user, although I do see the benefits of CLI usage by agents primarily. The SDK is probably the most interesting to me and will fit into some of the experiments below.

Experimenting

In my flows above, I set a target of trying at least 2-4 things per quarter based on new features that are out and the maturity of the models that we can use at work. This quarter, my list is:

1. Giving it a real shot with local models and Gemma 4. I’ve been building a local cluster with some old mini-PC’s. My colleague wrote about a setup he did here. Mine will probably be similar but I want to lean into the local model benefits and drawbacks as a part of my learning.

2. Implementing DESIGN.md or giving it the college try. I’ve read a few articles about the benefits of using DESIGN.md with agentic development and now I want to see if I can really accelerate my frontend work.

3. Standardizing my multi-agent pattern I build with. I’ve used a few but not enough to consider myself an expert. Local models fall in here somewhere and I’m hoping to have enough brain space to solidify this by the end of the summer.

Try things out!

The only way I’ve really learned in this space is being all in. I hope this gives you the confidence to keep trying things out, learning, and trying again with these tools. The learning pit is a real concept we all go through, AI just amplifies the ups and downs and compacts the overall time spent.

Share

I’ll be adding skills I create here for you to use / view!

Context

I’ve probably used ~100 sessions or chats in Antigravity this week developing said demo as I’m trying to be better about flushing and getting new context, iterating on a task, and starting fresh. But, being a data engineer at heart, I was starting to feel like some of the information from prior sessions was getting forgotten by me, and, selfishly, since I work in an outbound Developer team, I was probably losing ideas to talk about.

Enter the session-journal agent skill.

What is the session-journal skill?

Agent sessions in Antigravity / other platforms store detailed session logs for each of the sessions they create. I wanted to get rolling summaries of each session, and then create aggregates of sessions to find content, understand key strategies the agent took (or failed at), and ultimately improve my prompting and sharing of things I’ve been learning along the way.

Here’s almost literally the depiction of what this process does:

The session-journal works as a sidecar in Antigravity to continuously summarize sessions. Since I try to create a new session for each task, sometimes there are common things that happen across the tasks or things that I try to fix repeatedly.

Writing to .local, the session-journal skill creates the summaries for my use, and does a light brainstorm of potential content themes (this part might be optional if you use it).

How I built it

Most agents are actually pretty decent at using skills to create skills, and this is no different. After some brainstorming on what I actually needed this skill to do, I used Antigravity to create the skill. I have a colleague (@joeshirey) who took another approach using superpowers. We merged the skills together to get the best of both worlds.

Try it!

Please let me know if you try it and what works or doesn’t! Next up is to try this on other coding tools and refine how it’s running (sometimes the sidecar doesn’t start).

Share

How AI is challenging your moat

In business school, we often talk about the competitive advantages of a company. The moat becomes the durable patterns that sustain a company’s advantages.

In the software world, patterns might be the data advantage you have (see my prior article), the durable relationship you’ve built with your customers that no one else can do, or the fact that a switching cost is too high (although switching costs are less of a ‘desired advantage’ than a threat to the customer of change).

AI presents a new challenge to the durability of your moat, but sometimes not in ways you’d expect. I’ll present a tangible example I’ve recently experienced, and then talk about how you can better position yourself and your company to defend against this new competitor.

An example: the cost of landscape design

We have a rental property pretty close to our house. The front yard is massive, its covered in grass, and gets almost zero shade during the hot Colorado summers. This, in combination with the severe drought, presents a number of challenges (both in terms of the actual cost of water and the philosophical question of being environmentally responsible).

So I presented a picture of this yard to Gemini, and asked it to xeriscape based on the location the house is at, the size of the yard, and my requirements (very low water usage and maintenance).

And, unsurprisingly, the list of recommendations including an image of what this looked like are pretty decent!

In this example, why would I use a landscaping company then? I’d have to pay for the architect time, and then probably overpay for doing the work (I’m pretty handy and I like doing this type of work myself). AI did a ‘C’ job, which for this example translates to ‘good enough’.

The challenge to the existing landscape design and contracting firms

In the above, it’s clear I have some process expertise and I was able to leverage generalized knowledge, and multimodality in a model. I’m probably not the average customer. I used AI to reduce the knowledge and process component of the moat.

But how many of these examples exist that start to erode the moat?

For the landscape architecture firm, the firm must seek to shift and deepen the moat around expertise and design. This can include moving up in the desired customer base and focus, or using AI as a design partner in the flow to accelerate the process with customers to get their yards re-landscaped. This could also include capturing more data for things like water use or design traits that people really like.

Alternatively, a company could choose to double down on the execution side. While I know how to do the work, it’s also highly time intensive and not complex labor. Scaling the advantage of execution may be preferable to some.

All of these changes are valid approaches, but require ‘moving the moat’ from a customer perspective.

Tangible steps to defend your moat from AI

For any company, AI presents both advantages and threats (we probably should add this to a SWOT or Porter’s 5 Forces box). What are some things I’m thinking about to help strengthen the moat?

Here are a few things I’ve found along the way that are tactical steps you can take this week to start understanding the potential impact of AI, and what to start hardening.

1. Pay for AI. Even a cheap subscription limits the amount of information that could leak from seemingly innocent questions. This is a basic but required step to protecting your intellectual property.

2. Ask AI what it knows. In the example of Google Cloud Next, we used AI to help us build a skill about planning a marathon (most of us knew comparatively little about the actual event planning process). Someone on our team had worked as a coordinator before and helped validate the steps. The latest models were surprisingly close to understanding the full process. Do this in a paid account to see how much you think is your ‘secret sauce’ vs. generalized knowledge in a model.

3. Identify your data and process advantages AI doesn’t have. You should assume that as LLMs continue to improve, the 'distinct’ advantages you have will continue to erode. Your first party information (data, processes, relationships), becomes even more valuable in this world. Build around the things only you can do.

4. Leverage AI to harden your advantages. If you know how to prompt AI, the rate which you’ll be able to move is probably 2-10x someone else. Take training on AI skills even if you own a small business. In the landscaping example, if you can use AI to even get to a draft faster, this represents ways you can build new advantages in your relationships with your customers.

5. Understand the AI limiting tools at your fingertips. Most people dont think about robots.txt for internet scraping. There are new methods to help prevent your public knowledge from being used to train models. Consider what these mean, but also if your competitors allow scraping, what you should tangibly protect from training vs. what enables you to be a ‘trusted’ provider in GEO (generative engine optimization aka being a grounded link in something like a prompt response).

Please let me know what else you’re thinking about with AI and your organizations! Happy Tuesday!

Share

The simplest answer I have is that big data is coming back, and it’s going to be highly relevant to many teams as the agentic AI era continues. But the foundation of agents isn’t built on your structured data. It’s built on logs, interactions, and the mess of data residing in your Google Workspace, Office (CoPilot365), and other platforms that have been either off limits, too expensive to track, or frankly, just not considered as valuable.

A ‘simple’ CRM agent example

Here’s a simple business case. Let’s say you are going to create a chatbot for a sales agent. The agent is designed to be an assistant for your sales team to answer questions about specific deals, deal designs, customer analysis, and even deal support. You probably have a CRM that has the basics like what the customer account is, some sloppy opportunity data, and some light ‘contact’ information.

But the information your agent needs in order to learn is all about the interactions that take place across prospects, deals, and opportunities. This data might look something like:

• Meeting transcripts between a sales representative and a prospect. Other transcripts between your internal teams talking about the deal size, approach, and even contracting.

• Spreadsheets with analytics in them about the deal and different values.

• Log data from your other systems (say your actual CRM logs) for changes to the deal over time, or how a what-if scenario engine is used.

For most organizations, this represents a pile of very messy, large data. And frankly, it wasn’t that useful before. However, agents need to understand the glue (aka processes) that actually create outcomes in order to re-enact and improve. Imagine you have to improve one of your skills, and the only feedback you have is the input, and the outcome. Aligning your own trajectory to the outcome is a hit or miss activity, and where we typically say the agent failed and move on when we didn’t give the agent anything useful to get started with.

This is the ‘dark data’ problem in companies, and where big data, in my opinion, is going to make a comeback.

Building an agentic ‘big’ data foundation

So what do data and AI leaders do to get ahead of this?

1. Get ahead of this problem and hoard as much telemetry and unstructured data as possible. I know we say hoarding creates data swamps, but when change is constant, understanding what in this data is useful is unknown, the compute power these days can sift through it, and not having eval data means your agent is DOA.

2. Get value stream maps of the processes you are looking to ‘agentify’ and map your data domains to this. If you don’t have the data today, identify all the sources you might have available including (gasp: new SaaS systems) that can get this process data for you.

3. Start identifying golden data outcomes from your captured data. This may be something like ‘sessions that yielded a sale’ or ‘key process moments’. Labelling data provides you with an eval dataset that can be used when training or modifying agent behavior to yield similar agent trajectories (think an agent session with turns and tool calls).

There are many other things here that you’ll need like managing sensitive data effectively from the data governance playbook, and building effective and efficient data models over this, but step 1 is still ‘collect data’.

Happy Wednesday!

Understanding the Code Completion Curve

The feature or code completion curve tends to look something like this when you consider the amount of lines of code you write, to what actually ends up shipping based on completing the work:

What happens at certain parts of the graph above roughly equates to how we can think about using AI for different ‘tasks’ and ‘roles’ in the coding lifecycle, positioning AI for success, and also setting us developers up to maximize our value in the cycle.

Let’s talk about the first part of this graph, where we’re writing code in order to get a feature or requirement working, or what I’m calling the Generate zone.

The ‘Generate’ Zone: Using AI as a Creative Foil

Interestingly (or not), most of the AI examples I find online anymore are heavily weighted to this code zone (I see you vibe coding). If you’ve attempted an AI experiment, this is probably where you started, and either had success, mixed success, or no success.

This also happens to be the zone where we, as humans, are the most effective. Creativity isn’t bred from next token prediction, it’s about building business context in the broader context of our applications we create and operate, and design ways to creatively implement a requirement. If we just try to put AI in this box, no one ends up happy except for the 30 minute app creators.

As we shift from generation in a 0 to 1 scenario to an incremental feature, using AI here can also help to improve our ability to onboard to code, learn it, and make changes. Here, we are asserting control over which AI tools we are using, and leveraging components of AI for efficiency and quality. Yet, we are not giving up control over what we learn, and it’s important to draw the distinction so we use the right AI components.

You’ve also generated the most code that you have to maintain when you finally get the thing working the first time, which leads us into the second, or ‘Refactor’ zone.

The ‘Refactor’ Zone: Optimizing Code with AI Assistance

Here, in the refactor and optimize zone, we apply our knowledge of coding principles, pain of the past, and other knowledge about things like performance to slowly reduce the code we built, refactor it based on good coding practices, and make the code ready to ‘ship’. Here, there’s an art and a science, but we’re using what we’ve generated as the input, and our standards and observations to reduce and refactor.

The blue zone also happens to be the next ‘underserved’ area I think about more and more for AI. While we need to provide a lot of input here, as developers we can provide specs, build observability platforms, and set goals for code refactoring that tend to rely better on an existing knowledge base than in pure generation. Which leads us to the final, or ‘move on’ zone.

The ‘Move On’ Zone: Knowing When Good Enough is Best

I call this the ‘move on’ zone, but you could also call this the danger zone of development. If things are going good enough, we need to know when to stop optimization for optimization sake and move on, and avoid the refactoring loop that can happen as we add features and then continually revisit ‘good enough’ code..

Why These Zones Matter for Effective AI Integration

For starters, when we use AI, it’s important to recognize the benefit that AI and LLM’s can provide us while also not losing the creativity and systems thinking that humans can, in fact, provide. In all cases, its about the systems around AI that we need to create and maintain in order for the LLM and us to build good software.

Practical Recommendations for Using AI in Each Zone

If you’re getting started, here are a few recommendations I have as you think about the ‘zones’, and using AI in each zone. While not all inclusive, hopefully these provide a few nuggets or places you can use LLMs or your existing systems to improve LLM performance.

Generate

1. Use AI as your foil. Ask LLM models, based on what you need to do and your ideas, to critique them, find potential holes, and generate potential execution plans.

2. Using thinking models here can be high value to see how an LLM would approach the tasks associated with the outcome you want. This allows you to refine your approach to solving the business problem.

3. Next, you can use an LLM to generate a task list to provide to other LLMs for completion, or even as subsets of activities as you write and test the code.

4. Remember that the LLM is only as good as the context you provide. An LLM doesn’t know that you’re only in us-east5, or what models you have access to. Provide this as context to get the best answers.

Refactor

1. Your code is working! Using a new prompt or different model for refactoring starts with a clean slate and ensures you dont overweight context that was needed in generation.

2. Here, ensuring your standards are written down and having few shot examples helps the LLMs refactor code as required. Following some principles for TDD here can be useful: give the LLM the test cases the code needs to pass in order to make sure the refactored code meets the goal.

3. Next, your other tech systems are critical here. Refactoring is an art informed by science. What calls are typically long? Does a specific call need to be optimized further to fit the wall time for an API call? Giving access to this information for an LLM via MCP allows the LLM to get this information as needed to know when to stop.

Move on

1. The code is good enough! You can move on to the next problem.

2. The systems you set up can monitor the code in the context of the platform it operates in. Setting up agents to check for long running processes or tasks is a useful and easy way to find things that may need further optimization while avoiding the trap of ‘continuous optimization’.

Share your own experiences with AI in the code lifecycle in the comments below!

This week, in what happened in my mailbox last week:

Let’s address another comment from one of my initial posts about AI usefulness:

• AI is helping to remove the drudgery

YES! It can be a big daunting to think about AI especially in the context of the current market where you hear about ‘AI taking jobs’ and all these other things (look at annual reports for the full story on these companies and don’t get sucked into the AI-ness of it).

In one of my first roles out of college I was a process analyst where we value stream mapped a number of client activities. Value stream mapping is a great way to identify things in the process that arent fun or add value and see how AI can potentially work to remove the work.

Another way to help remove the drudgery is to consider what the end goals are of any process and simply apply these as a part of a prompt to thinking models. This may prompt new ways to think about a problem and remove the toil, or identify further toil that can be automated or changed.

Ultimately, the goal is to lean into what humans do best with AI being one of the most capable tools we have available today.

If I tried to link the number of posts and blogs about the ‘death’ of software engineering, I’d probably be at 5 or 10 pages printed in single space, 6 pt. Font. It’s clickbait or justification for some manager or director, and I just don’t believe it. Call me a luddite. However, I do believe generative AI is coming for the way we work, not the job itself.

What does this mean as a developer in 2025? As I talk to more developers in the community and at large organizations, the question seems to have shifted from ‘will I use AI’ to ‘how can we actually use it?’ You can use it to re-create nostalgic video games, but when you go into an existing environment, where there’s been … decisions … and issues, how can you actually start to use AI effectively?

Here are a few points I’m coalescing around in my mental model of 2025 and some actions you can take today in your usage / testing of generative AI. And these things go together if you’re wondering.

Outcome vs task completion

Remember Cucumber? Most engineering practice changes I see tend to follow a similar pattern, where, in order to re-create the wheel, we first have to go back to the goal we need to deliver. Engineering ultimately exists to apply tools at solving business problems. But why sub-optimize in the process when we should probably be better about writing requirements and defining what the user outcome is that we want to achieve. We cannot expect any AI solutions to know unwritten requirements; the primary value here we have as senior engineers is that we’ve seen enough of this to have implicit requirements to apply.

Goal: Spend more time defining what the actual outcome is of a user goal, and leave no rock unturned.

Bonus: Using AI to help you here is a powerful tool, here’s a post from a peer of mine about it.

Process and standards management

‘I know what looks good’ isn’t going to cut it. Communities of knowledge built over time at organizations maintain ‘silent standards’ across the board, and many dont fully understand the business or technical processes they work in. Whether this is commenting strategies, how to write specific blocks of code, or ‘when to use what’, these are all implicit pieces of knowledge about the process you gain from junior to senior. But we can’t expect a generative AI model to ‘just know’ all the non-standards we follow, and the limits of the standards loaded into LLM’s that we do follow. The role of senior increasingly will shift to ‘managing the system and standards’ from ‘writing killer code and mentoring’. By virtue of improving the system and standards, I expect over time that organizations will also continue to benefit with other employees having the benefit of written and transferred knowledge over time.

Goal: Over-document how things are done including the flows so you can build better AI workflows and agents, and provide the generative AI models the context they need to actually write successful code.

Bonus: Build a value stream map and see how you can flip the system on its head to shift back to outcome based coding.

Knowledge graphs

Graphs have been around for a long time, but my personal view is that they’ve struggled with applicability except in specific scenarios (hello Google Search). As we use AI to crawl more information, we need to be better about explicitly stating the relationships between points of data in the information (hello graphs!). I can see where many engineers become ‘keepers of the graph of knowledge’ over time, focusing on the curation of knowledge about the components that create value for the company and its customers, and the tech stacks that support value generation. Generative AI has both significantly lowered the experience bar needed to create effective graphs, and increased the need to actually build them. As AI becomes more prevalent, we’ll need the graph to build better tests, and use the graph to provide the most relevant information (be it code components, or recommendations for a customer).

Goal: Create a graph for a domain of knowledge in your company and apply it, Whether this is customers and sessions, or engineering standards.

Bonus: Find overlapping and conflicting information and build a better corpus.

Context engineering is now making the rounds as the next ‘engineering’ we need in order to successfully make generative AI work. This after we just introduced ‘prompt engineering’ no less than 18 months ago. But do we really need to talk about this as a fully separate discipline or should we reconsider what the need for ‘context’ and ‘memory’ actually means as it relates to the most common disciplines that already exist: software and data engineering.

I’ve been positing internally that data engineering as a discipline needs some evolution. We’ve gone from databases to warehouses to lakes to multimodal lakehouses, but these still seem stuck in the paradigm that we will make insights of the data, and then humans will take action on the insights in a way that makes sense. I can tell you the breakdown on the last part of that continues to happen. Joe Reis writes about action oriented architectures as the next major way to think about data, and I happen to agree.

So then what about context? If you take the sum of everything above, it’s all been with the goal in mind of some intent or outcome we want to drive, be it customers, efficiency, or other. Context and memory usually requires that we have the knowledge about all things for a customer: structured sales, unstructured logs, interactions, and even unstructured data like images, video, and audio. We have to make sense of all this. To be fair, the use cases discussed in context engineering and the pipelines are needed, but we need to understand how we can use our existing tools to accomplish these outcomes without making all of this feel exclusive and unknown.

In some way, graph data engineering to me feels like the pinnacle of all our prior work put together, in the context of taking real time action using our engineered knowledge to make experiences better for our customers. Ultimately, I hope that context engineering becomes another data point and just the next iteration of data engineering, using graph databases of information to drive great business outcomes.

How to get started with graph data engineering

If you’re unfamiliar working with graphs currently, that’s okay! I’d typically recommend you start with something like this where you can learn about graphs in the context of structured data engineering concepts you already know. From there, you can then start storing data in graph formats like this.

Once you’ve got those down, take a spin with LLMs to help you build out graph entities and edges, and see what the LLM’s will extract from your content. You can use this to tune the graph that you’re building and maximize value without having preconceived notions about what the edges or entities necessarily should be.

This time, in what happened in my mailbox since last time

Last post got a lot of great comments and thoughts (even from my parents). Each following post, I’ll address a few of the comments I have particular feelings about, although they might also come from Youtube series I also make.

• What about the death of the junior engineer?

This one has been top of mind for a while. In my last post I explored what existing engineers need to be doing to start the mindset shift. I think in the next 12-24 months, we will begin to see the software engineering ladder be actually rewritten, and junior engineers who’ve used AI extensively for the past few years will start to get a leg up. What can you do now if you’re a junior engineer looking? Here are a few thoughts from someone you might or might not listen to:

1. Focus on the fundamental components of code and how things like security actually fit. You can’t trust LLM’s to hand off the full process without supervision. Do this, and this.

2. Learn about the business. I don’t know where in engineering we removed accountability from engineers to understand the business they support, but I think it’s a problem. I’m a believer that those who understand the business the best, along with understanding AI tools, will be the ultimate winners in whatever this reset is. This article is a good starting point.

Second part of the question. What happens when the pipeline for seniors dwindles?

I fully believe the pipeline for seniors will go to zero in the current implementation of the engineering ladder. This is not an incremental change in role profiles (see above). If you’re a senior engineer, don’t wait to ‘blow up the ladder’. These tools are changing what skills everyone needs, and better to do it to yourself than wait and have it be forced on you. Junior engineers have a fundamentally different skillset, strengths, and issues than when a lot of us entered the workforce as analysts or engineers.

Wrap up

My hope is to explore more AI in the code lifecycle in the next few posts, but as always, let me know your comments and questions so I can make this useful for everyone.

Subscribe now