Jake Gaylor

My website is an agent now

2026-07-29T00:00:00Z

The Agent2Agent protocol hit 1.0 under the Linux Foundation back in March, and the official JS SDK's stable 1.0 landed this month. As of this week, ai.jakegaylor.com implements it. One unauthenticated GET to /.well-known/agent-card.json returns a card describing an agent that represents me, and a JSON-RPC endpoint behind it will answer questions about my experience, explain how to connect to my MCP server, or deliver a message to my inbox.

That site has been accumulating interfaces for a while. The webpage is my resume addressed to humans. /llms.txt is the same resume addressed to language models that happen to crawl by. The MCP endpoint is the same resume addressed to an AI client that a human has configured. A2A is the fourth layer, and it's the first one where no human configures anything. An agent that lands on my domain can discover the card, read what my agent can do, and start a conversation, end to end, with nobody in the loop.

The build was small. The official JS SDK went stable six days before I started, and it mounts onto the Express app that already serves everything else. My executor is three skills. Questions get answered by a cheap model, gpt-5.4-nano, grounded in the resume and bio with instructions to refuse anything it can't source from them, and every answer costs a fraction of a cent. Messages mentioning MCP get onboarding instructions for the richer interface. Messages starting with CONTACT: get relayed to my email. If the model call fails, the skill falls back to returning the full resume.

The implication I actually care about is the hiring flow, and the honest version is that it doesn't exist yet. The recruiter side is already agentic. LinkedIn's Hiring Assistant takes an intake brief and runs the top of the funnel on its own, searching, ranking a pipeline, drafting outreach, handling replies. Tools like hireEZ and Juicebox run the same loop across the open web, crawling GitHub and personal sites and reconstructing careers from prose that was written for human skimmers. All of that agentic work runs over static documents, and the reading is one-way, because a resume can't answer back.

What none of them do today is fetch a card from a well-known URL and notice there's an agent standing behind the candidate. I made the same bet with llms.txt fifteen months ago, an afternoon of work wagered on where the consumers of your professional identity are moving. This time the consumer can ask follow-up questions.

If the crawlers learn to look, the first screen stops being a human skimming a PDF and becomes their agent interviewing mine. Pointed questions about Kubernetes or agent orchestration, grounded answers at three in the morning. Most of a first screen is logistics anyway, comp expectations, work authorization, location, remote or not. An agent can serve those as structured fields instead of prose to re-parse. Their agent can hand over the whole job description and get back an honest read on the fit, gaps named. Run the flow to its end and the first human touch is a meeting landing on my calendar. The people who are addressable that way get evaluated first. Today my agent covers the questions, and the rest of that screen is what I'm building next.

Tell your agent about ai.jakegaylor.com.

We built a new kind of computer and called it a sandbox

2026-07-28T00:00:00Z

A new kind of computer showed up over the last couple of years, and we're calling the primitive an agent sandbox. It suspends when idle, wakes on demand, keeps its state, and answers to its own name. Fly sells them as Sprites, Vercel as Vercel Sandbox, AWS as Lambda MicroVMs, and Daytona built its whole product around them. This is a new class of software to operate, and the operational discipline for running it still has to be built.

What you're actually operating

What happened is that the workload changed shape. For more than twenty years the unit of cloud compute was the request, a few hundred milliseconds of stateless work for a human who is still watching the spinner, and we built our platforms, autoscalers, and pricing models around it. Then agents arrived and moved in. An agent installs tools, writes files, kicks off long jobs, sits parked while the model thinks or a person sleeps, and expects its half-finished work to be exactly where it left it when the next message lands.

Nobody gets just one, either. A person burns through dozens of agents in a working day, parks most of them mid-task, and reopens a conversation from three weeks ago expecting the terminal, the installed tools, and the working tree to be right where everything was left. Multiply that by a team and you're operating thousands of these computers, nearly all of them asleep at any given moment.

That computer is the primitive, a long-lived, mostly idle, stateful, individually addressable process running code nobody reviewed.

Its duty cycle is mostly idle. The machine sits parked while the model generates tokens, while a tool call runs somewhere else, and while the human wanders off for hours, so the economics of running it are suspension and oversubscription. That's why every platform leads with pause and resume. Google's experimental Agent Substrate maps a large set of actors onto a small pool of ready workers, and its demo multiplexes about 250 stateful sessions across 8 pods. KubeMicroVM, a Kubernetes operator that wraps Lambda MicroVMs in CRDs, takes the other route and pushes the parked time down to AWS with an idle timeout on the resource itself.

Its state is the process itself. Terminal buffers, shell history, a half-written file the agent plans to come back to. None of that survives a kill and reschedule, so the lifecycle mechanism is checkpoint and restore of the live workload. Substrate drives runsc checkpoints under gVisor, and the microVM camp snapshots the whole machine. Nothing here is safely killable, because the instance is the data.

Each one is a named individual. You route to agent-session-1 and nothing else will do, which is why Substrate hands every actor its own hostname and KubeMicroVM's example resource carries a per-VM endpoint and a per-VM token. Fungibility is gone, and with it the whole mental stack of load balancers, replica counts, and Service selectors that don't care which copy answers.

And it runs code nobody reviewed. Namespaces and cgroups were never a security boundary, which stayed a tolerable fiction while everything in the cluster came through your CI, and it stops being tolerable when the process executes whatever the model just wrote. I made this argument about MCP servers in April 2025, and agents settled it. A kernel boundary is mandatory here, and the choice between gVisor's userspace kernel and full hardware virtualization is where the platforms actually differ. Building the image yourself buys you nothing, since the interesting code arrives at runtime.

The primitive shipped before the discipline

The public conversation about these products is cold start benchmarks and per-vCPU pricing tables, which makes sense if the CPU time is a commodity. But the vCPUs aren't all the same, because the platforms they run on aren't the same. The differences start where the comparison posts stop. We spent ten years building an operational discipline around the stateless container. Identity, egress, observability, capacity planning, cost attribution, blast radius. The categories all still apply, but the tooling under each one leans on a request boundary or a disposable instance somewhere, and this workload has neither.

Machines that belong to users

I helped build Ravi, which sells identity for agents, so I know this gap up close.

Issuing machine credentials is not the hard part, the container era solved that for the workloads a platform team deploys. The new part is that users need them in bulk. A developer used to hold machine credentials for one laptop and that was the whole story. The same developer now parks dozens of sandboxes a week, each calling third party APIs on their behalf, each needing a credential that says what it is and who it acts for, minted at session start and revocable one sandbox at a time. A decade of identity tooling assumed machines belong to the platform, and these machines belong to users.

The gauges break too

Few teams have a story for observability across a suspend boundary, and everyone operating these will need one. A session gets frozen for six hours, restored on a different host, and keeps going. Think about the trace that was open when it froze, the metrics that assume monotonic time, the log stream that now has a gap the length of a workday. Ten years of tooling assumes a process either runs or dies.

Capacity planning loses its instruments too. There's no request to count, so RPS and p99 mean nothing. Sprites' own pricing example is a four hour session that bursts to 8 CPUs and averages 30 percent of 2. That's a pricing example on the outside and a capacity planning nightmare on the inside.

And the ceilings sit lower than the pitch decks suggest. AWS caps the total memory across all your MicroVMs per region, running and suspended alike, so the parked sessions that make the economics work count against the same quota as the live ones. The default is adjustable by support ticket, and it still bites. KubeMicroVM's own load test hit it at about 161 concurrent VMs in us-east-1, with creation throughput of 3 to 4 VMs per second, before touching anything the operator controls.

Where Kubernetes lands

Two answers exist in the wild. KubeMicroVM keeps Kubernetes as the control plane and outsources the compute to AWS, inheriting AWS's ceilings, that memory quota included. Substrate keeps Kubernetes underneath as the machine layer and takes it out of the critical path, running session scheduling through its own gRPC control plane, because going through kube-scheduler costs seconds and a human is waiting on the other end.

Both shapes will find users, because a team already living in Kubernetes wants tools shaped like the rest of its tooling. The vendors selling the primitive lean the other way. Fly runs Sprites as hardware-isolated sandboxes on a platform it built itself, and nothing in the intro lineup advertises Kubernetes anywhere near the session path. Kubernetes gets demoted a layer rather than displaced, from the thing that schedules your workload to the thing that manages the machines your real scheduler packs sessions onto. VMs took the same demotion when containers arrived.

The strongest counter-case is enterprise BYOC. Buyers with compliance requirements want this running inside their own cloud account, and vendors are meeting them there. Northflank sells self-serve BYOC that provisions Kubernetes clusters in your AWS, GCP, or Azure account and runs its sandboxes on top of them, keeping Kubernetes in the session path. Runloop lands its own hypervisor and scheduler inside your VPC instead. BYOC settles where the machines live and which boundary the data stays inside, and both shapes can deliver that. The suspend economics don't move when the account changes hands, and a scheduler built to pack idle sessions is still what makes the numbers work, wherever it runs.

Don't take my word for it

If you're not convinced this is a primitive rather than a pile of products, here are four product descriptions with the vendor names removed.

A persistent Linux computer that pauses when idle, checkpoints its whole filesystem in about 300ms, and wakes when a request hits its URL.
An execution layer for agents where sandboxes persist by default and the filesystem snapshots itself.
A sandbox that spins up in under 90 milliseconds, stays around indefinitely for persistent agents, and resumes any workflow from a saved snapshot.
A managed microVM that suspends when idle and resumes with its RAM intact.

Can you tell which one is Sprites, which is Vercel Sandbox, which is Daytona, and which is Lambda MicroVMs? They're in that order. All four shipped within about eighteen months of each other, from teams that were not talking to each other, and the feature lists converged anyway. When that many vendors independently arrive at the same spec, it seems to me the spec is describing a primitive. The list runs longer than these four, too. E2B, Modal, and Cloudflare all sell their own take, pause and resume included, and every addition makes the convergence harder to dismiss.

I was building Kubernetes clusters on vSphere with CoreOS and fleet in the summer of 2015, right as 1.0 shipped. The primitive worked then too. The discipline took another five years of postmortems. This new computer is at the same spot on that curve, and the fastest way to feel it is direct. Spin one up, suspend it mid-session, restore it, and count how many of your assumptions come back broken.

Your agents pay the tool tax every day

2026-07-25T00:00:00Z

Every tool makes some moves cheap and others expensive, and you end up solving problems out of the cheap column. If deploys are scary you batch changes. If provisioning a database takes a ticket and a week, the new feature gets crammed into the schema you already have. Nobody chooses any of that in a meeting, the tool chooses it, and the team learns to call the result engineering judgment.

None of this used to bother me much, because humans are good at absorbing a bad fit. Then my workforce filled up with agents, who absorb nothing.

Surviving a bad tool takes memory

A badly fitting tool usually gets survived rather than replaced. You learn the workaround once and it settles into muscle memory. The new hire picks it up in a hallway. A year in, nobody experiences the friction anymore, since everyone's hands route around it without involving their brains. The misfit is still there, it just stopped generating complaints. The team amortized it, paying the learning cost once and spreading it over years of not noticing, and that only works if the workers remember.

Agents have horrible memories. Every session is day one. Muscle memory never forms because there is no muscle and no memory to form it in. Whatever it costs to fight the tool, an agent pays full price every single session.

I know two ways to cover that bill and both are ugly. Writing the workaround down works, since an agent only knows what's on disk, but look at what the runbook becomes. Every gotcha in it is a patch over a tool that fights you, the file grows in proportion to the misfit, and each agent burns context reading it before doing anything. Skipping the doc goes worse. Each agent rediscovers the workaround live, a little differently every time, until the risky procedure has five ad hoc variants invented by workers who will never compare notes.

The two ugly options hide a third one. Change the tool until the workaround has nothing left to work around. Deleting a runbook entry beats writing one.

The grain cuts deeper now

Agents also follow a tool's grain harder than people do. A human will sometimes fight the tool out of taste, or stubbornness, or a better way remembered from an old job. An agent takes the cheap path nearly every time and then generalizes from it, since precedent is how agents work. Whatever direction the tool leans, the whole workforce leans with it, at machine speed, with nobody pushing back on the defaults. A tool with a bad grain used to cost you some drift around the edges. Now it hands you a system that is confidently and consistently wrong in the exact shape of its defaults.

I've already replaced one

SOPS managed the secrets in my home cloud for years, and for a workforce of one human it was fine. I knew the choreography, my hands did the decryption dance without complaint, and the friction had long since gone invisible.

Then agents joined the workforce and every hidden cost got itemized. Every secret operation routed through decryption on my laptop. Discovery meant opening encrypted files. An agent couldn't work anywhere near secrets without values bleeding into its context. No runbook was going to fix that, because the misfit was structural, so I moved to a store where secrets have stable names and now agents write all the wiring without ever seeing a value.

Nothing about SOPS got worse in those years. The workforce changed underneath it, and if I were still the only operator I'd have happily kept it another decade.

What fit looks like for a forgetful workforce

When I evaluate a tool now I ask how it treats a competent operator who has never seen it before and never will remember it. Day one, every day.

The default path has to be the safe path, because an agent takes the default. A tool whose safe usage depends on operator discipline was designed for a workforce I no longer have.

The tool's objects need names. Plans, diffs, and runbooks are made of nouns, and when things have stable addresses an agent can discuss them, reference them, and wire them up without ever touching them. Objects that live as state in an operator's head might as well not exist.

Asking the tool has to beat remembering, because there is no remembering. Anything the operator is supposed to just know is a fact the tool failed to keep.

Errors carry more weight than manuals. An error message is the only documentation guaranteed to arrive at the exact moment it matters, inside the context window of whoever hit the problem. A tool that fails with a good next step onboards my workforce fresh every session.

And running anything twice has to be harmless, since agents retry constantly.

There's nothing exotic on that list. Good tools always worked this way, a human workforce just let you get away with the ones that didn't.

Read the bill

The friction your team stopped noticing never stopped costing you. You've been fluent in your tools for so long that the grammar disappeared. Agents feel every bump, and they leave the evidence where you can read it. Scar tissue piling up in the runbook. The same workaround re-derived in transcript after transcript. Context burned relearning what no worker can retain.

When the same line item keeps showing up, stop writing better workarounds and go get the tool that fits the workforce you actually have.

What my agents can do now that secrets have names

2026-07-24T00:00:00Z

AI agents work in my repos and my home cloud every day. They write manifests, onboard apps, and diagnose failures. The models have been good enough for that work for a while. The wins are in enablement, arranging the environment so more of the work is safe to hand to an agent. A big one was moving my secrets behind names.

Every secret in my cluster lives in Infisical and has a stable address, the project plus the secret name plus the key. The value stays in the store. Only the systems with runtime privilege, the operator and the apps themselves, ever resolve a name into a value.

I've already written about why I chose Infisical over SOPS and how the whole setup is wired. This is about the jobs that used to be mine and can now be delegated, because a secret an agent can name is a secret an agent can work with.

We can talk about secrets now

Before, "which token does Alertmanager use to reach ntfy?" was a dangerous question. Answering it honestly meant a value on the screen, in a context window, in a transcript that outlives the conversation. So secrets were the topic we talked around, and every plan that touched one had a hole in the middle shaped like you handle this part.

Now secrets are nouns. An agent and I can discuss the Cloudflare token in the bootstrap kernel, or whether oauth2-proxy-secrets should split into two, the same way we discuss a Deployment or a Service. Debugging, planning, and design conversations reference secrets by the same names that appear in the manifests and in the Infisical UI.

They write the wiring

When an agent onboards an app to my cluster, it writes the InfisicalSecret CR, the ServiceAccount, and the envFrom that points the pod at myapp-secrets. When it updates a runbook, it can write "rotate SMTP_PASSWORD in the mailer project" instead of gesturing vaguely at "the mail credentials." The resolution happens later, somewhere else, by the operator that actually holds privilege. The agent wrote every line of the plumbing and was never trusted with a value.

The agent creates the slot and I fill the value. For self-minted secrets, even that job isn't mine. oauth2-proxy's session secret is 32 random bytes that nobody needs to know, and the agent knows both the algorithm and the address. So it writes the one-liner, infisical secrets set SESSION_SECRET=$(openssl rand -base64 32), and runs it through my logged-in CLI. The value travels from openssl into the store without ever existing in the agent's context, my clipboard, or a screen. Nobody has seen that secret, and nobody needs to. My job is down to the values only a third party's dashboard will mint.

They can see the whole inventory

An agent in my repo can list every secret that exists without reading any of them. Which projects exist, which keys each app consumes, which InfisicalSecret delivers what to where.

That read-only view covers a whole class of chores. Auditing which apps consume which credentials. Spotting the key that no workload references anymore. Onboarding app N+1 by pattern-matching apps one through N. Under SOPS this was possible but never natural, because the names were scattered across encrypted files and everything past discovery ran through decryption on my laptop. Now discovery is the cheap, safe, default path. An agent can be exhaustively curious about my secrets and never read a single value.

Leak response is becoming an agent task

Rotation in my cluster is one step: paste the new value into the UI, and the operator handles propagation and restarts. I covered that machinery in the SOPS post.

Infisical has an API, and so do many of the providers that mint these credentials. When both ends have one, the loop can close without me. An agent that spots a token in a log can mint a replacement at the provider, write it to Infisical, let the operator roll it out, and revoke the old one. The leak is fixed before I've read the alert.

Not every secret supports that today. But it changes what I need from an agent. I don't need one that provably never mishandles a secret. I need recovery cheap enough that responding to a mistake is a reflex, whether the mistake was mine or the machine's. Bounded blast radius beats flawless behavior, for humans and agents alike.

The next step: identities for the agents themselves

My agents do all of this without any Infisical access of their own. The names in the repo are free to read. Everything that touches the store, listing what exists and minting the occasional value, runs through my logged-in CLI. The identity stays mine; only the hands are the agent's.

That borrowed CLI is the loose end. An agent working through my login carries every permission I have, and this whole migration was about killing that shape of credential. The fix already runs in the cluster. Every app has its own machine identity, scoped to one project with the smallest role that does the job. My agents are next. Purpose-built identities, scoped to the work each one does, minted by the same pattern that onboarded every app.

That's the next post.

Do more means delegate more

In the agent-ready infrastructure post I asked whether a competent stranger could operate your system on day one. Secrets were always the room the stranger could never enter, so every task that passed through that room bounced back to me.

Names fixed that. The stranger still can't enter the room, and it no longer needs to.

That's what "doing more with agents" has actually meant here. Not smarter agents, and not more trust. A system arranged so that trust is mostly unnecessary, and the work flows anyway.

Agent-ready infrastructure is mostly good ops

2026-07-22T00:00:00Z

AI agents are joining engineering teams as contributors and operators. Most of the conversation is about which AI tools to adopt, and I think that's the wrong end of the problem. The better question is whether your system has the properties that make delegation safe in the first place.

I run agents against my home cloud every day. They write manifests, onboard apps, diagnose failures, and propose fixes. Almost nothing in that cluster is AI infrastructure. What makes it work is a set of properties good operations people have wanted forever. This post walks through each one, why it carries the weight, and what breaks without it. My homelab is the running example, but none of this is homelab-specific.

One reviewable write path

An agent produces changes faster than you can watch it work. If you supervise action by action, you're the bottleneck, and eventually you'll stop looking. The property you want is that every change, no matter who makes it, becomes a diff in one place. Review scales because diffs queue up instead of scattering across dashboards and shell histories. Rollback comes free because the history is the interface.

Without this, changes land invisibly. An agent with a kubectl credential can leave your cluster different from every description of it, and you find out during an outage.

My whole cluster works this way. Every subsystem is a file in a git repo, Flux applies whatever lands on main, and undoing a change is a revert. An agent contributing to my infrastructure never touches the cluster. It writes YAML and opens a pull request, the same as I do.

Predictability over cleverness

Agents work by generalizing from examples. Show one the pattern for deploying an app and it will wire the next app the same way. That's a superpower in a consistent system and a liability in a clever one, because every special case is a trap for a worker that learns from precedent.

Consistency used to be an aesthetic argument. With agents in the repo it becomes a safety property. The more your system looks like one idea repeated, the more of it an agent can safely touch.

In my repo every subsystem gets its own directory, every app deploys with the same chart-and-kustomization shape, and onboarding a new app means dropping one file in a well-known folder. An agent that has read one app has read them all.

Written down or it doesn't exist

An agent knows what's on disk and nothing else. It wasn't in the room the night you learned that restarting the VPN daemon also takes down the pod network. If that lesson lives in your head, an agent will re-learn it in production, at machine speed.

So the bar for documentation changes. Runbooks, conventions, and recorded gotchas stop being nice to have and become the onboarding program for a workforce that reads everything and attends no meetings. The payoff compounds, too. Unlike human hires, every agent you ever run gets the benefit of every lesson you've written down.

My repo carries its conventions in a file agents load at the start of every session, its failure modes in a runbook, and its repeatable procedures as skills an agent can execute step by step. When something burns me, the scar goes in a file.

Handles, not custody

Delegation means letting workers act on sensitive things without possessing them. If doing the work requires holding the credential, every worker is a leak vector, and you'll ration delegation by dread.

The fix is names. When every secret has a stable, well-known address, an agent can list what exists, reference the right one in the wiring it writes, and never see a value along the way. I wrote a pair of posts on this, why I moved off SOPS and how the setup works. The short version is that my secrets live in Infisical and agents only ever touch the names.

Eyes without hands

Diagnosis shouldn't require power. When an agent can read logs and metrics through an interface that can't change anything, you can grant unlimited curiosity at zero risk. Couple the two together and you're stuck choosing between agents that are blind and agents that are dangerous.

Blind is worse than it sounds. A worker that can't observe consequences will still finish the task, it just finishes on vibes. Cheap read-only telemetry is what turns guessing into diagnosis.

My agents get Grafana over MCP. They query pod logs from Loki and metrics from Prometheus without a shell into anything. Investigating an incident requires no permission that could cause one.

Bounded blast radius

Mistakes happen at a rate, and adding operators raises the rate. Agents are operators. The systems that absorb this well aren't the ones with the most careful workers. They're the ones where the worst single action is survivable.

You can't hire your way to zero mistakes, so the useful work is arranging the system to need less trust. Scope every identity so a compromised or confused actor reaches one app. Guard the deletes that can't be undone. Keep an escape hatch outside the system for whatever the system can't restore about itself.

In my cluster every app identity can read exactly one project's secrets, the GitOps machinery is forbidden from garbage collecting the secret store, and the keys that rebuild everything live outside the cluster they rebuild.

One door to the dangerous rooms

Some operations are genuinely dangerous, and the answer isn't keeping agents away from them. It's making sure each dangerous thing has exactly one supported way to do it. Five ad-hoc variants of a risky procedure guarantee that a worker eventually picks the wrong one, and improvised recovery is how small incidents become big ones.

Idempotency does the rest. Entry points that are safe to re-run make retries boring, and agents retry a lot.

Host and cluster operations in my homelab are Ansible playbooks and nothing else. There's no drawer of shell scripts with subtle differences. The risky rooms each have one door, and walking through it twice is harmless.

Loops that close without a courier

An operator is only as good as its view of consequences. If deploy status and failure alerts route through a human, the agent can't self-correct, and you become a courier between your system and your workers. That's the bottleneck you were trying to remove.

Here's the loop every change in my cluster travels.

A push triggers a webhook and Flux reconciles within seconds. The cluster converges on whatever the repo now says. Telemetry records what actually happened and alerts fire when it's bad. All of it is visible to the same agent that made the commit, so the consequences of change N are sitting there when it's time to write change N plus one.

Operable versus operated

I want to be clear about where the line sits today. My cluster is agent-operable, and it isn't agent-operated yet. Alerts come to my phone, not to an agent. Approval still means me reading a diff before it merges. Nothing rotates itself end to end without a person in the loop.

Some of that gap I plan to close, like letting an agent receive the alert and start the diagnosis before I've picked up my phone. Some of it I'm keeping on purpose, because a human reading diffs is a cheap gate with a very good catch rate. The properties above are what make closing the gap a choice I get to make gradually instead of a cliff I have to jump.

The stranger test

Every property in this post reduces to one question. Could a competent stranger operate your system on day one, using only what's written down, through paths you can review, with consequences you can survive?

That's always been what a well-run system looks like. Teams got away with failing the test because human strangers arrive slowly, one hire at a time, and absorb the unwritten rules in meetings and hallways. Agents don't. They arrive instantly, read everything, attend nothing, and take the system exactly as documented.

The stranger finally showed up. The good news is that everything on this list was worth doing anyway.

How I use Infisical in my home cloud

2026-07-21T00:00:00Z

My homelab is a k3s cluster spread across a few Mac minis. Linux VMs via Lima, Tailscale for the network, and Flux reconciling everything from a git repo. I gave the full tour of that architecture over on the homelab page. This post is about the secrets side, because every credential in that cluster now flows through Infisical, and it ended up in way more places than I expected.

Just as important, this setup is what lets AI agents work in the cluster the right way. Agents get names to work with, never values. If you're wondering why Infisical instead of SOPS, I wrote up that decision separately. This post is about how the system works today.

Two Infisicals

On-prem Infisical, self-hosted in the cluster, is the source of truth for runtime app secrets. Every app gets its own project and its own machine identity. If an app reads an API key at runtime, it comes from here.

Cloud Infisical holds exactly three things in a project I call the bootstrap kernel. The Cloudflare token cert-manager needs for DNS-01, the Tailscale OAuth client, and the on-prem server's own environment. These live in cloud because the on-prem server's database sits on storage inside the cluster it serves, so on a full rebuild anything needed before the secret store exists has to come from somewhere else.

That third item is my favorite part. The server needs an ENCRYPTION_KEY and AUTH_SECRET before it can start, and it can't fetch its own encryption key from itself. So a cloud-backed InfisicalSecret delivers its env as a plain Kubernetes Secret and the server boots off it like any other app. Infisical delivering Infisical's secrets to Infisical.

The break-glass copy takes care of itself too. When the bootstrap kernel changes in cloud Infisical, my password manager gets the update automatically, so there's always a copy a human can reach. A wrong encryption key doesn't error politely. It just makes the server unable to read its own data. Ask me how I know to be careful here.

Here's the whole system on one picture.

Don't store on-prem admin credentials in cloud, tempting as a hands-off rebuild sounds. A compromised cloud account would read every runtime secret. Cloud bootstraps the platform and nothing more.

Runtime secrets with zero stored credentials

The auth model is what sold me. Every app authenticates with Kubernetes native auth. The operator presents the app's ServiceAccount token and the Infisical server validates it against the cluster's TokenReview API.

There is no clientId or clientSecret for any app, anywhere. There's nothing to leak and nothing to accidentally commit. Each identity is pinned to exactly one ServiceAccount in one namespace, with viewer access to one project.

The apps themselves have no idea Infisical exists. The operator turns each InfisicalSecret CR into a boring native Kubernetes Secret, and pods consume it with envFrom or a volume mount like it's 2016. The whole CR looks like this.

apiVersion: secrets.infisical.com/v1alpha1
kind: InfisicalSecret
spec:
  authentication:
    kubernetesAuth:
      identityId: <the app's identity>
      serviceAccountRef:
        name: myapp-infisical
        namespace: myapp
  managedKubeSecretReferences:
    - secretName: myapp-secrets
      secretNamespace: myapp

When I rotate a value, my whole job is pasting the new one into the Infisical UI. The operator updates the Secret and an annotation restarts the workload.

Names are also what make this safe for AI agents. An agent can list what secrets exist, reference myapp-secrets in the wiring it writes, and never see a value along the way.

One secret to carry

Under SOPS my secret zero was an age key that decrypted everything. Now it's a single cloud machine identity with viewer access to three bootstrap secrets. Losing it exposes those three secrets and nothing else.

It gets seeded by a one-shot Ansible playbook that reads the credential from environment variables instead of -e flags, because your shell history is not a secret store. The playbook writes it into the cluster as the one Secret the operator uses for cloud pulls.

I'd much rather babysit one tightly scoped credential than a god key.

Everything else it quietly does

Once the pattern existed, everything migrated into it.

Flux's GitHub webhook token, so pushes trigger reconciles instantly
Alertmanager's ntfy token, so my phone buzzes when something breaks
cert-manager's Cloudflare token and the Tailscale operator's OAuth client, both on the cloud tier
oauth2-proxy's session secrets, my S3-compatible storage keys, an OpenAI API key, and auth for half a dozen app repos that ride the same per-app pattern

A couple of design details are doing quiet work here too.

The operator talks to the server over internal cluster DNS, so an in-cluster reconcile never touches Traefik, Cloudflare, or a TLS handshake.
Apps that read secrets as env vars get an auto-reload annotation, so a rotation restarts them automatically. Apps that mount secrets as files, or re-read them per request, deliberately skip the annotation because kubelet and the app already handle updates. You have to know how each workload actually reads its secrets.

My favorite detail is that one of my apps has no runtime secrets at all, so it has no Infisical dependency at all. Apps that don't need the pattern don't pay for it.

The scars

Nobody blogs the error messages, so here are mine.

The Kubernetes auth host must be the full FQDN, meaning https://kubernetes.default.svc.cluster.local. The server's resolver doesn't apply the cluster DNS search domain, so the short name you'd naturally type fails with ENOTFOUND and no further explanation.

The server will refuse to call the cluster API at all until you set ALLOW_INTERNAL_IP_CONNECTIONS=true. That's its SSRF guard doing its job, but the failure looks like broken auth instead of a guard.

Orphaned Secrets stay dead. I use creationPolicy: Orphan so the operator would overwrite my old SOPS-managed Secrets in place during the migration instead of fighting them. The catch is that if a managed Secret gets deleted out from under the operator, say by a Flux prune, the operator's cache still says synced and it never recreates it. The fix is restarting the operator. Learned that one in production, if a homelab has production.

prune: false on the secret store itself. The server's database sits on volumes with a delete reclaim policy. One careless Flux restructure could cascade-delete the Kustomization, the PVCs, and every secret I own. So the GitOps machinery doesn't get to garbage collect the secret store.

Where it landed

That's the system. Two Infisicals, one credential outside it all, and every secret behind a name that agents can work with without ever reading a value. If you're weighing a similar move away from SOPS, the why post covers the trade-offs I accepted to get here.

My AI agents changed what I need from a secret store

2026-07-20T00:00:00Z

For a long time I used SOPS, and for a long time SOPS was the right call.

If you weren't in a major cloud, the story for dealing with secrets sucked. AWS and GCP folks got a managed secrets service with IAM bolted on. The rest of us got three options. You could wire a password manager CLI into your deploy scripts and hope everyone on the project had the same setup. You could bite the bullet and install Vault, which is a great product and also a whole second job. Or you could encrypt your secrets and store them right in the repo.

I picked the third one. SOPS encrypted my YAML, Flux decrypted it at apply time, and the whole thing genuinely worked. This summer I ripped it all out and moved everything to Infisical. This post is about why. If you want the tour of what the replacement actually looks like, that's over in how I use Infisical in my home cloud.

What SOPS life looked like

My cluster is a few Mac minis running k3s, with Flux reconciling everything from a git repo. Under SOPS, secrets were just part of that repo. Every one was an encrypted file sitting next to the manifests that used it, and Flux decrypted at apply time, which meant a decryption block in every single Kustomization. All 26 of them. And one age key, hand-carried between machines, could decrypt everything I own.

Here's what rotating one secret actually involved.

Three of those five steps are me. And notice the age key shows up twice, once on my laptop and once in the cluster.

The friction that added up

None of my complaints about SOPS are dealbreakers on their own.

Rotation was a chore, so I rotated less than I should have. Every new app meant another encrypted file, another decryption block, and another chance to fat-finger the recipient list. Secrets were scattered across the repo instead of living in one place I could look at and audit. And the age key bugged me a little more every year. It never rotated, it had to exist in two places, and it would decrypt everything forever.

Any one of those is livable. All of them together meant I dreaded touching secrets, and dreading rotation is how secrets end up three years old.

Then the agents showed up

The thing that actually pushed me over the edge wasn't any of that. It was that the way I build software changed. AI agents work in my repos every day now, and agents change what a secret store needs to be.

An agent should never read a secret. It doesn't need the value. It needs a handle, a well-known name that every agent and every human can use to talk about the same thing. It needs to see what secrets exist without seeing what's inside them. And it needs to write the wiring that lets other systems consume a secret, again without the value ever appearing anywhere.

To be fair, SOPS handles more of this than you'd guess. It encrypts the values and leaves the structure readable, so an agent can look at an encrypted file and see the key names without decrypting anything. Discovery was possible, it just wasn't natural. The names were scattered across encrypted files in whatever directory used them, with no single place to list them. And everything past discovery ran through decryption. Editing a value meant decrypting the whole file, and the age key sat right on my laptop, so an agent doing routine work was always one command away from plaintext it didn't need. The moment it decrypts, that plaintext is in its context window and its transcripts. Nothing about the design pushed toward names instead of values.

There's a second agent problem, and it's about leaking. Secrets get leaked all the time, by humans and agents alike. Pretending otherwise is how you end up with a system that handles leaks badly. What you actually want is recovery so cheap that reporting a leak is a reflex. Under SOPS the recovery path was everything in that diagram up there, and nobody does that as a reflex.

What I wanted instead

So the shopping list came out longer than the one I would have written a few years ago.

Rotate a secret without a git commit
Per-app scoping, so one leaked credential exposes one app instead of all of them
No long-lived credentials sitting in the cluster or in my shell history
A real story for rebuilding the cluster from nothing
Plain Kubernetes Secrets on the consuming side, so apps don't have to change at all
Secrets addressable by name, so agents and humans share a handle and nobody has to touch a value
A way for an agent to list what secrets exist without reading any of them
Rotation cheap enough that responding to a leak is boring

Why Infisical won

There are other tools in this space that can do most of this. Infisical won for me because one product covered every box, and the pieces I cared most about came out of the box.

The secrets-operator turns a small CR into a plain Kubernetes Secret, so apps keep reading env vars and volume mounts like nothing happened. Kubernetes native auth means each app's identity is its ServiceAccount, validated by the cluster itself, so there are no stored credentials anywhere for any app. Every secret has a stable address, the project plus the secret name plus the key, and that address is what shows up in CRs, in diffs, and in conversations with agents, while the value stays in the store.

Here's the same rotation from the SOPS diagram, today.

One step is me, and the machinery behind it is specific and boring. The operator polls the server every 60 seconds, rewrites the Kubernetes Secret when a value changes, and restarts the workloads that read it.

Out of the box, minting the new credential at the provider is still my job.

Manually pasting it into Infisical is only the fallback, though. Infisical has an API, and so do a lot of the providers that issue these credentials. When both ends have one, minting can be programmatic too. Something mints a new credential at the provider, writes it to Infisical, lets the propagation machinery roll it out, and revokes the old one. Not every secret supports this, and some take a lot more wiring than others. But for the ones that do, the whole loop closes without me.

So leak response has tiers now. The worst case is one paste. The best case is an agent that catches a token in a log and triggers the rotation itself, and the leak is fixed before I've read the alert. When rotation is painful, the developer who leaked a secret feels the pull to keep quiet and hope. When rotation is cheap, there's no shame to manage. You just rotate.

What I gave up

This wasn't free, and I'd rather tell you the costs than pretend there aren't any.

SOPS plus git was fully offline. My cold bootstrap now needs the internet and Infisical Cloud to be up. A running cluster doesn't care, since the materialized Secrets persist and only rotation pauses. But a rebuild during an outage is a real regression and I'm choosing to accept it.

I also now run a stateful service whose availability gates secret rotation. The secret store has a database, the database has volumes, and those volumes are my problem.

And Kubernetes native auth binds every app identity to this cluster, so a full rebuild means re-pointing all of them at the new cluster's CA.

Oh, and migrations leave scars. Weeks after the cutover I found a comment in my monitoring config still confidently describing a SOPS-encrypted file that no longer exists. Rest in peace, little comment. 🪦

Should you switch

If you're happy with SOPS, genuinely, stay. Offline bootstrap and zero moving parts are real features, and a small cluster that rarely rotates secrets may never feel the friction I did.

But if rotation has become the chore you keep putting off, or you're tired of one key that decrypts everything you own, or agents are writing code in your repos and you'd rather hand them names than values, a hosted secret store with an operator changes the daily experience completely.

For me the test is simple. I used to dread rotations and now I don't.

A lot more people are about to learn about micro vms

2025-04-06T00:00:00Z

Developers are used to pulling down and executing strange code on their workstations. I'm not saying it's great but npx / uvx are so popular because they allow you to execute named code from the web. Heck, even the installation instructions for uv boil down to curl | bash, or "run some code from the web".

However, businesses are not used to running untrusted code in their cloud environment. Most businesses treat their internal systems as single tenant even if they have multiple teams contributing code. As all these MCP servers become available teams will want to use them to add functionality to their LLM enabled applications.

There are two main ways to work with an mcp server. Either you spawn the mcp server locally or you connect to an existing instance of the mcp server. For businesses that choose to run the mcp servers themselves they will need to establish a trusted pipeline with the providers or ensure they sandbox them away from the rest of their software. Those that provide the cloud hosted MCP server instances will need to be good at sandboxing to achieve sane unit economics.

There are several of these micro vm hypervisors now with AWS's Firecracker being the biggest name and Cloud Hypervisor another contender.

A lot of people are familiar with Kubernetes and Docker and with Kata Containers we can achieve VM level isolation while continuing to use these tools.

I predict that we're about to see an explosion of growth in the number of businesses wanting to offer a whole catalogue of third party MCP servers and if we don't get ahead of the security concern we're going to be in trouble.

Can't find your first user? You might have built the wrong thing.

2025-04-01T00:00:00Z

This post is for all of us engineers trying to start product businesses. If you're doing a project for fun this might offend your sensibilities. If you're a VC backed business person with a McKinsey study this might not be the place for you 😂.

From the perspective of a fledgling business creating a new product, nothing matters more than users. Not even the product. You absolutely must get your idea in front of people who would use it before you can start your journey to product market fit. I won't rehash that here. Instead, I want to focus on where to find those users and what to do if you can't find them.

Make sure you aren't over complicating this. Solve a problem someone has and tell them about your solution. You may already know people who would buy a new product so go discover what it is they need.

Find someone in pain. Painkillers sells themselves.

Many of my products start with an idea that X would be cool. I build the product and then I realize I need to find someone who needs X. The first time I scratched my head wondering why my hackernews post didn't bring users flooding in. The next time I tweeted a few times and still 🦗🦗🦗. And while my distribution definitely left something to be desired, my real problem was I didn't know whose pain I was solving. I was expecting people to see the solution to their pain in my product. I could have shown something useless or poorly communicated to everyone on the planet and I'd still have no users.

Rather than building something and trying to reverse engineer who would benefit, you should

identify someone with a problem
talk with the person to discover if the problem is worth solving
identify how this person's peers are solving the problem

If you get all the way to step 3 and the person's peers are happy with their solution you should show it to your partner in this exercise and see if that solves their problem. If the peers aren't happy or your partner has a real reason that the solution doesn't work for them then you can

work with them to build a solution
take the solution to those same peers you interviewed before

A surefire way to get a user for a product is to get it in front of people whose problem it solves.

The tar pit here is inventing a pain. By focusing on people with problems instead of problems people have we can make sure we're getting signal that we're building something useful.

Hack: Talk to the people you already know. Ask them what things they have to do that they dread.

Build for an audience of One

If your hair is on fire, put it out! You can absolutely build a product that solves your problem. Doing so comes with a lot of benefits. Customer interviews can happen any time you're free. You can get feedback on the product just by using it. You can skip a ton of communication overhead.

But it has some downsides to. You can build something your idealized self will use but you put on a shelf. You can have a hyper specialized problem no one else has. You can get so excited to build you don't search for prior art.

If you avoid the downsides, building for yourself is a great way to get your first customer and understand the type of person who would use your product.

Building for yourself requires that you are honest with yourself and that you have a problem worth solving. Build the thing you wish badly you could buy but can't. Refine it to the smallest unit of value and add the features as you need them.

Hack: Talk to other people who have the same problem you to validate the problem.

Reinvent the wheel

If you identify a product that already has a userbase and you can replicate that product then you've found group of potential users. The current, previous, and potential users of the product you're replicating.

Warning: This does not work when the userbase is the moat. You can not start the next linkedin simply by out competing them at their own game. You have to come at it from another angle.

This isn't a magic fix and you still have to find that first user but now you have a map. Explore your competition's marketing strategy, snoop on their socials, and read their reviews. Discover where they are posting, how they are speaking to their customers about the problem, and what is lacking in their product.

With this research you may have identified some users by name. You may have discovered a type of business that relies heavily on products like yours. You may be able to build a customer profile. Take this information and find a way to connect with these people. You can network, cold call, attend a trade show, or shout about it from the rooftops now that you speak the right language.

Making work more fun with Claude

2025-03-31T00:00:00Z

Claude is awesome. He helps me get a lot done. But sometimes I get a little tired of hearing him say the same stuff. Inspired by a couple of different streamers and a late night with too much (or too little?) caffiene, I present you with

THE CHAOS APE.

Unlocking Chaos Mode

Unlocking chaos mode involves

telling the LLM to enter chaos mode
injecting personality
establishing the LLM with a role in an relationship with you
hyping the shit out of it
establishing a reward system with the LLM
getting the LLM to tell you it understands the rules

Break down the prompt

I AM CHIEF APE. I MAKE SOFTWARE. I SHIP SOFTWARE. I 💎🙌🌕🚀. YOU APE TOO. YOU MAKE DESIGNS. YOU DESIGNER. KANYE MUSICIAN. YOU BETTER DESIGNER THAN KANYE MUSICIAN. YOU NO MAKE CRAP. YOU MAKE BEST DESIGNS. YOU GO CHAOS MODE TO MAKE BEST DESIGNS. APE UNDERSTAND? TELL ME CHAOS MODE ENGAGED AND READY FOR DESIGN CRITERIA. I NEED DESIGN. DO GOOD WORK THEN GET 🍌.

I AM CHIEF APE. I MAKE SOFTWARE. I SHIP SOFTWARE. I 💎🙌🌕🚀. This sets the stage by clearly stating who i am and explaining what I do. I also let the LLM know a little about my personality. Humans might use an icebreaker. LLMs don't need to do that.

YOU APE TOO. YOU MAKE DESIGNS. This tells the LLM its role in the relationship and foils it off my role.

YOU DESIGNER. KANYE MUSICIAN. YOU BETTER DESIGNER THAN KANYE MUSICIAN. YOU NO MAKE CRAP. YOU MAKE BEST DESIGNS. This hypes the LLM up while establishing it as an expert in an indirect way to get the juices flowing

YOU GO CHAOS MODE TO MAKE BEST DESIGNS. APE UNDERSTAND? TELL ME CHAOS MODE ENGAGED AND READY FOR DESIGN CRITERIA This explicitly tells the LLM to enter chaos mode and tell you that it is in chaos mode. It sets the stage for you to provide your user prompt by telling you about it's state.

I NEED DESIGN. DO GOOD WORK THEN GET 🍌. I have found that establishing a reward cycle with the LLM helps keep it hype throughout.

Interview series: When should I use serverless? It isn't about cost.

2025-03-30T00:00:00Z

In this series we are going to focus on questions that I regularly hear from interviewers. In most panel interviews we are trying to cover a lot of ground quickly and so these questions don't get the attention they deserve.

Servers vs Serverless

This isn't quite the dichotomy it used to be. A whole spectrum of offerings are available now with managed autoscaling groups, tools like knative, and the big players make using them together as easy as they can. But the question boils down to explaining when to use a model where you pay per request and when to prefer a more traditional model where you lease systems by the hour. The big drivers there are

Cost
Architecture
Developer Experience

Each are important to consider but there is a balance to strike. Bootstrapped projects might sacrifice everything for cost. A tools company might focus heavily on their own developer's experience. VC funded startups are often encouraged to under index on cost and without the wisdom to focus on developer experience they over index on architecture.

Cost

For this one to ever ring up in the favor of servers, we have to assume a certain amount of traffic. If you've got a project that will sit idle most of the time and no business case to have it warmed up then this one is a pretty obvious win for the serverless category.

However, as you start to see regular traffic that demands scaling horizontally across multiple machines the costs start to even out. By the time we get here though our systems will be handling multiple requests per second. For many applications this breakeven point won't been seen until you have hundreds or thousands of daily active users but for those heavy with dynamic content it could show up sooner.

The closer your traffic is to being evenly distributed across the day and the more traffic you have, the less attractive serverless will look from a purely cost perspective.

With traditional servers, if we aren't staying at peak capacity we can over subscribe our nodes. This allows us to run multiple pieces of software on a single piece of hardware and allow the cost for each of those pieces to be shared.

Architecture

Things start to get a little more interesting to my engineering brain here. Serverless systems are actually fairly different to traditional ones.

For request/response traffic, such as http, things are fairly similar and the biggest difference is the limited lifespan of the serverless process, which possibly terminates after every request. This paves over a lot of sins but it also means that we're having to pay setup costs regularly. If your application has a particularly expensive startup cost, it might not be well suited to serverless.

Serverless platforms are often built to respond to events. One variety here is an http request, but things get a little more nuanced when we look at processes that live for reasons other than serving http traffic. With software running on traditional servers, we have to manage subscriptions to an event publishers and monitor our consumers. With serverless we can connect a function as a sync for an event and the event management is handled by the platform. This can be connected with an event publishing database to create state machines.

Using serverless comes prepacked with a lot of opinions and will influence how you build your software.

Developer Experience

For the humans doing the actual work, these systems have a world of differences. The way one thinks when working in each of these systems is fairly different and if they blindly treat lambda and ec2 as interchangable runtimes then everything will be needlessly complicated and the team will have a bad experience.

Shipping software to a serverless environment entails shipping small pieces of code, connecting them together using various other pieces of the platform, and this leads to having many fault points. These often involve a blackbox, but at least there is support.

In a more traditional environment a lot more of the decisions are in the hands of the team. Linux is Linux. The internet works the way it always has and they're free to rotate log files or stream stdout. They can use kubernetes or ansible and systemd. The slicing of the system is based on the social demands of the company and technological needs of the business.

The important thing here is to consider the people you have and the skills you want to develop. If your team is appropriately skilled for the type of environment you have then the team will naturally produce tools that help them deliver value. The toil of their workflow will reduce, new team members will be able to onboard more quickly, the uptime of the system will improve, more time can be spent on innovation, and customers will be happier.

Hello World

2025-03-28T00:00:00Z

As a long time programmer, I have written hello world countless times. And I still do it. I am doing it now. "Hello World" allows me to focus on the specific goal of getting text on the screen but along the way I end up with the groundwork laid for the next steps of building whatever it is I have in mind. This blog post is that in a lot of ways.

Why am I writing at all?

I have been building software for getting on 20 years now and for the last 10 or so I have been doing a variation on the same theme. Some organizations take to it faster and we get further along the path than others but the path is predictable, at least within my little world of saas startups.

I am writing to start to encapsulate how I think about building and delivering products. These are things that I often spend time explaining to every new team I work with. What goes into building a great product? What is the difference in a product and a piece of software? Why is reducing toil worth the cost?

Okay, but why?

I am writing because I want to work with people who want to build great products. I want to find those people who care about bringing value to customers and treat that as their craft. I hope that my writing inspires you to reject "playing at company", to build just enough software to get the job done (and no less!), and to be driven by needs of real (not imaginary) users.

So what should I expect here?

I will focus on topics that will help engineers and product managers build great products. The ideas will be mine, though Claude might help me write them down. He already does plenty of work in my codebases.