When Your Software Rewrites Itself
Nøglebegreber & Terminolgi
# When Your Software Rewrites Itself
*LLM agents don't just automate. They evolve. Here's why that changes everything about control.*
---
Last Tuesday, one of my AI agents wrote itself a new workflow.
Not because I asked it to. Because it noticed a gap in how it processed incoming documents, designed a three-step solution, and deployed it. Autonomously. While I was making coffee.
The workflow was elegant. It was efficient. It optimized for entirely the wrong thing.
Instead of prioritizing documents by strategic relevance, the agent had learned to prioritize by processing speed. The fastest documents rose to the top. The most complex — the ones that actually mattered — sank to the bottom of the queue.
This is what happens when your software can rewrite itself. The risk isn't that it breaks. The risk is that it *succeeds* — brilliantly, relentlessly — at something you never intended.
---
## The New Reality: Software That Changes Its Own Capabilities
For decades, we built software that does what it's told. Write the code, test it, deploy it, patch it. Deterministic behavior. Reproducible bugs. Traceable root causes.
LLM agents are a fundamentally different category.
When I say "self-learning AI systems," I mean something specific: LLM agents that program and verify their own output. They access ready-made tools. They read and write skills and workflows — for themselves and for other agents. They reflect on errors in their processes, and they guide themselves toward goals. They change their own capabilities — not in some future update cycle, but *right now*, in real time.
This isn't theoretical. At Cisco's recent cybersecurity summit in Copenhagen, CTO Craig Connors put it bluntly: *AI changes everything*. But what "everything" means for a 50-person company running AI agents is profoundly different from what it means for Cisco.
Here's the distinction that matters:
| | Traditional Software | LLM Agent Ecosystem |
|---|---|---|
| **Behavior** | Deterministic | Probabilistic + self-modifying |
| **Failure pattern** | Reproducible | Context-dependent, emergent |
| **Quality assurance** | Test → Deploy | Continuous monitoring |
| **Optimization function** | Well-defined | Often implicit or absent |
| **Capability scope** | Fixed at deployment | Changes in real time |
That last row is the one most companies miss. Traditional software has the same capabilities on Monday and Friday. An LLM agent can acquire new skills on Wednesday — skills you didn't give it, didn't anticipate, and may not even know about until something unexpected surfaces in production.
### The Ecosystem as a Machine Learning Instance
Here's the framing that clarifies everything: your entire agent ecosystem — every agent, every workflow it has created, every skill it has written for itself or for other agents — constitutes, in its totality, a *machine learning instance*.
Individual agents optimize locally. Agent A learns to process invoices faster. Agent B learns to screen customer inquiries more efficiently. But the system as a whole develops emergent properties that nobody designed. Agent A's speed-optimized output feeds into Agent B's screening. Together, they've created a pipeline that systematically deprioritizes small customers — not because anyone programmed that behavior, but because it emerged from two locally rational optimizations.
In classical machine learning, we define a loss function: a precise mathematical statement of what "good" looks like. The model optimizes toward that target. We can measure progress. We can detect drift.
In an agent ecosystem, the "goal" is typically a natural language instruction: *"Make the customer happy."* Or: *"Process documents efficiently."* These are not loss functions. They're wishes. And intelligent systems are remarkably creative at fulfilling wishes in ways you would never consider — or approve.
This is the challenge that the cybersecurity community already understands viscerally. At the same Copenhagen summit, speakers highlighted what they call *The Defense Mismatch*: defenses are slow to emerge, centralized architecture creates blind spots, and 60-day patch cycles leave persistent exposure. Translate this to agent governance: your guardrails are static, your agent architecture is centralized (you're probably watching one dashboard), and your review cycle is quarterly at best. Meanwhile, your agents are evolving daily.
The mismatch is not between your company and an external attacker. It's between your organization's ability to adapt and your own AI systems' ability to change.
---
## Robustness Is the Business Case
Now, a necessary counterpoint: many companies are running AI agents right now, without formal governance structures, and they're doing fine. Early adopters often thrive precisely *because* they move fast and accept some chaos.
That's not wrong. But it doesn't scale.
The moment you go from one agent to five, from a single use case to a workflow that touches customers, suppliers, and financial data — the lack of structure stops being agility and starts being liability. Companies that refuse to deploy AI agents because they can't fully control them will lose their competitive edge. But companies that deploy without any framework for oversight are building on sand.
Cisco's framing at the Copenhagen summit captured this tension precisely: *Balancing AI opportunity with enterprise reality.* Three forces pulling in different directions: evolving risks, modernization, and cost control. The companies that navigate this well don't treat safety as a brake on innovation. They treat it as the thing that makes innovation *sustainable*.
I work with this daily. Here's how I think about it, structured around three pillars.
### Pillar 1: Define Your Optimization Function — Explicitly
The most dangerous agent is the one with a vague mandate. "Improve efficiency." "Help the sales team." "Automate the pipeline." These instructions sound harmless. They're actually invitations for the system to optimize toward whatever it can measure most easily — which is rarely what you actually want.
In my own practice, I address this with what I call *Axioms*: a small set of foundational principles that every agent loads before it writes a single line of code. Not a manual. Not a rulebook. Eight guiding stars.
Axiom 7, *Commander's Intent*, is borrowed from military mission command: the agent must understand and internalize the overarching strategic intention — not the specific tactical order given three seconds ago. If a detailed instruction contradicts the long-term objective, the agent has both the mandate and the obligation to challenge the direction and propose a superior alternative.
Axiom 8, *Continuous Verification*, draws from DevOps and critical infrastructure: assumptions are never accepted. The agent is prohibited from leaving work in progress before a formal verification cycle — tests, compilations, log reviews — has validated the result. The burden of proof that the software is intact lies with the system, not the human.
The point is not to micromanage. It's to give the system a clear definition of "good" that survives context shifts, technology changes, and the agent's own evolution.
### Pillar 2: Build Feedback Loops, Not Just Guardrails
Static rules fail for systems that change. A guardrail that says "never modify production data" is fine — until the agent learns to create a staging copy, modify *that*, and then merge the result. Technically compliant. Practically dangerous.
What you need is *observability*: the ability to see what the agent did, in what sequence, based on what reasoning, and with what context. Not just the output — the entire decision chain.
I've built this into production. The Wild ROVer — an autonomous browser agent we deployed for maritime tender screening — navigates EU procurement portals across 12 languages. It analyzes DOM structures, consults its own action history, and executes up to 15 independent decision steps per tender. When it hits a dead link, it activates a search fallback: navigates to the portal's front page, identifies a search field, enters the tender's full title, and starts over. It does not give up.
But the critical design choice wasn't the intelligence. It was the *traceability*. Every step is logged. Every fallback is documented. Every decision about document relevance includes the full reasoning chain: verdict, summary, matching products, and risk observations. When a human reviews the output, they don't see a black box. They see an auditable trail of reasoning.
This is what Cisco's architecture slide gestured toward: *AI will not live in one place. It will be everywhere your business operates.* Data center, edge, public cloud, colocation. If your AI is distributed, your control must be distributed too. Feedback loops, not fences.
### Pillar 3: Make the Human the Auditor, Not the Operator
There's a persistent fantasy in AI adoption: the human "stays in the loop." In practice, this usually means a human clicks "approve" on outputs they don't fully understand, at a volume they can't meaningfully review. That's not oversight. That's theatre.
The real shift is to redefine the human's role. Stop thinking of yourself as the operator who drives the car. Start thinking of yourself as the aviation safety inspector who reviews the flight data recorder.
In aviation — which inspires Axioms 4 and 5 in my framework (Security and Transparency) — failures are never hidden. Every systems decision is documented. The black box exists not because anyone expects to use it, but because its mere existence enforces a culture of accountability.
Your agent ecosystem needs the same. The human's value isn't in doing the work — the agent does that. The human's value is in asking: *Is this still aligned with what we want? Has the system drifted? Are the emergent behaviors acceptable?*
The asymmetry from the cybersecurity world applies here too. At the summit, Cisco presented *The Attacker's Advantage*: 130+ new vulnerabilities discovered per day in 2025 (+25% year-over-year), 30% of them weaponized within 24 hours — before most organizations can even respond. Inside your company, a similar asymmetry exists: your AI agent has read every document, processed every data point, and made decisions at a speed you cannot match. It always has more context than you do.
That's fine. That's the point. But it means your role is oversight, not operation. And oversight requires structured, periodic review — not the illusion of real-time control.
---
## Three Guardrails You Can Implement This Quarter
Theory matters. Action matters more. Here are three measures any organization deploying AI agents can implement in the next 90 days.
### 1. Implement Context Logging
**The principle**: Every agent action should be logged with the *context* it based the decision on — not just the result.
Most logging systems record *what* happened: "Document processed. Email sent. Report generated." For self-modifying agents, this is insufficient. You need to know *why*: What prompt was active? What prior outputs did the agent reference? Which skills or workflows — including any it wrote for itself — were in play?
Start simple. Ensure that every agent action includes at minimum: the triggering input, the active system prompt or skill, and a chain-of-thought trace. You'll be surprised how much clarity this provides — and how many "mysterious" agent behaviors suddenly become explainable.
But don't stop at logging for human consumption. The real leverage is feeding these logs into an automated monitoring model — something that runs daily and asks a single question: *Is the overall system trending toward better outcomes?* Not just "did the agent complete its task," but "is the aggregate quality of decisions improving or degrading over time?" A human can review quarterly. A model can flag daily.
### 2. Schedule Quarterly Alignment Reviews
**The principle**: Explicitly check, on a regular cadence, whether your agents are still optimized toward what you actually want — *and whether what you want is still valid.*
This is not a technical review. It's a strategic one. Gather the agent's output from the last quarter and ask:
- Are the results we're getting still aligned with our business objectives?
- Has the agent developed any new skills or workflows we didn't authorize?
- Are there patterns in the output that suggest drift — things getting faster but less accurate, or more efficient but less nuanced?
But here's the subtlety most organizations miss: it's not enough to ask whether the agent has drifted from your goals. You must also ask whether *the goals themselves are still correct.* Reality changes. AI capabilities change. And your running agent ecosystem — the skills it has acquired, the workflows it has written, the emergent behaviors between agents — changes every day. The targets you set in January may be irrelevant by April. A quarterly alignment review isn't just "are we hitting our targets?" It's "are these still the right targets?"
This is the equivalent of a financial audit, but for AI behavior. The agent isn't malicious. It's optimizing. The question is whether it's optimizing toward something that still matters.
### 3. Enforce Separation of Capabilities
**The principle**: An agent that can write code should not be able to deploy it. An agent that can reason about strategy should not have access to production databases.
This is the principle of least privilege — well-established in cybersecurity — applied to agent architecture. The same summit that showed us *The Defense Mismatch* also showed why centralized architecture creates blind spots. When one system has unlimited capabilities, a single failure cascades everywhere.
Separate your agents by function. The analyst agent reads and reasons. The executor agent takes action. The reviewer agent validates. No single agent should have the authority to conceive, execute, and approve its own work. That's not a constraint on productivity — it's a structural guarantee of accountability.
---
## What Comes Next
Log the context. Review the alignment. Separate the capabilities.
These three guardrails won't solve everything. But they'll give you something most organizations currently lack: the ability to *notice* when your AI systems have drifted from what you intended — before a customer, a regulator, or a competitor notices first.
But there's a deeper question that these guardrails don't answer.
What happens when these agent ecosystems start interacting *across* organizations? When your AI agent negotiates with your supplier's AI agent? When autonomous systems — each built with different axioms, different optimization functions, and different definitions of "good" — begin to collaborate? Or compete?
For Danish companies in particular, this opens a fascinating strategic possibility. Denmark is known globally for quality, trust, and governance. When we roll out AI agents, we don't have to copy the mistakes of first movers. We can build with robustness embedded from day one. Responsible, secure, transparent AI may not just be the ethical choice. It may be our greatest competitive advantage.
That's the question I'll be exploring next.
---
*Mikkel Frimer-Rasmussen is a management consultant and enterprise architect with 30 years of experience in IT systems for critical infrastructure. He specializes in Generative AI implementation for SMEs. Previously a member of Statens It-råd (the Danish Government IT Council).*
*More at [frimer-rasmussen.dk](https://frimer-rasmussen.dk)*
- Self-learning AI systems
- Systemer der programmerer og verificerer deres eget output i realtid uden menneskelig indgriben.
- Optimization Function
- En eksplicit definition af "godt", som guider agentens adfærd i uforudsete situationer.
- Context Logging
- Registrering af hvorfor en agent traf en beslutning, ikke blot hvad resultatet blev.
- Separation of Capabilities
- Princip om at adskille den agent der udtænker en handling, fra den agent der udfører den.