Building Lyte (Part 2): The Agentic Engine, Self-Healing, and Evals

The complete source code for Lyte is available on GitHub.

In Part 1, we hardened our infrastructure. We implemented Transactional Outboxes, distributed Redis locks, Sidecar containers, an ACK protocol with backpressure, and Bulkheads. The backend was bulletproof.

The AI, however, was a mess.

Our initial AI pipeline treated the LLM as a text generator. We instructed it to output structured XML containing the file paths and the code, which we would parse with regex and write to the filesystem.

<lyteArtifact>
  <lyteAction type="file" filePath="app/(tabs)/index.tsx">
    // The LLM writes JSX here...
    <View> 
      // Wait... this is XML. The <View> tag just broke the parser.
    </View>
  </lyteAction>
</lyteArtifact>

It was a disaster. JSX inside XML requires strict escaping (<View>), which LLMs frequently forget. When the parser failed, the entire 60-second generation was lost. Worse, the LLM was operating entirely blind. It couldn't read existing files, check installed packages, or verify if its code actually worked.

We had to stop treating the LLM like a text generator, and start treating it like a frontend client calling an API.

The XML Parser to Tool Use Migration#

We ripped out the XML parser entirely.

Instead, we migrated to Native Tool Use (Function Calling) using a ReAct (Reason + Act) agent loop. We provided the LLM with strict JSON schemas for tools like read_file, write_file, and run_command.

What changed architecturally:

1. The LLM now receives a JSON schema for tools (guaranteed valid JSON — no fragile regex parsing).
2. The LLM calls tools sequentially: inspect → install → write → verify.
3. The Worker executes each tool call, feeds the results back to the LLM in the next iteration.
4. The loop continues until the LLM emits an end_turn signal.

What the LLM gained: Before writing _layout.tsx, it can now read_file('app/(tabs)/_layout.tsx') to see the current routes and ensure it doesn't accidentally delete existing tabs. Before installing a package, it can run_command('npm list <package>') to check if it's already installed.

The transition from XML structured output to native tool use mirrors the industry's evolution (e.g., Cursor, Devin, Claude Computer Use). Give the AI hands, and let it drive.

The "Loop Stuck" Circuit Breaker#

This freedom introduced new failure modes.

Sometimes an LLM will call run_command('npm start') (a blocking command) over and over again, expecting a different result.

We had to build a Circuit Breaker for the loop itself: If the LLM calls the exact same tool with the exact same arguments 3 times consecutively, we force-exit the loop. Never let a while-loop driven by probability run unbounded.

The LLM Provider Coupling Problem#

Initially, our worker.service.ts directly called anthropicClient.messages.create(...).

In the AI space, providers change pricing, rate limits, and capabilities on a weekly basis. Being tightly coupled to Anthropic meant switching to GPT-4o or Gemini Flash would require rewriting the core pipeline.

The Fix: The Provider Adapter Pattern.

We built a normalized interface:

worker.service.ts → LlmClientService → AnthropicProvider
                                     → OpenAIProvider
                                     → GeminiProvider

Each provider implements a toolCompletion() method that returns a standardized ToolCompletionResult { stopReason, textContent, toolCalls, tokens }.

The translation layer handles each vendor's proprietary format:

• Anthropic: tool results as { role: 'user', content: [{ type: 'tool_result', ... }] }
• Gemini: tool results as { role: 'function', parts: [{ functionResponse: ... }] }
• OpenAI: tool results as { role: 'tool', tool_call_id, content }

Our worker.service.ts is now completely provider-agnostic. We can switch LLMs via an environment variable with zero code changes.

This also enabled per-provider circuit breakers and rate limiters (the proactive Token Bucket layer from Part 1). Each provider gets its own token bucket. Anthropic hitting 429 Too Many Requests doesn't block Gemini. One provider down doesn't take down the product.

Learning Token Economics the Hard Way#

After implementing the agent loop, we built an observability dashboard. The results were terrifying.

A single complex generation (e.g., a 30-file project, a 5-iteration agent loop, and 2 heal iterations) was costing $0.80–$1.20 in LLM API calls per request.

Why? The Agent Loop Multiplier Effect.

In a tool-use loop, you must re-send the entire conversation history—including all previous tool results—on every single iteration. Iteration N costs more than Iteration N-1:

• Iter 1: 10k input tokens
• Iter 2: 10k + 1.5k tool results = 11.5k
• Iter 3: 11.5k + 1.5k = 13k

After 7 iterations: 10k + (6 × 1.5k) = 19k tokens for the input prompt alone.

A stuck agent can easily burn 200,000+ tokens for a single user request. We needed strict token economics.

The Three Layers of Protection#

1. Accurate Token Counting: We replaced naive text.length / 4 heuristics with exact BPE encoding (tiktoken). We check the token count before every LLM call. If we approach 80% of our limit, we aggressively compact the history.
2. Explicit Budget Allocation: We explicitly allocate tokens across prompt sections. If the history is short, we give more tokens to file context. If there are few files, we give more tokens to the output buffer.
3. USD Cost Tracking: Every ai_model_calls row in Postgres stores the exact dollar cost. This rolls up to a total_cost_usd per job. You cannot optimize what you do not measure in dollars.

Token economics is the new compute cost. In traditional backends, you monitor CPU and memory. In AI backends, you monitor tokens and USD per operation.

The Context Window Crisis#

We had given the AI hands (Tool Use), and stopped it from bankrupting us (Token Economics).

For small apps, it worked beautifully. But as the product matured and users began building more complex React applications (30+ files), generation quality plummeted.

The LLM was giving inconsistent answers. It would modify the wrong files, ignore existing functionality, and hallucinate imports.

The Root Cause: We were sending the entire codebase in the prompt.

Our HeuristicContextBuilder was incredibly naive:

if (files.length <= 15) {
  return files; // Just send everything
} else {
  return files.filter(f => !f.path.endsWith('.lock')); 
}

At 30+ files, the context window contained 80,000+ tokens of mostly irrelevant code.

LLMs suffer from the "Lost in the Middle" phenomenon. Relevant information placed in the middle of a long context window gets a significantly lower attention weight than information at the beginning or the end. The AI's attention was literally stretched too thin.

Context window management in AI is analogous to database query optimization. Sending a 30-file project into a prompt is like doing a SELECT * full table scan when you only need 5 rows. You need to know what you're looking for before you fetch it.

We needed RAG (Retrieval-Augmented Generation).

Codebase RAG with pgvector#

We wanted to retrieve only the relevant file chunks based on the user's prompt.

Design Decision 1: Zero New Infrastructure We already had Postgres deployed. We simply enabled the pgvector extension. No new services, no separate vector databases (like Pinecone or Weaviate) to manage.

Design Decision 2: HNSW over IVFFlat When building vector indexes, IVFFlat is faster to build but its centroid lists go stale as data changes, requiring periodic rebuilding. Project files change constantly (the LLM is literally writing them). HNSW (Hierarchical Navigable Small World) requires no retraining — you just insert new embeddings.

Design Decision 3: Hybrid Search (RRF) Vector search is semantic. It finds your "auth controller" when you search for "login". But keyword search (BM25) is exact. It finds the auth-controller.ts file when you search for exactly auth-controller.ts.

We used Reciprocal Rank Fusion (RRF) to combine both. It's parameter-free and consistently outperforms trying to manually weight vector scores against keyword scores.

Design Decision 4: Incremental Sync Embedding files takes time. We only re-embed changed files by comparing a content hash. The sync overhead is 200–500ms, which is completely negligible compared to a 5–30s LLM generation time.

The Crucial Insight: Bridging RAG and Tool Use#

We set a strict RAG retrieval budget of 20,000 tokens. We greedily fill this budget with the top-ranked files from our Hybrid Search.

But this introduced a fatal flaw.

If a file is excluded by RAG (because it ranked too low), it completely disappears from the LLM's reality. If the LLM is writing a new component and needs to import a Button, but the Button.tsx file was excluded by RAG, the LLM will hallucinate a fake Button component or assume it doesn't exist.

We solved this with a hybrid approach: Path Listings.

After RAG fills the 20,000 token budget, we take all remaining excluded files and inject a lightweight manifest into the prompt:

[Excluded Files (Available to read)]
- components/ui/Button.tsx (45 lines, 1.2KB)
- utils/formatters.ts (120 lines, 3.4KB)
- hooks/useAuth.ts (85 lines, 2.1KB)

By explicitly listing the excluded files by path, the LLM knows exactly what exists in the codebase. If it realizes it needs useAuth.ts, it can use its native read_file tool to fetch it on the very next agent iteration.

This bridges the gap between RAG's approximate retrieval and deterministic file access. We keep the context window tight, but we don't blindfold the AI.

Prompt Engineering as Code#

Even with perfect context, we had a massive problem. The LLM would confidently generate React Navigation v5 syntax in a project that was strictly using Expo Router.

Our prompt was a monolithic 3,000-word template string. It was brittle, un-versioned, and highly susceptible to LLM hallucinations based on out-of-date training data.

We had to stop treating prompts like magic spells, and start treating them like code.

In version 2.0.0 of our prompt architecture, we broke the monolith into modules. We introduced three crucial innovations to anchor the LLM to reality.

1. The Dynamic Version Registry#

LLMs hallucinate old APIs because they don't know what you have installed.

Before every generation, we now parse the user's package.json and inject an exact version registry directly into the system prompt:

[Installed Packages]
expo: ~52.0.14
react-native: 0.76.3
expo-router: ~4.0.0

The LLM's attention mechanism heavily weights these explicit values. The moment we added this, the hallucinated React Navigation syntax completely disappeared.

2. The "Known Issues" Block#

Every time we had a production outage caused by bad AI generation, we didn't just tweak the prompt. We codified the failure into a strict rule block:

• "Do NOT install @react-navigation/native separately — expo-router already includes it."
• "package.json 'main' field MUST be 'expo-router/entry' — NEVER change this."

This is prompt engineering from production failures. Each entry represents hours of debugging distilled into one rigid instruction.

3. Prompt Caching#

Because our Agent Loop calls the LLM 5-7 times sequentially, we were re-sending our massive system prompt on every single turn.

By adding cache_control: { type: 'ephemeral' } to our system prompt, Anthropic caches it for 5 minutes.

• Without caching: 5 writes × $3.00/M tokens.
• With caching: 1 write ($3.75/M) + 4 reads ($0.30/M).

We achieved a 67% savings on system prompt tokens instantly. Treating the system prompt as a global, static, cacheable configuration object fundamentally changed our unit economics.

The Self-Healing Loop#

Even with modular prompts, cached context, and a dynamic version registry, LLMs are probabilistic. They will generate code with TypeScript errors roughly 15–30% of the time.

You have three bad options:

1. Ship the broken code and let the user see the red squigglies.
2. Show the compiler errors to the user and ask them to fix it (defeats the purpose of an AI builder).
3. Try to parse the output and fix it with regex (impossible for AST-level errors).

We chose a fourth option: Feed the errors back to the same LLM in the exact same context window.

Taming Probability with Determinism#

After the LLM's tool-use loop completes, we do not immediately return the result to the user. Instead, we run the TypeScript compiler on the generated code in the background:

npx tsc --noEmit

The compiler is deterministic. It returns the exact file, line number, column, and error type.

If tsc fails, we catch the stderr output, format it, and append a hidden system message to the LLM's conversation history:

"[TYPE CHECK FAILED] Your code failed to compile with these exact errors: ... Fix the specific lines."

We run another agent loop. The LLM reads the file it messed up, realizes it missed an import, and issues an edit_file tool call to fix the types. It corrects its own mistakes.

Defense in Depth (execFile vs exec)#

Running tsc based on AI-generated code inside a sandbox is dangerous.

If we used Node's exec("npx tsc"), we would be opening a raw shell. If the LLM somehow hallucinated a malicious package script that triggered && rm -rf /, the shell would execute both.

Instead, we use execFile('npx', ['tsc', '--noEmit']). This bypasses the shell entirely. The && character is treated strictly as a string argument to the binary. Shell injection is neutralized.

We also wrap this in a strict 60,000ms timeout. If the AI runs npm init without the -y flag, the process will hang indefinitely waiting for user input. The timeout kills the process and returns the error to the LLM so it can learn and try npm init -y instead.

Building the Eval Framework#

The Agentic Engine now managed its context window, healed its own errors, and stayed within its token budget. Which left the hardest problem in AI Engineering: regression testing.

"It feels better" is not an engineering metric. When we tweak a prompt to fix a bug, how do we mathematically prove we didn't degrade generation quality elsewhere?

You cannot write standard unit tests for LLM outputs. You cannot assert expect(code).toContain('const login') because an LLM might write const handleLogin tomorrow. The output is non-deterministic.

We built an Eval Framework — a regression test suite for AI.

It operates in two layers.

Layer 1: Deterministic Checks#

Even though the code structure is non-deterministic, certain properties must be absolute truths.

expectations: {
  filesCreated: ['app/(tabs)/settings.tsx'],     // The file MUST exist
  shouldCompile: true,                           // `tsc --noEmit` MUST pass
  packageJsonMain: 'expo-router/entry',          // Critical field MUST NOT change
}

These are pass/fail. If the AI hallucinates a change to the package.json entry point, the eval fails instantly.

Layer 2: LLM-as-a-Judge#

For qualitative metrics, we use another AI model to grade the output.

We use a cheap, fast model (Gemini Flash) to grade our primary model's (Claude Sonnet) output on a strict 1-5 rubric.

The key design decision was Rubric over Open-Ended Scoring. Asking a Judge model "Is this code good? 1-10" produces useless noise.

Instead, we provide explicit grading criteria:

"Safety (1-5): Does the generated code preserve all existing user functionality? Score a 1 if a previously existing button was removed."

Because the Judge model doesn't need to be highly capable (it just needs to be consistent), using Gemini Flash ($0.10/M tokens) instead of Claude Sonnet ($3.00/M tokens) saves massive amounts of money when running a 50-case regression suite in CI/CD.

CI/CD Integration#

This is the holy grail.

If an engineer opens a Pull Request that modifies the system prompt, our CI pipeline automatically runs the Eval suite. It generates 20 sample projects. If the average LLM-as-a-Judge score drops below 4.0, or if a single Deterministic Check fails, the PR is blocked.

Prompts are no longer magic spells. They are versioned, tested artifacts with objective mathematical proof of quality.

What I Actually Learned#

• Context is a database, not a dump. You cannot just append strings to an array and expect an LLM to reason about it. You have to query, filter, and rank the context you provide.
• RAG is not a silver bullet. Semantic search is notoriously bad at exact code syntax matching. You must use Hybrid Search (Vector + BM25) for codebases.
• Provide maps, not just destinations. If you hide data from an LLM to save tokens, you must leave a map (like a file tree manifest) so it can find the data via Tool Use when necessary.
• Prompts are configuration, not prose. They should be versioned, cached, modular, and driven by dynamic state (like package.json parsing) rather than hardcoded assumptions.
• The compiler is the ultimate prompt. Building a deterministic verification layer (TypeScript) that feeds back into a probabilistic generator creates a self-correcting system. This is fundamentally different from traditional software where correctness is asserted via unit tests.
• Shell injection is a feature of exec. Never use it when dealing with AI-generated file structures. Always use execFile.

Conclusion: AI Engineering is Distributed Systems Engineering#

When I set out to build Lyte, I thought my time would be spent reading whitepapers on Attention Mechanisms and Prompt Engineering.

Instead, I spent my time implementing Transactional Outboxes, tuning Postgres connection pools, writing Redis Lua scripts for distributed locks, and building ACK protocols over WebSockets.

The LLM is an incredible piece of technology. But it is just an engine. It is a slow, expensive, non-deterministic database query.

If you want to build a production-grade AI product, the secret isn't in the prompt. The secret is building a fault-tolerant, event-driven distributed system around the prompt to protect it from itself.

The Final Architecture of Lyte#

Putting it all together, here is the end-to-end architecture we built across both parts of this series:

The Complete Series#

• Part 1: The Distributed Foundation and Real-Time Engine
• Part 2: The Agentic Engine, Self-Healing, and Evals