Building Lyte (Part 1): The Distributed Foundation and Real-Time Engine

The complete source code for Lyte is available on GitHub.

I am a backend engineer. My background is in distributed systems, Postgres, Redis, SQS, and event-driven architecture. I had zero applied AI experience.

When I started building Lyte — a platform that generates complete React Native (Expo) mobile apps from natural language prompts with live previews — I had a specific goal. I didn't want to build a toy demo. I wanted to understand how AI-native applications are structurally different from traditional backends.

The starting architecture was rock solid on paper: A NestJS API, a background worker consuming from AWS SQS, Postgres for state, and Redis for pub/sub. The AI pipeline was a simple wrapper: send a prompt, get an XML string back, parse it, and execute it.

The infrastructure was solid from day one. Or so I thought.

Within a week of testing, the system buckled under the weight of AI workloads. Here is how the infrastructure broke, and the distributed systems patterns required to fix it.

This is a two-part write-up. Part 1 (this one) is everything underneath the AI: the execution environment, the job dispatch guarantees, the concurrency control, the real-time streaming layer, and the scaling patterns we needed before production traffic. Part 2 is the AI itself: the agent loop, context retrieval, prompt architecture, self-healing code, and evals.

The Sidecar Architecture Decision#

The problem: To run an AI-generated React Native app, you need four things to happen concurrently:

1. The AI Worker writes files to a filesystem.
2. Shell commands (npm install, expo start) must be executed.
3. The Expo Metro bundler must watch that same filesystem for changes.
4. The output must stream back to the user in real-time.

My initial thought was a Shared EFS (Elastic File System) volume. The worker writes to EFS, the preview server reads from EFS, and a relay routes shell commands over HTTP.

It failed instantly. EFS I/O latency sits between 0.5–5ms per operation. Expo Metro's file watcher relies on near-instant (< 0.1ms) changes to trigger hot-reloads. The EFS delay broke hot-reloading completely. Shell command routing also devolved into a complex distributed systems problem.

The Solution: The Sidecar Pod Model.

We moved the Worker, the Preview (Expo) server, and the WebSocket Relay into the exact same ECS (Elastic Container Service) Task, sharing a localized ephemeral filesystem.

The Worker writes a file, Metro detects it instantly, and the app hot-reloads. Shell commands run locally in the exact same filesystem context.

The sidecar pattern is the same architecture Kubernetes uses for co-located workloads. The choice to use ECS on EC2 saved us 4× over Fargate at scale while achieving the exact same isolation model.

The Dual-Write Bug#

Early in development, the system dispatched generation jobs like this:

// 1. Save to Postgres
await db.insert('jobs', { id: jobId, status: 'QUEUED' });
 
// 2. Send to SQS
await sqsService.sendMessage(jobId);

In testing, a pattern emerged: some jobs were created in the database but never processed. The worker never saw them. They were stuck forever.

Root cause: The Node.js process would occasionally crash (or be restarted by ECS) between step 1 and step 2. The job existed in Postgres with status=QUEUED, but the SQS message was never sent. No recovery mechanism existed. The inverse was also possible: SQS send succeeded, but the Postgres transaction rolled back. The worker would wake up to process a job that didn't exist in the database.

The fix: The Transactional Outbox Pattern.

We moved to a two-phase commit strategy. We write the Conversation, the Job, AND an Outbox record in one atomic Postgres transaction.

await db.transaction(async (tx) => {
  await tx.insert('jobs', { id: jobId });
  await tx.insert('outbox', { payload: jobId, status: 'PENDING' });
});

A background poller reads the outbox and sends to SQS, marking rows as SENT.

This pattern is standard in distributed systems for financial ledgers. The insight here was applying it to LLM job dispatch to guarantee resilience.

The Polling Delay Problem#

Initially, the outbox poller checked the database every 5 seconds. This meant our average dispatch latency was 2,500ms. For a generative AI product that is supposed to feel instantaneous, a 2.5-second artificial delay before the AI even starts thinking is unacceptable.

Fix: PostgreSQL LISTEN/NOTIFY.

We added a Postgres trigger that fires NOTIFY outbox_inserted on every single INSERT to the outbox table. The Node.js poller maintains a dedicated LISTEN connection.

The wake-up happens in ~50ms.

Our dispatch latency plummeted from 2,500ms to near zero, while retaining the absolute safety of the Outbox pattern. We kept a 10-second polling timer as a fallback safety net (defense in depth), but it rarely fires.

The Sandbox Race Condition#

When a generation job is picked up by the worker, it needs a Docker sandbox to execute the LLM's commands.

Under load, we saw a massive problem. Multiple API requests (e.g., a user furiously double-clicking "Generate") for the same project would concurrently check findRunningContainer(). Both would see null. Both would proceed to create a container.

We ended up with two containers for one project: one visible to the user, and one orphaned, burning expensive compute in the background indefinitely.

The fix: Distributed locks using Redis.

// Acquire lock
const lockAcquired = await redis.set(
  `lock:sandbox:${projectId}`, 
  lockValue, 
  'PX', 
  30_000, 
  'NX' // Only set if it does NOT exist
);

The PX 30_000 (TTL of 30 seconds) prevents deadlocks if the Node process crashes while holding the lock.

Releasing the lock is equally dangerous. If you just call DEL, you might accidentally delete a lock that expired and was acquired by a new process. We use a Lua script for a Compare-And-Swap (CAS) delete:

if redis.call("get",KEYS[1]) == ARGV[1] then
    return redis.call("del",KEYS[1])
else
    return 0
end

Redis SETNX + Lua CAS delete is the minimum correct implementation of a distributed mutex.

Even after acquiring the lock, our code re-checks findRunningContainer() (a classic TOCTOU — Time-Of-Check to Time-Of-Use defense). This handles the edge case where lock acquisition is slow and another process successfully created the container in that microscopic window.

The Real-Time Bridge#

Dispatch was now safe, and every project got exactly one sandbox. But from the user's side, the product still looked like a spinner.

When a user clicks "Generate", the job is dropped into an SQS queue. The HTTP request finishes instantly. Meanwhile, a background worker spins up a Sandbox and runs a 5-minute loop of intense file I/O and shell execution.

If the user just stares at a loading spinner for 5 minutes, they will refresh the page. When they refresh the page, they lose their HTTP context.

We needed to stream the AI's internal thoughts and file writes directly to the user in real-time, completely bypassing the HTTP cycle.

The Fire-and-Forget Relay Problem#

Our first attempt at a WebSocket relay was naive.

When the Worker generated a file, it emitted an event to our WS Relay: "Write app/page.tsx." The Worker then immediately moved on to the next task. Fire-and-forget.

What went wrong:

1. If the Relay crashed mid-write, the Worker assumed the file was written. It wasn't. The app broke.
2. If the Relay was slow (disk I/O bottlenecks), the Worker kept generating at LLM-speed. The queue built up in memory until the Relay crashed with an Out-of-Memory (OOM) error.
3. If the Worker temporarily disconnected, it had no idea which files had actually been written when it reconnected.

Fire-and-forget is fine for logging. It is fatal for file system operations that downstream systems depend on.

The Five Fixes#

We had to harden the WebSocket relay using standard distributed systems patterns.

Fix 1: The ACK Protocol Every event emitted by the Worker now gets a UUID. The Relay must send back an Acknowledgment: { ack: true, eventId, checksum: SHA256(content) }. The Worker awaits this ACK (with a timeout) before treating the write as confirmed.

Fix 2: Idempotent Writes File writes are inherently idempotent. Writing the exact same content twice has no harmful effect. This means that if an ACK times out, we can safely blindly retry the write without needing complex deduplication logic on the Relay.

Fix 3: Staging-then-Promote Expo Metro (the bundler watching our files) would occasionally crash if it read a file while it was only half-written. The Relay now writes to /tmp/.staging/{path} first. Once complete, it calls fs.rename() to move it to /app/{path}. fs.rename is an atomic OS-level operation. Metro never sees a partial file again.

Fix 4: Backpressure Instead of fire-and-forget, the Relay client now maintains a count of pending ACKs. If pending ACKs hit 50, the emitEvent() function blocks. This backpressure propagates all the way up the stack:

• emitEvent() blocks.
• executeTool() blocks in the Worker.
• The for await (const chunk of llmStream) async iterator pauses.
• TCP-level backpressure reaches the LLM API stream itself.

We physically pause the LLM generation if our disk I/O can't keep up.

Fix 5: Reconnection with Replay On disconnect, we use exponential backoff with full jitter to prevent thundering herds. When the Worker reconnects, it queries Postgres: SELECT * FROM actions WHERE relayed_at IS NULL ORDER BY sequence_number. It replays all unacknowledged events in perfect order.

Multiplexing at Scale (Redis Pub/Sub)#

With the Worker-to-Relay connection hardened, we still had to push these events to the user's browser.

The problem: The API Gateway holds thousands of active WebSocket connections from users. The Worker doesn't know who the user is, and it isn't connected to them. Furthermore, we run multiple API Gateway instances behind a load balancer.

If Worker A emits an update for Project 123, but the user is connected to API Node B, how does the message reach them?

The Solution: Redis Pub/Sub Multiplexing.

We attached a Redis Pub/Sub adapter to Socket.IO. All API Gateway instances subscribe to the Redis cluster. When the Worker wants to update the user, it publishes a message to Redis:

// The Worker screaming into the void
await redis.publish(
  `project-updates:123`, 
  JSON.stringify({ type: 'progress', message: 'Writing page.tsx' })
);

Redis instantly broadcasts this to all API nodes. The node holding the specific WebSocket connection for project:123 catches it and forwards it to the browser.

We can scale the API nodes horizontally to handle millions of user connections, and scale the Worker nodes based on SQS queue depth. They never talk directly to each other.

The Frontend: Why Zustand?#

On the frontend, we needed to receive these events and update the UI instantly.

Initially, we tried using React's useEffect to manage the WebSocket connection. This was a nightmare. React's render lifecycle is completely at odds with a persistent, stateful socket connection. The socket would disconnect on hot-reloads, state updates would batch incorrectly, and the UI would stutter.

We ripped it out and moved to Zustand.

Zustand is an un-opinionated state manager that exists outside the React render tree.

export const useProjectStore = create((set) => ({
  events: [],
  connectWebSocket: (projectId) => {
    const socket = io(API_URL);
    socket.emit('subscribe_project', projectId);
    
    socket.on('project_event', (event) => {
      // Updates state directly. Subscribed React components auto-render.
      set((state) => ({ events: [...state.events, event] }));
    });
  }
}));

By decoupling the WebSocket connection from the React component tree, the UI became bulletproof. You could navigate away from the page, come back, and the Zustand store would still be holding the live socket connection, quietly buffering events.

The Scaling Bottleneck Audit#

The system was correct, and it was live in the browser. But as we prepared for real production traffic, a systematic audit revealed 9 critical bottlenecks that would have crushed it under load.

Here are the three most dangerous ones and how we fixed them using classic distributed systems patterns.

1. Database Connection Starvation#

Our Transactional Outbox poller maintained a persistent LISTEN connection to Postgres. Under heavy generation load, the poller was actively querying the database, aggressively competing for connections in our Node.js pool. The result: Standard HTTP API requests (like a user fetching their profile) would hang because the connection pool was exhausted by the background poller.

The Fix: The Bulkhead Pattern. We created two entirely separate TypeORM connection pools. The primary pool (max: 10) is exclusively reserved for HTTP handlers. The secondary pool (max: 3) is strictly for background outbox operations.

If the outbox gets overwhelmed, its pool saturates, but the primary API remains lightning fast. The ship doesn't sink.

2. Head-of-Line Blocking#

We were using a single SQS queue. If User A requested a massive 40-file regeneration (a 5-minute job), and User B requested a tiny 1-line text edit (a 3-second job), User B had to wait 5 minutes for User A to finish.

The Fix: Priority Queues. We split the pipeline into three separate SQS queues: high, default, and bulk. The Worker polls the high queue first. Quick edits for active users get immediate capacity, while heavy background generations fall to default.

3. Reactive Circuit Breakers vs. Proactive Rate Limits#

We had circuit breakers on our LLM providers (the provider abstraction that makes this per-vendor isolation possible is covered in Part 2). But 429 Too Many Requests is an expected state at high throughput. Running 429s through a circuit breaker would trip it, completely halting all outbound API calls for 30 seconds.

The Fix: Proactive Token Buckets. We placed an in-memory Token Bucket rate limiter before the circuit breaker. If Anthropic allows 2 requests per second, our Token Bucket only allows 2 outbound requests per second. It smooths the traffic proactively, meaning we rarely ever hit a 429, keeping the circuit breaker closed and the system flowing.

What I Actually Learned#

Traditional backend engineering experience is incredibly valuable in Applied AI.

When you build an AI application, you are inherently building an asynchronous, event-driven, distributed system. The AI is just a very slow, very expensive, non-deterministic database query.

• Fire-and-forget is an anti-pattern. If a downstream system depends on a state change, you must implement an ACK protocol and backpressure.
• Backpressure can control LLMs. Pausing an async iterator effectively halts TCP traffic, which pauses the LLM API stream. You can physically slow down an AI if your infrastructure needs time to catch up.
• State managers should not be tied to UI lifecycles. Managing WebSockets inside a React component is a mistake. Always manage persistent connections in a store that outlives the component.

If your infrastructure drops messages, creates race conditions, or suffers from I/O latency, the AI will fail. Fixing these bottlenecks proactively meant that when we finally turned our attention to the AI itself, the foundation didn't crack.

What's Next in Part 2#

The infrastructure was solid. But the AI pipeline was still a naive "LLM Wrapper". We were asking an LLM to generate XML, parsing it with regex, and hoping for the best.

It was a disaster.

In Part 2, we will explore how we ripped out the XML parser, migrated to native JSON Schema Tool Use (giving the AI actual hands), discovered the terrifying token economics of a recursive Agent Loop, fixed the context window crisis with pgvector and Hybrid Search, built a self-healing TypeScript compiler loop, and proved generation quality with an LLM-as-a-Judge eval framework.

Up Next: Building Lyte (Part 2): The Agentic Engine, Self-Healing, and Evals