Building Lyte (Part 1): The Distributed Foundation and Crises in Concurrency

The complete source code for Lyte is available on GitHub.

I am a backend engineer. My background is in distributed systems, Postgres, Redis, SQS, and event-driven architecture. I had zero applied AI experience.

When I started building Lyte — a platform that generates complete React Native (Expo) mobile apps from natural language prompts with live previews — I had a specific goal. I didn't want to build a toy demo. I wanted to understand how AI-native applications are structurally different from traditional backends.

The starting architecture was rock solid on paper: A NestJS API, a background worker consuming from AWS SQS, Postgres for state, and Redis for pub/sub. The AI pipeline was a simple wrapper: send a prompt, get an XML string back, parse it, and execute it.

The infrastructure was solid from day one. Or so I thought.

Within a week of testing, the system buckled under the weight of AI workloads. Here is how the infrastructure broke, and the distributed systems patterns required to fix it.

The Sidecar Architecture Decision

The problem: To run an AI-generated React Native app, you need four things to happen concurrently:

1. The AI Worker writes files to a filesystem.
2. Shell commands (npm install, expo start) must be executed.
3. The Expo Metro bundler must watch that same filesystem for changes.
4. The output must stream back to the user in real-time.

My initial thought was a Shared EFS (Elastic File System) volume. The worker writes to EFS, the preview server reads from EFS, and a relay routes shell commands over HTTP.

It failed instantly. EFS I/O latency sits between 0.5–5ms per operation. Expo Metro's file watcher relies on near-instant (< 0.1ms) changes to trigger hot-reloads. The EFS delay broke hot-reloading completely. Shell command routing also devolved into a complex distributed systems problem.

The Solution: The Sidecar Pod Model.

We moved the Worker, the Preview (Expo) server, and the WebSocket Relay into the exact same ECS (Elastic Container Service) Task, sharing a localized ephemeral filesystem.

The Worker writes a file, Metro detects it instantly, and the app hot-reloads. Shell commands run locally in the exact same filesystem context.

The sidecar pattern is the same architecture Kubernetes uses for co-located workloads. The choice to use ECS on EC2 saved us 4× over Fargate at scale while achieving the exact same isolation model.

The Dual-Write Bug

Early in development, the system dispatched generation jobs like this:

// 1. Save to Postgres
await db.insert('jobs', { id: jobId, status: 'QUEUED' });
 
// 2. Send to SQS
await sqsService.sendMessage(jobId);

In testing, a pattern emerged: some jobs were created in the database but never processed. The worker never saw them. They were stuck forever.

Root cause: The Node.js process would occasionally crash (or be restarted by ECS) between step 1 and step 2. The job existed in Postgres with status=QUEUED, but the SQS message was never sent. No recovery mechanism existed. The inverse was also possible: SQS send succeeded, but the Postgres transaction rolled back. The worker would wake up to process a job that didn't exist in the database.

The fix: The Transactional Outbox Pattern.

We moved to a two-phase commit strategy. We write the Conversation, the Job, AND an Outbox record in one atomic Postgres transaction.

await db.transaction(async (tx) => {
  await tx.insert('jobs', { id: jobId });
  await tx.insert('outbox', { payload: jobId, status: 'PENDING' });
});

A background poller reads the outbox and sends to SQS, marking rows as SENT.

This pattern is standard in distributed systems for financial ledgers. The insight here was applying it to LLM job dispatch to guarantee resilience.

The Polling Delay Problem

Initially, the outbox poller checked the database every 5 seconds. This meant our average dispatch latency was 2,500ms. For a generative AI product that is supposed to feel instantaneous, a 2.5-second artificial delay before the AI even starts thinking is unacceptable.

Fix: PostgreSQL LISTEN/NOTIFY.

We added a Postgres trigger that fires NOTIFY outbox_inserted on every single INSERT to the outbox table. The Node.js poller maintains a dedicated LISTEN connection.

The wake-up happens in ~50ms.

Our dispatch latency plummeted from 2,500ms to near zero, while retaining the absolute safety of the Outbox pattern. We kept a 10-second polling timer as a fallback safety net (defense in depth), but it rarely fires.

The Sandbox Race Condition

When a generation job is picked up by the worker, it needs a Docker sandbox to execute the LLM's commands.

Under load, we saw a massive problem. Multiple API requests (e.g., a user furiously double-clicking "Generate") for the same project would concurrently check findRunningContainer(). Both would see null. Both would proceed to create a container.

We ended up with two containers for one project: one visible to the user, and one orphaned, burning expensive compute in the background indefinitely.

The fix: Distributed locks using Redis.

// Acquire lock
const lockAcquired = await redis.set(
  `lock:sandbox:${projectId}`, 
  lockValue, 
  'PX', 
  30_000, 
  'NX' // Only set if it does NOT exist
);

The PX 30_000 (TTL of 30 seconds) prevents deadlocks if the Node process crashes while holding the lock.

Releasing the lock is equally dangerous. If you just call DEL, you might accidentally delete a lock that expired and was acquired by a new process. We use a Lua script for a Compare-And-Swap (CAS) delete:

if redis.call("get",KEYS[1]) == ARGV[1] then
    return redis.call("del",KEYS[1])
else
    return 0
end

Redis SETNX + Lua CAS delete is the minimum correct implementation of a distributed mutex.

Even after acquiring the lock, our code re-checks findRunningContainer() (a classic TOCTOU — Time-Of-Check to Time-Of-Use defense). This handles the edge case where lock acquisition is slow and another process successfully created the container in that microscopic window.

What I Actually Learned

Traditional backend engineering experience is incredibly valuable in Applied AI.

When you build an AI application, you are inherently building an asynchronous, event-driven, distributed system. The AI is just a very slow, very expensive, non-deterministic database query.

If your infrastructure drops messages, creates race conditions, or suffers from I/O latency, the AI will fail. Fixing these bottlenecks proactively meant that when we finally turned our attention to the AI itself, the foundation didn't crack.

What's Next in Part 2

The infrastructure was solid. But the AI pipeline was still a naive "LLM Wrapper". We were asking an LLM to generate XML, parsing it with regex, and hoping for the best.

It was a disaster.

In Part 2, we will explore how we ripped out the XML parser, migrated to native JSON Schema Tool Use (giving the AI actual hands), decoupled our LLM providers using the Adapter pattern, and discovered the terrifying token economics of a recursive Agent Loop.

Up Next: Building Lyte (Part 2): The Agentic Engine and Token Economics