Beyond the LLM Call: Anatomy of a Production AI Application
A production AI application must separate bounded, latency‑sensitive query work from unbounded ingestion tasks. By using durable queues, idempotent processing, and clear resource limits, developers can build scalable, cost‑efficient systems on AWS that serve real users reliably.
When developers first experiment with large language models (LLMs), the simplest pattern works: a user sends a prompt, the application calls an LLM API, and the response is returned. That pattern is fine for demos, but it breaks down once the system must handle real documents, multiple tenants, unpredictable traffic, and strict uptime guarantees. The LLM becomes just one component of a larger distributed system that must ingest data, retrieve context, control cost, and survive failures.
Why the Simple Demo Pattern Fails in Production
In a demo, a single HTTP request can drive the entire pipeline: upload a file, extract text, chunk it, generate embeddings, index the vectors, and then answer a question. In production, the same request can take minutes, hit provider throttles, crash midway, and tie up worker capacity that should be reserved for fast query responses. The result is head‑of‑line blocking, timeouts, and unpredictable cost spikes. The root cause is treating two fundamentally different workloads—interactive queries and bulk ingestion—as if they share the same runtime constraints.
Defining Bounded vs. Unbounded Work
Bounded work is everything that can finish within a short, predictable window. Typical query requests are bounded by:
- Maximum number of retrieved chunks (e.g., 8)
- Maximum context tokens (e.g., 8,000)
- Maximum model output tokens (e.g., 1,000)
- Strict timeout limits (e.g., 8 seconds)
- Limited retry attempts (e.g., 1)
Unbounded work includes document ingestion, OCR, large‑file processing, and any task that can take minutes or hours. These tasks must be handled asynchronously so that they do not block the user‑facing API.
Architectural Split: Query Plane and Ingestion Plane
Separating the system into two independently scalable planes keeps the user experience responsive while allowing heavy lifting to run in the background.
- Query Plane – Authenticates the caller, enforces tenant policy, applies rate limits, retrieves relevant context from a vector store, calls the LLM within a deadline, validates the output, and returns a traceable response.
- Ingestion Plane – Accepts uploaded documents, extracts and normalizes content, creates stable chunks, generates embeddings, indexes vectors, tracks document state, retries recoverable failures, and routes terminal failures for investigation.
In practice, the query plane is a fast, stateless service that talks to a cache, a vector store, and the LLM provider. The ingestion plane is a durable, event‑driven pipeline that uses queues, workers, and batch processing to handle large volumes of data reliably.
Key AWS Services and Their Roles
The example architecture relies on AWS services chosen for the properties they provide rather than brand recognition.
- Amazon API Gateway – The public entry point that handles routing, throttling, authentication, and request size limits.
- Amazon S3 – Durable object storage for uploaded documents. Clients receive pre‑signed URLs to upload directly to S3, preventing large payloads from clogging the API.
- Amazon EventBridge – Routes S3 object creation events to interested consumers.
- SQS – Durable queue that buffers ingestion tasks, providing backpressure and at‑least‑once delivery.
- Step Functions – Orchestrates the extraction, chunking, embedding, and indexing steps, allowing retries and error handling.
- ECS Fargate – Runs stateless query workers and stateful ingestion workers.
- Amazon Bedrock – The LLM provider that executes generation requests.
- ElastiCache Redis – Caches frequently accessed data to reduce latency.
- DynamoDB – Stores metadata about documents, chunks, and tenant policies.
- OpenSearch Serverless – Provides vector search capabilities for retrieval‑augmented generation.
- CloudWatch & OpenTelemetry – Observability stack for metrics, logs, and traces.
Ensuring Idempotency and Handling Retries
Because SQS delivers messages at‑least‑once, workers must be idempotent. A deterministic chunk ID derived from tenant ID, document ID, version, and chunk index guarantees that reprocessing the same message does not create duplicate vectors. This approach keeps the system’s state consistent even when workers crash or network partitions occur.
Managing Cost and Latency with Hard Limits
Unbounded context can silently inflate token usage and latency. By enforcing maximum numbers of retrieved chunks, token budgets, and output lengths, the system prevents runaway costs and ensures predictable performance. Rate limits per tenant further protect the overall service from abuse or accidental spikes.
What Happens Next?
With the architecture in place, teams can focus on improving model performance, adding new retrieval sources, or expanding to other AI workloads like agents or internal search. The key is to keep the user‑facing path bounded and move heavy, variable work into durable, retryable pipelines. As traffic grows, each plane can scale independently, and observability dashboards will surface any backlog or failure patterns early.
In short, a production AI application is a distributed system that treats the LLM as a service, not a bottleneck. By designing for bounded query work, unbounded ingestion, idempotent processing, and strict cost controls, developers can deliver reliable, cost‑effective AI experiences to real users at scale.
Why it matters
Building a production‑ready AI system requires more than calling an LLM; it demands a thoughtful architecture that separates latency‑sensitive queries from heavy ingestion, ensures reliability, and controls cost—critical for any business that depends on AI at scale.
Key points
- Separate bounded query work from unbounded ingestion
- Use durable queues to handle backpressure and retries
- Enforce hard limits on context and token usage to control cost
- Design workers to be idempotent to avoid duplicate work
- Leverage AWS services that match specific system properties
Frequently asked questions
What is the difference between bounded and unbounded work in an AI system?
Bounded work finishes within a short, predictable window (e.g., a user query), while unbounded work can take minutes or hours (e.g., document ingestion).
Why is idempotency important for ingestion workers?
Because message queues like SQS deliver at‑least‑once, workers may receive the same task multiple times. Idempotent processing ensures duplicates do not corrupt the system state.
