Arch is a critical piece of the agent infrastructure stack because autonomous agents are inherently 'chatty' and require high-frequency model interactions. For an agent to be reliable, it needs model fallbacks, cost controls, and specialized caching—functions that Arch provides at the gateway level. By offloading these infrastructure requirements from the agent logic itself, developers can build more resilient systems that are model-agnostic and easier to debug.
Furthermore, Arch's focus on semantic routing and prompt governance is essential for agents that handle sensitive data or operate in regulated environments. The ability to mask PII or enforce safety guardrails before a prompt reaches a third-party model is a key requirement for production-grade agent deployments. Arch pushes the ecosystem forward by treating LLM interactions as a networking problem, providing the 'traffic control' necessary for the next generation of agentic software.
Arch represents a fundamental shift in the generative AI stack, moving beyond simple API wrappers to a dedicated infrastructure layer. The core problem Arch addresses is the 'messy middle'—the space between an application's business logic and the disparate set of Large Language Models (LLMs) it might call. In a production environment, developers face significant overhead managing rate limiting, security, prompt versioning, and cost across multiple providers. Arch centralizes these functions into an intelligent, LLM-native gateway.
Built on top of the Envoy proxy—the high-performance networking tool that powers service meshes at companies like Google and Netflix—Arch brings traditional networking discipline to the non-deterministic world of LLMs. It operates as an L7 gateway, meaning it understands the content of the traffic passing through it. This awareness allows for semantic routing, where the gateway can analyze a prompt and decide which model is best suited to handle it based on complexity, cost, or latency requirements. For example, a simple query might be routed to a small, local model like Llama 3, while a complex reasoning task is sent to GPT-4.
The company is led by Brendan Falk and the team previously responsible for Fig, the terminal autocomplete tool that was acquired by Amazon Web Services in 2023. This pedigree is significant; the team has a history of building high-utility developer tools that improve efficiency at the lowest levels of the software stack. With Arch, they are applying that same philosophy to AI infrastructure. The project is primarily written in C++ and uses WebAssembly (Wasm) for its extensibility. This technical choice ensures that adding Arch as a 'hop' in the request chain introduces negligible latency, a critical factor for real-time agentic workflows.
While many developers initially turned to orchestration libraries to manage their AI logic, Arch argues that these concerns are better handled by infrastructure. By moving PII masking, safety guardrails, and semantic caching into the gateway, Arch allows developers to keep their application code clean and focused on business logic. This separation of concerns is particularly valuable as companies scale their AI usage. It provides a single point of observability and control, making it easier to audit model interactions and swap providers without redeploying code.
In the competitive field of AI proxies, Arch distinguishes itself by being open-source and standard-aligned. By extending Envoy rather than building a proprietary stack from scratch, Arch offers a path for enterprises that are already comfortable with modern cloud-native networking. As the industry moves from experimental chatbots toward autonomous agents that perform hundreds of calls per minute, the demand for a stable, performant gateway like Arch is likely to grow. The company is betting that the winning AI stack will look a lot like the current web stack, with a specialized proxy sitting at the center of every conversation.
An intelligent, Envoy-based gateway for managing LLM traffic and governance.
Arch is hiring.