OpenAI’s Jalapeño results turn custom inference silicon into a strategic AI infrastructure story

OpenAI has published the first measured results from Jalapeño, its first custom inference chip, moving a long-rumored hardware effort from strategy into performance data. The company said on August 25 that Jalapeño delivered a stronger combination of throughput, latency and power efficiency than comparison systems on SemiAnalysis’ InferenceX benchmark. For an AI industry now pressed to show that massive compute spending can translate into sustainable economics, the announcement is more than a chip update. It is OpenAI’s clearest statement yet that serving models cheaply and quickly is becoming as important as training them.

Jalapeño is aimed at inference, the production phase in which models answer user requests, operate agents and generate output token by token. OpenAI said it tested the accelerator on GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T, a deliberate choice that shows the architecture is not limited to OpenAI’s own models. Across the three, the company reported 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than comparison systems. For highly interactive workloads, OpenAI said performance rose by 2.1 to 4.1 times.

The emphasis on per-watt performance reflects the changing economics of generative AI. Frontier labs have spent heavily on training clusters, but inference is the cost that repeats every time a product is used. Agents intensify that pressure because they do not only answer once. They read files, call tools, inspect intermediate results and loop through many steps. A small delay or extra energy cost in each step can compound across a long task, turning chip-level efficiency into a user-experience and margin problem.

OpenAI says Jalapeño was designed around those workloads rather than adapted from general accelerator assumptions. The system tries to reduce data movement by keeping model state, including the KV cache used during generation, close to the compute and networking resources that need it. That matters because language model serving moves through distinct phases. Processing a long prompt is compute-heavy, while generating each subsequent token can be limited by memory bandwidth and communication latency. A balanced inference system has to perform well across both phases.

The company also framed Jalapeño as evidence of a broader full-stack strategy. OpenAI said earlier models helped engineers design and bring up the chip, while newer models are assisting optimization and programming. It claimed that selected AI-generated implementations for attention and mixture-of-experts blocks ran 1.5 to 1.8 times faster than previous human-written versions, while noting that those results apply to selected blocks rather than full models. The point is not only that AI helped design hardware, but that future chips may be programmed and tuned by the same class of systems they are built to serve.

The hardware push does not mean OpenAI is abandoning outside suppliers. The company said it will continue using accelerators from NVIDIA and other partners for training and inference. Jalapeño instead gives OpenAI a first-party path for workloads where tight co-design among model, serving software, memory, network and chip can improve cost or responsiveness. That choice could also give OpenAI more leverage in a market where access to top-tier compute remains expensive and strategically sensitive.

OpenAI plans to begin deploying Jalapeño inside its own compute infrastructure by the end of the year, while continuing production qualification and software maturity work. Second- and third-generation designs are already in progress, according to the company. The benchmark claims will need independent validation at scale, especially under real customer traffic, but the direction is clear. The next phase of AI infrastructure is not only about who can build the largest model. It is about who can make increasingly capable models fast, efficient and affordable enough to run all day.