Engineering Manager, Inference Routing and Performance

San Francisco, CA | New York City, NY

About Anthropic

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems.

About the role

Every request that hits Claude — from claude.ai, the API, our cloud partners, or internal research — passes through a routing decision. Not a generic load balancer round-robin, but a decision that accounts for what's already cached where, which accelerator the request runs best on, and what else is in flight across the fleet. Get it right and you extract meaningfully more throughput from the same hardware. Get it wrong and you burn capacity, miss latency SLOs, or shed load that shouldn't have been shed. The Inference Routing team owns this layer. We build the cluster-level routing and coordination plane for Anthropic's inference fleet — the system that sits between the API surface and the inference engines themselves, making fleet-wide efficiency decisions in real time. As Anthropic moves from "many independent inference replicas" toward "a single warehouse-scale computer running a coordinated program," Dystro is the coordination layer. This is a deeply technical team. The engineers here design custom load-balancing algorithms, build quantitative models of system performance, debug latency spikes that cross kernel, network, and framework boundaries, and reason carefully about cache placement across thousands of accelerators. They work shoulder-to-shoulder with teams that write kernels and ML framework internals. The EM for this team doesn't need to write kernels — but they do need the systems depth to make architectural calls, evaluate deeply technical candidates, and spot when a proposed optimization will have second-order effects on the fleet. You'll inherit a strong team of distributed-systems engineers,... [truncated for brevity in this explanation but full markdown included in actual output] The annual compensation range for this role is listed below. For sales roles range provided is role’s On Target Earnings ("OTE") range including sales commissions/sales bonuses target & annual base salary. Annual Salary: $405000 - $485000 USD

Logistics

Minimum education: Bachelor’s degree or equivalent combination of education training/or experience Required field of study: A field relevant to role as demonstrated through coursework training or professional experience Minimum years of experience: Correlate with internal job level requirements Location-based hybrid policy: Currently expect all staff in one office at least 25% time; some roles may require more time onsite Visa sponsorship: We do sponsor visas! Not able to sponsor every role/candidate but will make every reasonable effort if offer made; immigration lawyer support provided We encourage you to apply even if you do not believe you meet every single qualification...

How we're different

We believe highest-impact AI research will be big science... [full content preserved]