Inference Optimization Engineer

About the role

An Inference Optimization Engineer is one of the rarest and highest-paid specialists in the AI industry, working on the physics of LLM serving: latency, throughput, memory efficiency, and cost-per-token. Where other engineers deploy models, this role makes them faster and cheaper to run at scale. The work draws on GPU architecture knowledge, LLM serving algorithms (PagedAttention by Kwon et al. 2023, speculative decoding by Leviathan et al. 2023), quantization theory (QLoRA by Dettmers et al. 2023), and TensorRT-LLM compiler passes. A single inference optimization that cuts latency by 30% across a high-volume endpoint can save seven figures annually in GPU cost. Salary anchors from Levels.fyi 2025-2026 data place this role at $220,000-$380,000 total compensation, making it the top-of-market specialty within AI Operations.

What this role actually does

• Profile LLM inference endpoints to identify latency bottlenecks at the kernel, memory, and network layers using NVIDIA Nsight and PyTorch profiler.
• Implement and tune vLLM serving configurations including PagedAttention block size, KV cache memory ratio, max batch size, and continuous batching settings to hit latency SLOs at target throughput.
• Apply and validate quantization schemes (INT8, FP8, GPTQ, AWQ) using TensorRT-LLM or llama.cpp to reduce memory footprint without unacceptable accuracy degradation.
• Evaluate speculative decoding (Leviathan et al. 2023) with draft models tuned to the target model family, measuring acceptance rate and wall-clock latency improvement in production traffic distributions.
• Build benchmark harnesses that simulate realistic request distributions (prompt length, output length, concurrency) to evaluate serving configuration changes before production rollout.
• Collaborate with AI engineers to set per-endpoint latency and cost SLOs, then deliver the serving configuration that meets both constraints simultaneously.
• Research and prototype new serving techniques from academic literature, evaluate them against production workloads, and write internal reports on feasibility and expected impact.
• Monitor serving metrics in production (TTFT, TBT, ITL, GPU utilization, KV cache hit rate) and respond to degradation with targeted configuration changes.

An average week

• Run at least one profiling session on a production or staging serving endpoint, identifying where compute time is actually spent versus where engineers assume it is.
• Evaluate one new quantization or batching configuration in a benchmark harness, document results, and share findings with the AI engineering team.
• Review production serving metrics for latency regression or GPU utilization drift and trace the cause to a model update, traffic pattern change, or configuration issue.
• Prototype or read at least one paper from the ML systems literature (MLSys, OSDI, SOSP, NeurIPS Systems Track) and assess its applicability to the current serving stack.
• Sync with the AI engineering team on upcoming model releases to plan serving configuration changes and capacity requirements before the rollout date.

Required skills

• LLM serving internals: deep understanding of how vLLM's PagedAttention manages KV cache memory as virtual paging, including how block eviction policies affect latency under memory pressure (Kwon et al., 2023 paper).
• Speculative decoding: implementing and tuning speculative decoding with draft models, measuring token acceptance rate in production, and understanding when the technique degrades throughput instead of improving it (Leviathan et al., 2023).
• Quantization techniques: applying GPTQ, AWQ, and QLoRA (Dettmers et al., 2023) quantization to reduce model memory footprint, measuring perplexity and task-specific accuracy before and after to validate quality.
• TensorRT-LLM: using NVIDIA TensorRT-LLM to compile model weights with FP8 or INT8 precision, configure inference plugins, and run optimized engines on H100 or A100 GPUs.
• GPU architecture: working knowledge of NVIDIA SM counts, memory bandwidth, NVLink topology, and PCIe bottlenecks that determine throughput ceilings for transformer inference.
• Benchmark engineering: writing reproducible benchmark harnesses in Python that simulate realistic prompt length distributions, concurrency levels, and streaming vs. batch request patterns.
• Serving metrics: tracking and interpreting TTFT (time to first token), TBT (time between tokens), ITL (inter-token latency), and KV cache hit rate as the core production quality signals for inference serving.
• Python and CUDA familiarity: reading and modifying PyTorch model code, writing Python profiling scripts, and understanding basic CUDA kernel execution to interpret profiler output without writing kernels from scratch.

What differentiates strong candidates

• Flash Attention: understanding the IO-aware attention algorithm (Dao et al.) and its role in reducing memory reads during the attention computation, relevant when evaluating attention-layer optimization opportunities.
• Continuous batching and chunked prefill: deep familiarity with how continuous batching (Orca, vLLM) differs from static batching and how chunked prefill reduces first-token latency for long-context requests.
• Multi-GPU tensor parallelism: configuring tensor parallel and pipeline parallel inference across multiple GPUs using Megatron-style sharding, relevant for large models that do not fit on a single GPU.
• MLPerf Inference benchmarking: running and interpreting MLPerf Inference benchmarks as a neutral comparison point for serving optimizations.
• Cybersecurity AI serving: applying inference optimization techniques to security-specific models (threat classification, log anomaly detection, NER for entity extraction) where latency constraints are often stricter than general-purpose LLM products.

Salary bands by experience

Source anchors: Levels.fyi 2025-2026 + Glassdoor public ranges. Total compensation varies by location, company, and negotiation.

Career ladder

1. ML Engineer / AI Systems Engineer (0-5 yrs): Model deployment, inference serving basics, Python profiling, familiarity with vLLM and TGI.
2. Inference Optimization Engineer (5-8 yrs): Quantization, speculative decoding, TensorRT-LLM, benchmark harness engineering, production latency tuning.
3. Senior Inference Optimization Engineer (8-12 yrs): Owns serving architecture for a model family; leads research-to-production of new optimization techniques.
4. Principal Inference Optimization Engineer (12+ yrs): Sets serving strategy across the organization; evaluates novel hardware (custom ASICs, next-gen GPUs) against production needs.

Transition paths into this role

Recommended courses

• AI Engineering Mastery: Module 9 (Cost and Latency): Module 9 covers per-token cost math, latency budgeting, and GPU utilization measurement, the quantitative foundation that Inference Optimization Engineers use to justify and measure optimization work.
• AI Engineering Mastery: Module 8 (Observability): Module 8 covers OpenTelemetry instrumentation and LLM serving metrics (TTFT, TBT, KV cache hit rate) that tell you where to focus an optimization effort.
• Efficient Deep Learning (MIT 6.5940 lecture notes, publicly available): MIT's course on efficient inference covers pruning, quantization, distillation, and hardware-aware model design. The lecture notes are free and give you the academic grounding behind the techniques you apply in production.

Companies that hire for this role

Anthropic · OpenAI · Together AI · Fireworks AI · Anyscale · NVIDIA · Replicate · Hugging Face · Modal · Cohere

DecipherU is not affiliated with, endorsed by, or sponsored by any company listed. Information is compiled from publicly available job postings for educational purposes.

Representative certifications

• NVIDIA-Certified Associate: AI Infrastructure and Operations (NVIDIA)
• AWS Solutions Architect Professional (Amazon Web Services)
• Google Cloud Professional ML Engineer (Google Cloud)
• Databricks Generative AI Engineer Associate (Databricks)
• Certified Kubernetes Administrator (CKA) (Cloud Native Computing Foundation)

Verify current pricing, exam format, and requirements directly with the certifying organization before making decisions.

Inference Optimization Engineer questions and answers