Here’s a structured summary of Disaggregated Prefill and Decode (DPD) on Amazon SageMaker HyperPod, based on your detailed explanation:

---

🚀 Disaggregated Prefill and Decode (DPD) Overview

✅ What is DPD?

• DPD separates the prefill and decode phases of large language model inference into two distinct GPU pools.
• Prefill: Compute-bound phase, where the model processes the input context.
• Decode: Memory-bandwidth-bound phase, where token generation occurs.
• This separation avoids resource contention between compute and memory bandwidth on a single GPU.

📌 Key Benefits

• Improved Latency: Isolates decode from prefill interference, keeping per-token latency (TPOT) stable under load.
• Scalable Architecture: Enables independent scaling of prefill and decode resources.
• Better Throughput: Dedicated decoder runs at full CUDA graph efficiency without interruption.
• Optimized for Long Contexts: Especially effective for workloads with long input sequences or retrieval-augmented generation (RAG).

---

🧠 How DPD Works

🔁 Routing Logic

• Short prompts (<4,096 tokens) go directly to the decoder.
• Long prompts (>4,096 tokens) are routed through the prefiller first, then to the decoder.

🔄 Communication Between Prefill and Decode

• KV Cache Transfer: The key-value (KV) cache from the prefill phase is transferred to the decode phase over EFA RDMA using GPU-Direct.
• This avoids latency penalties from CPU-GPU data transfer, ensuring efficient communication between GPU pools.

---

📦 Deployment and Configuration

🧰 Required Components

• InferenceEndpointConfig: Define prefiller and decoder pods, router, and routing strategy.
• Routing Strategy Options
• kvaware: Optimizes for repeated prefixes (maximizes L1 cache hits).
• session: Keeps user context on a single prefiller for multi-turn conversations.

📌 Scaling Guidance

Workload Type	Prefill-to-Decoder Ratio
Chat, Code Generation	1:1
Summarization, RAG	2:1 to 3:1

---

📈 Performance Benchmarks

Metric	H100 (ml.p5.48xlarge)	H200 (ml.p5en.48xlarge)
TPOT	22%–66% improvement	28%–48% improvement
Throughput	Up to +35%	Up to +64%
E2E Latency (P50)	14–32% improvement	29–41% improvement

---

📦 Monitoring and Observability

• Use SageMaker HyperPod Observability for real-time metrics.
• Monitor:
• DPD-specific metrics in the Inference Dashboard
• CPU/GPU usage via Tasks Dashboard
• Cluster-level metrics via Cluster Overview Dashboard

---

🧹 Clean Up

To avoid ongoing costs, delete resources after testing:

kubectl delete inferenceendpointconfig ${DEPLOYMENT_NAME} -n ${NAMESPACE}
aws s3 rm s3://${MODEL_BUCKET}/${MODEL_PREFIX}/ --recursive --region ${AWS_REGION}

---

✅ Summary

DPD on SageMaker HyperPod is a powerful architecture for large-scale LLM inference, offering: – Latency optimization by isolating decode from prefill. – Scalable resource allocation with independent scaling of compute and memory bandwidth. – Improved throughput via dedicated decoders and efficient KV cache transfer.

---

📚 Next Steps

1. Deploy a DPD endpoint using the example manifest in this post.
2. Explore Amazon SageMaker HyperPod documentation.
3. Dive into HyperPod Inference Operator model deployment guide.

---

📚 Authors

• Xuan Lu – Software Development Engineer, AWS
• Nicolas Jourdan – Specialist Solutions Architect, AWS
• Vinay Arora – Specialist Solution Architect, Generative AI, AWS
• Piyush Daftary – Senior Software Engineer, AWS
• Kirupa Gunaseelan – Software Development Engineer, AWS
• Richa Shalom Gadagotti – Software Development Engineer, AWS
• Swapnil Palod – Senior Manager, AWS

---

Would you like a step-by-step guide to deploy DPD on HyperPod, or help with custom routing strategies?

---

Original Source

This article is based on publicly available reporting. For the complete original story, visit the publisher’s article.