Build real-time voice applications with vLLM-Omni on SageMaker AI – Part 1

來源原標題

Build real-time voice applications with vLLM-Omni on SageMaker AI – Part 1

作者

Yadan Wei

原始標籤

Advanced (300) · Amazon SageMaker AI · Technical How-to

原始連結

https://aws.amazon.com/blogs/machine-learning/build-real-time-voice-applications-with-vllm-omni-on-sagemaker-ai-part-1/

摘要原文

Voice agents, interactive learning applications, accessibility tools, and customer service assistants need to respond without long silent pauses. In this tutorial, you deploy a text-to-speech (TTS) model on Amazon SageMaker AI that can start playing speech before it finishes generating the full response. You use the AWS vLLM-Omni Deep Learning Container (DLC) to deploy Qwen3-TTS , stream text in and audio out over one persistent bidirectional connection, and try the workflow through a Gradio application. AWS Deep Learning Containers provide Docker images with deep learning frameworks and dependencies for training and inference on AWS. AWS provides deployment guidance for broadly adopted serving frameworks such as vLLM and SGLang . This post is Part 1 of a series about specialized DLCs, including vLLM-Omni , WhisperX , and llama.cpp . It focuses on streamed speech for real-time voice applications. Part 2 applies the vLLM-Omni DLC to image and video generation. The series pairs focused use cases with deployment examples and reproducible benchmarks where they add useful evidence. The vLLM-Omni project extends vLLM beyond text generation to serve models that process or generate text, audio, images, and video. The AWS vLLM-Omni DLC packages tracked vLLM-Omni releases in AWS images and adds routing middleware for SageMaker AI. You use SageMaker bidirectional streaming to send text and receive audio chunks over the persistent connection. The earlier post, Build real-time voice applications with Amazon SageMaker AI and vLLM , demonstrates the input side of a voice pipeline. It streams microphone audio to the Voxtral-Mini-4B Realtime speech-to-text (STT) model and returns transcription events. This post adds the output side by sending text to Qwen3-TTS and streaming generated speech back through the vLLM-Omni DLC. You will clone the code sample , deploy a Qwen3-TTS model, and try streaming speech through a Gradio application. Specialized inference runtimes support model architectures and media pipelines that differ from general text generation. AWS DLCs package these runtimes with their framework dependencies and deployment configuration, giving you a consistent image-based path to AWS compute and managed inference services. vLLM-Omni extends vLLM from text-focused autoregressive generation to models that process and generate multiple modalities. Its heterogeneous pipeline abstraction coordinates multi-stage model workflows, including autoregressive and diffusion stages. It also provides streaming outputs and OpenAI-compatible APIs. The runtime is not limited to TTS. Its supported models include unified omni models, automatic speech recognition (ASR), TTS, audio generation, image generation, and video generation. This post uses Qwen3-TTS as a focused example because streamed speech provides a direct way to demonstrate bidirectional audio output. The previous vLLM post covers the input path: microphone audio enters a Voxtral model and transcription events return to the application. A voice application can pass that transcription to its conversation logic, then use this example to turn the response text into streamed speech. The orchestration between the two endpoints remains outside this walkthrough. The complete sample lives in 03-features/bidirectional-streaming-vLLM-Omni . Clone the repository to use the deployment script, shared streaming transport, and Gradio client together. SageMaker bidirectional streaming exposes a full-duplex WebSocket transported over HTTP/2. The client connects to the SageMaker Runtime endpoint on port 8443 . The SageMaker inference sidecar forwards the connection to the vLLM-Omni native WebSocket route inside the container. Figure 1 shows the request and response path. The client sends session configuration and text events. Qwen3-TTS returns audio lifecycle and chunk events over the same connection. Figure 1: A Gradio application uses an HTTP/2 WebSocket to stream text events through Amazon SageMaker AI, the inference sidecar, and the vLLM-Omni DLC to Qwen3-TTS. Audio events and 24 kHz PCM chunks return over the same connection The vLLM-Omni v1.5 DLC adds the bidirectional streaming capability label and routing middleware required by SageMaker AI. The sample uses v1/audio/speech/stream , one of the native WebSocket routes exposed by vLLM-Omni. The endpoint configuration also uses SageMaker instance pools . An instance pool defines a priority-ordered list of compatible instance types for one production variant. SageMaker first tries ml.g6.xlarge , then falls back to ml.g6e.xlarge , ml.g5.xlarge , or ml.g4dn.xlarge when capacity is unavailable. It still provisions one instance, not one instance per pool. SageMaker validates quota for every configured pool entry when it creates the endpoint, so you need available quota for each included type. Your hourly cost can also change if SageMaker selects a different instance. Use --instance-types to restrict the pool to the types that meet your quota, price, and performance requirements. The walkthrough uses the US East (N. Virginia) Region. The sample builds the DLC image URI and runtime endpoint from AWS_REGION . The inference AMI setting is required for the SageMaker sidecar that handles the HTTP/2 WebSocket. The model invocation path must omit the leading slash because SageMaker adds it before forwarding the request. Wait until the endpoint reaches InService . The first deployment takes time because the instance downloads the DLC image and model artifacts. Open http://127.0.0.1:6006 in your browser. The --share option creates a Gradio public link, so leave it disabled unless you need that behavior. The Gradio audio component plays each 24 kHz PCM chunk as the client receives it. The status field reports the chunk count and total audio bytes. Confirm that the cleanup command reports deletion of the endpoint, endpoint configuration, and model. If the process stops before cleanup completes, delete the resources through the SageMaker AI console or API. A running GPU endpoint continues to incur charges. By deploying a text-to-speech model, you used the AWS vLLM-Omni DLC on SageMaker AI to stream audio while the model was still generating it. This example shows how specialized AWS DLCs support multimodal models whose execution and streaming patterns differ from general text generation. Paired with the previous vLLM speech-to-text example , the samples show complementary paths: stream microphone audio into transcription, then stream the application’s text response back as speech. Part 2 which will show how to use the same DLC family for image and video generation. Try the code sample , then explore the vLLM-Omni DLC documentation and SageMaker AI bidirectional streaming developer guide to apply the pattern to other supported multimodal models. The authors thank Zhuofu Bai, Ayush Sharma, Christian Kamwangala, and Jon Chua for their contributions to the solution and publication process.