Deploy decision models with llama.cpp on Amazon SageMaker AI

來源原標題

Deploy decision models with llama.cpp on Amazon SageMaker AI

作者

Jyothirmai Kottu

原始標籤

Advanced (300) · Amazon SageMaker AI · Technical How-to

原始連結

https://aws.amazon.com/blogs/machine-learning/deploy-decision-models-with-llama-cpp-on-amazon-sagemaker-ai/

摘要原文

Applications that route requests, select agent actions, or assign policy outcomes often need scores over a fixed set of choices instead of free-form text. Deploying these decision models with llama.cpp on Amazon SageMaker AI gives you a managed path for serving quantized models on CPU or GPU infrastructure. This post continues a series on specialized AWS Deep Learning Containers (DLCs) for generative AI inference. Previous posts cover WhisperX , vLLM-Omni – Part 1 , and vLLM-Omni – Part 2 . In this post, you first use the AWS DLC for llama.cpp to serve OpenAI-compatible chat completions through SageMaker AI InvokeEndpoint . You then extend the same runtime with SemIf , formerly OpenJev, which returns probabilities over declared options without decoding an answer token. The accompanying code sample includes CPU and GPU deployment options. Figure 1 shows two deployment paths from the same packaged model. The text-generation path uses the AWS DLC for llama.cpp and returns generated responses through the OpenAI-compatible API. The structured-decision path builds a child image that replaces the default server with a SemIf adapter and returns probabilities for declared options. Figure 1: Run the AWS DLC for llama.cpp or the SemIf child image on CPU or GPU, depending on the API and latency requirements of your workload For the standard path, package the model as model.tar.gz in Amazon Simple Storage Service (Amazon S3) and reference the published DLC image directly. The container finds the first .gguf file under /opt/ml/model , maps POST /invocations to llama-server at /v1/chat/completions , and passes SM_LLAMA_CPP_* environment variables to llama.cpp arguments. The llama-server HTTP API doesn’t expose the final-position logits and state operations that SemIf uses. The structured-decision path therefore uses AWS CodeBuild to build a child image and store it in Amazon Elastic Container Registry (Amazon ECR) . The child image keeps the DLC’s compiled CPU or CUDA libraries while replacing its entry point with a SageMaker-compatible adapter. The AWS DLC for llama.cpp accepts the OpenAI Chat Completions schema. Send a messages array with role and content , plus generation controls such as max_tokens and stream : The response uses the OpenAI-compatible choices array. Read generated text from choices[0].message.content . SemIf uses a constrained decision schema instead of chat messages. The request contains a caller-defined ID, unstructured state, a question, and an ordered list of allowed options: The response returns option_ids , probabilities , and raw option_logits in the same order, along with prompt, timing, and pinned model metadata. With stable option IDs, your application can apply thresholds, fallbacks, and policy checks without parsing generated prose. SemIf belongs to a broader class of decision models that score application-defined outcomes from unstructured state. Other implementations include Jev , TypeSafe’s decision model and /v1/systemone API; Kev , a model server designed for TypeSafe API compatibility; and Strands Decider 2B , a purpose-built model for agent workflows with calibrated outputs. Use this pattern when your application defines the allowed choices at request time and needs a model to interpret unstructured context. Examples include routing a case to a queue, extracting one value from a known set, assigning an incident priority, checking a policy outcome, or selecting an agent action. Keep authorization and deterministic business rules outside the model, and validate model probabilities before treating them as operational confidence. Use the standard SageMaker AI text-generation path when you need OpenAI-compatible chat completions through InvokeEndpoint . The SageMaker AI image maps POST /invocations to the llama-server /v1/chat/completions endpoint. Build a child image only when your application needs behavior that this interface doesn’t provide, such as SemIf’s direct logits and state access. Removing answer-token generation doesn’t remove model inference. The model still processes the complete state, question, and option descriptions before SemIf reads the final logits. Keep prompts focused, and evaluate the selected model with labeled decisions from your application before production use. The following benchmark applies to the SemIf extension, not the text-generation endpoint. Model size, quantization, prompt length, runtime, and CPU thread count affect each decision. To quantify these trade-offs, we sent the same 231 public JevBench tasks serially to one CPU endpoint and one GPU endpoint. Compared with the CPU configuration, the GPU endpoint was 4.9 times faster at p50 and 32.2 times faster at p95. The sample uses the SageMaker AI instance pools feature to list compatible GPU instance types in priority order, so SageMaker can select an alternative when the first preference isn’t available. The pool ordered g6e , g6 , g5 , then g4dn . SageMaker selected ml.g6e.xlarge , and startup logs confirmed that llama.cpp offloaded all 34 model layers to an NVIDIA L40S. The CPU and GPU images used different llama.cpp builds and produced different predictions on six hard tasks. Treat the one-answer accuracy difference as runtime variation, not evidence that GPU offload improves model quality. Smaller models reduced median latency, but the 2B and 0.8B candidates achieved 66.7 percent and 58.4 percent accuracy. The sample therefore keeps the 4B Q4_K_M model as its quality-first default. These measurements include network latency and aren’t service guarantees. Benchmark the complete model and infrastructure configuration against your own decisions. Use CPU for background, offline, or low-throughput workloads that aren’t latency-sensitive and can tolerate seconds of response time. Use GPU for interactive decisions, long prompts, or workloads with tight tail-latency targets. Keep state and option descriptions concise, group decisions that share the same state into one request, and scale endpoint instances for concurrency. Shared-state scoring reuses the common prompt prefix, while the adapter returns retryable HTTP 503 responses instead of allowing an unbounded queue around its stateful llama.cpp context. The text-generation example uses the CPU DLC and a fixed CPU instance. The SemIf extension keeps CPU as the default for workloads that aren’t latency-sensitive and offers an optional CUDA image with ordered instance pools. Review SageMaker AI pricing before creating an endpoint. Use the following procedure to deploy the standard text-generation endpoint, then build the optional SemIf extension for structured decisions on CPU or GPU. Wait for the returned build ID to reach SUCCEEDED . The build records the source revisions and checksums in model-manifest.json and stops if the GGUF checksum differs from the validated value. Resolve a digest-pinned server-sagemaker-cpu-v1 image from the AWS DLC for llama.cpp documentation , then create the SageMaker AI model, endpoint configuration, and endpoint: Select a server-sagemaker-cpu-v1 or server-sagemaker-cuda-v1 image rather than the corresponding EC2 image. The SageMaker AI images implement the /ping and /invocations container contract and map invocation payloads to chat completions. This walkthrough supplies the GGUF model through the SageMaker AI model artifact in Amazon S3. As an alternative, you can configure SM_LLAMA_CPP_HF_REPO and SM_LLAMA_CPP_HF_FILE to download a GGUF file from Hugging Face when the endpoint has network egress. You don’t need a Dockerfile or an ECR repository for this path. Wait for the endpoint to enter InService , then send an OpenAI-compatible chat request: For GPU, build the CUDA child image instead: After the build succeeds, construct the immutable image URI: The deployment script rejects mutable image tags. The CPU configuration uses a fixed instance type: Wait for the returned endpoint name to enter InService . This path sets n_gpu_layers=0 .