GPU Partitioning for SageMaker Real-Time Endpoints: Packing Small Models onto One GPU
Prompt & Circuit Editorial
November 2, 2026 · 12 Min Read

Why it matters
Fine-tuned variants, embedding models, rerankers and classifiers are rarely big enough to fill an accelerator. AWS says customers serving such models previously had to give every model an entire accelerator, so an instance hosted no more models than it had GPUs [1]. Announced on 30 October 2026, SageMaker AI GPU partitioning changes that for real-time endpoints: it uses NVIDIA Multi-Instance GPU (MIG) to divide one physical GPU into multiple independent partitions [1][2]. For architects, the question is no longer only "which instance?" but "how do we carve it up, and what happens when the carve-up is wrong?"
How the feature works
GPU partitioning is switched on with two optional API fields [2]. The first, AcceleratorPartitionLayouts, goes on a production variant of an endpoint that hosts inference components and declares how every accelerator on every instance in that variant is partitioned [2]. The second, AcceleratorPartitionType, goes on an inference component and assigns it to a partition type [2].
Each partition gets a fixed share of the GPU's compute slices and a dedicated block of its memory [2]. A partition type states one partition's share; for example, mig-4g.20gb is 4 compute slices and 20 GB of GPU memory [2]. The documentation says the feature adds no new API operations, resource types or IAM actions, and that endpoints omitting both fields behave as they do without partitioning [2].
Invocation does not change. You invoke a model by inference component name, as you do without partitioning, and the partition does not appear in the request [2][4].
Layout rules that shape your design
The layout is declared once per variant, so it deserves care.
AcceleratorPartitionLayoutsmust contain exactly one entry. That entry describes how to divide one accelerator, and SageMaker AI applies it to every accelerator on every instance in the variant [3].- The requested partition types must fit on one accelerator, otherwise SageMaker AI returns a 400 error [3].
- To request several partitions of the same type, you repeat the type in the list [3].
- Some NVIDIA profile references are not supported:
+me,+gfx,+me.alland-me[3]. - The layout of a running endpoint cannot be changed. To use a different layout you create a new endpoint, and an
UpdateEndpointrequest to an endpoint configuration with a different layout returns a 400 error [2].
The documentation's own example shows the arithmetic. An ml.p4d.24xlarge has 8 NVIDIA A100 accelerators, each with 7 compute slices. Dividing each A100 into one mig-4g.20gb, one mig-2g.10gb and one mig-1g.5gb partition uses all 7 slices and provides 24 partitions in total, 8 of each type [3]. Because the layout applies uniformly, every GPU on the instance gets the same mix, so the mix has to match the models you expect to place.
Placing models on partitions
Once the endpoint is InService, every accelerator is partitioned and the partitions are empty; you then place models by creating inference components [2][4]. The rules for those components are [4]:
AcceleratorPartitionTypeis required, and omitting it returns a 400 error.- The requested type must be one that the endpoint layout declares.
NumberOfAcceleratorDevicesRequiredmust be 1, because it refers to one partition.MinMemoryRequiredInMbandNumberOfCpuCoresRequireddescribe host memory and CPU, while the partition type sets the GPU memory.- Each copy of an inference component uses one partition, and you raise
CopyCountto place more copies on free partitions of the same type.
An InService endpoint with no inference components is valid, but the instance incurs charges while the endpoint runs, whether or not inference components run on its partitions [3]. An idle, partitioned endpoint is therefore still a billing line item.
Supported instances and Regions
GPU partitioning is available on four instance families [2]:
| Family | Accelerator | Partition types (partitions per GPU) |
|---|---|---|
| P4d | NVIDIA A100 (40 GB) | mig-1g.5gb (7), mig-1g.10gb (4), mig-2g.10gb (3), mig-3g.20gb (2), mig-4g.20gb (1), mig-7g.40gb (1) [2] |
| P5 | NVIDIA H100 (80 GB) | mig-1g.10gb (7), mig-1g.20gb (4), mig-2g.20gb (3), mig-3g.40gb (2), mig-4g.40gb (1), mig-7g.80gb (1) [2] |
| G7 | NVIDIA RTX PRO 4500 (32 GB) | mig-1g.16gb (2), mig-2g.32gb (1) [2] |
| G7e | NVIDIA RTX PRO 6000 (96 GB) | mig-1g.24gb (4), mig-2g.48gb (2), mig-4g.96gb (1) [2] |
The Regions are US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland) [1][2]. Instance availability varies by Region, so confirm that your instance type is available there and that you have endpoint usage quota for it [2]. Specifying a partition type that belongs to a different GPU is rejected with a 400 error that lists the valid values [2].
AWS also notes that right-sizing an accelerator to a workload opens up newer instance generations that would otherwise be more capacity than a single model needs [1]. Read that as a design option, not a price claim: the sources do not give costs for any instance.
Limits and failure modes
Isolation is partial. Each partition has its own compute slices and GPU memory, so one model cannot read another model's GPU memory or take its throughput [2]. However, models on the same instance still share host CPU, host memory, storage and networking [2]. A noisy neighbour can therefore still hurt you through the host even though the GPU is isolated.
Monitoring gaps. Partitioned endpoints and their inference components do not publish GPU, CPU or memory utilization metrics, so you rely on the invocation and latency metrics the endpoint publishes to Amazon CloudWatch [2]. Capacity planning by utilization graph is not available; plan from partition counts and latency.
A silent driver trap on P4d. The mig-1g.10gb type on P4d requires a newer NVIDIA driver than the default image provides. If you request it on the default image, SageMaker AI accepts the request, and the host then fails to partition the accelerators without reporting which partition type caused the failure [2]. The documented fix is to set InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on the production variant [2].
Slow feedback. SageMaker AI partitions the accelerators while it launches the instances, so partition problems surface during endpoint creation [3]. The layout appears in describe-endpoint output only once the status is InService; while the status is Creating or Failed it is shown as null, which is expected [3]. If the waiter exits with an error, the documentation says to check the endpoint status rather than recreate the endpoint [3].
Not the same as HyperPod. SageMaker HyperPod has its own MIG support, which lets a cluster administrator split a supported GPU into up to seven partitions [5]. It is a separate feature with its own instance list, and MIG there is not supported with flexible instance groups that use InstanceRequirements [5]. This article concerns real-time endpoints only.
When to use it
Partitioning fits when several small models each need a fraction of a GPU and you can predict the mix. It fits less well when one model needs the whole GPU: the mig-7g.40gb type on P4d, for instance, is a single partition per GPU [2], so you gain isolation but no consolidation.
Consider the smallest partition that holds each model. On G7, the largest GPU-memory partition is 32 GB with one per GPU, and the smaller type gives two 16 GB partitions per GPU [2]. A model that needs more than 16 GB therefore consumes a full G7 GPU in practice.
Architect's checklist
- List each model's GPU memory need and choose partition types whose memory fits, using the per-family tables [2].
- Confirm your instance family and Region are among the supported ones, and that you hold endpoint usage quota for the instance type [2].
- Design one layout per endpoint, since it applies to every accelerator on every instance and cannot be changed in place [2][3].
- Verify that requested partitions fit on one accelerator and avoid the unsupported profile suffixes [3].
- On P4d, decide whether you need
mig-1g.10gb, and if so set the newerInferenceAmiVersion[2]. - Set
AcceleratorPartitionTypeandNumberOfAcceleratorDevicesRequiredof 1 on every inference component [4]. - Plan monitoring around invocation and latency metrics, because utilization metrics are not published [2].
- Remember that host CPU, memory, storage and networking are shared, and size the instance for the sum of the models [2].
- Delete idle partitioned endpoints, because the instance is billed whether or not components run [3].
Sources
Numbers in square brackets refer to these sources, checked on 2026-11-02.
- Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints - AWS
- Using GPU partitions on SageMaker AI real-time endpoints - Amazon SageMaker AI
- Create an endpoint with GPU partitions - Amazon SageMaker AI
- Deploy models to GPU partitions with inference components - Amazon SageMaker AI
- Using GPU partitions in Amazon SageMaker HyperPod - Amazon SageMaker AI
Responses (0)
Sign in to join the conversation and share your thoughts.

Prompt & Circuit Editorial
Contributing Editor
Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.
Continue Reading

Sharing One GPU Across ECS Tasks with NVIDIA MPS: Limits and Trade-offs
Amazon ECS can now pack several tasks onto a single NVIDIA GPU using MPS. Here is how placement works, where the isolation ends, and what to check before you adopt it.

DRA on EKS Auto Mode and Karpenter: Choosing Between Device Plugins and Dynamic Resource Allocation for GPUs
Amazon EKS Auto Mode and Karpenter now support Dynamic Resource Allocation for NVIDIA GPUs and EFA devices. Here is what changes for GPU scheduling, and where the documented limits still push you back to device plugins.

CPU, NPU, GPU, TPU, DPU and QPU: Key Differences and When to Use Each
A practical guide to the six processor types architects hear about most: what each is built for, how they differ, and how to match them to a workload.