AI Infrastructuresagemaker

GPU Partitioning for SageMaker Real-Time Endpoints: Packing Small Models onto One GPU

P&C

Prompt & Circuit Editorial

November 2, 2026 · 12 Min Read

GPU Partitioning for SageMaker Real-Time Endpoints: Packing Small Models onto One GPU

Why it matters

Fine-tuned variants, embedding models, rerankers and classifiers are rarely big enough to fill an accelerator. AWS says customers serving such models previously had to give every model an entire accelerator, so an instance hosted no more models than it had GPUs [1]. Announced on 30 October 2026, SageMaker AI GPU partitioning changes that for real-time endpoints: it uses NVIDIA Multi-Instance GPU (MIG) to divide one physical GPU into multiple independent partitions [1][2]. For architects, the question is no longer only "which instance?" but "how do we carve it up, and what happens when the carve-up is wrong?"

How the feature works

GPU partitioning is switched on with two optional API fields [2]. The first, AcceleratorPartitionLayouts, goes on a production variant of an endpoint that hosts inference components and declares how every accelerator on every instance in that variant is partitioned [2]. The second, AcceleratorPartitionType, goes on an inference component and assigns it to a partition type [2].

Each partition gets a fixed share of the GPU's compute slices and a dedicated block of its memory [2]. A partition type states one partition's share; for example, mig-4g.20gb is 4 compute slices and 20 GB of GPU memory [2]. The documentation says the feature adds no new API operations, resource types or IAM actions, and that endpoints omitting both fields behave as they do without partitioning [2].

Invocation does not change. You invoke a model by inference component name, as you do without partitioning, and the partition does not appear in the request [2][4].

Layout rules that shape your design

The layout is declared once per variant, so it deserves care.

  • AcceleratorPartitionLayouts must contain exactly one entry. That entry describes how to divide one accelerator, and SageMaker AI applies it to every accelerator on every instance in the variant [3].
  • The requested partition types must fit on one accelerator, otherwise SageMaker AI returns a 400 error [3].
  • To request several partitions of the same type, you repeat the type in the list [3].
  • Some NVIDIA profile references are not supported: +me, +gfx, +me.all and -me [3].
  • The layout of a running endpoint cannot be changed. To use a different layout you create a new endpoint, and an UpdateEndpoint request to an endpoint configuration with a different layout returns a 400 error [2].

The documentation's own example shows the arithmetic. An ml.p4d.24xlarge has 8 NVIDIA A100 accelerators, each with 7 compute slices. Dividing each A100 into one mig-4g.20gb, one mig-2g.10gb and one mig-1g.5gb partition uses all 7 slices and provides 24 partitions in total, 8 of each type [3]. Because the layout applies uniformly, every GPU on the instance gets the same mix, so the mix has to match the models you expect to place.

Placing models on partitions

Once the endpoint is InService, every accelerator is partitioned and the partitions are empty; you then place models by creating inference components [2][4]. The rules for those components are [4]:

  • AcceleratorPartitionType is required, and omitting it returns a 400 error.
  • The requested type must be one that the endpoint layout declares.
  • NumberOfAcceleratorDevicesRequired must be 1, because it refers to one partition.
  • MinMemoryRequiredInMb and NumberOfCpuCoresRequired describe host memory and CPU, while the partition type sets the GPU memory.
  • Each copy of an inference component uses one partition, and you raise CopyCount to place more copies on free partitions of the same type.

An InService endpoint with no inference components is valid, but the instance incurs charges while the endpoint runs, whether or not inference components run on its partitions [3]. An idle, partitioned endpoint is therefore still a billing line item.

Supported instances and Regions

GPU partitioning is available on four instance families [2]:

FamilyAcceleratorPartition types (partitions per GPU)
P4dNVIDIA A100 (40 GB)mig-1g.5gb (7), mig-1g.10gb (4), mig-2g.10gb (3), mig-3g.20gb (2), mig-4g.20gb (1), mig-7g.40gb (1) [2]
P5NVIDIA H100 (80 GB)mig-1g.10gb (7), mig-1g.20gb (4), mig-2g.20gb (3), mig-3g.40gb (2), mig-4g.40gb (1), mig-7g.80gb (1) [2]
G7NVIDIA RTX PRO 4500 (32 GB)mig-1g.16gb (2), mig-2g.32gb (1) [2]
G7eNVIDIA RTX PRO 6000 (96 GB)mig-1g.24gb (4), mig-2g.48gb (2), mig-4g.96gb (1) [2]

The Regions are US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland) [1][2]. Instance availability varies by Region, so confirm that your instance type is available there and that you have endpoint usage quota for it [2]. Specifying a partition type that belongs to a different GPU is rejected with a 400 error that lists the valid values [2].

AWS also notes that right-sizing an accelerator to a workload opens up newer instance generations that would otherwise be more capacity than a single model needs [1]. Read that as a design option, not a price claim: the sources do not give costs for any instance.

Limits and failure modes

Isolation is partial. Each partition has its own compute slices and GPU memory, so one model cannot read another model's GPU memory or take its throughput [2]. However, models on the same instance still share host CPU, host memory, storage and networking [2]. A noisy neighbour can therefore still hurt you through the host even though the GPU is isolated.

Monitoring gaps. Partitioned endpoints and their inference components do not publish GPU, CPU or memory utilization metrics, so you rely on the invocation and latency metrics the endpoint publishes to Amazon CloudWatch [2]. Capacity planning by utilization graph is not available; plan from partition counts and latency.

A silent driver trap on P4d. The mig-1g.10gb type on P4d requires a newer NVIDIA driver than the default image provides. If you request it on the default image, SageMaker AI accepts the request, and the host then fails to partition the accelerators without reporting which partition type caused the failure [2]. The documented fix is to set InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on the production variant [2].

Slow feedback. SageMaker AI partitions the accelerators while it launches the instances, so partition problems surface during endpoint creation [3]. The layout appears in describe-endpoint output only once the status is InService; while the status is Creating or Failed it is shown as null, which is expected [3]. If the waiter exits with an error, the documentation says to check the endpoint status rather than recreate the endpoint [3].

Not the same as HyperPod. SageMaker HyperPod has its own MIG support, which lets a cluster administrator split a supported GPU into up to seven partitions [5]. It is a separate feature with its own instance list, and MIG there is not supported with flexible instance groups that use InstanceRequirements [5]. This article concerns real-time endpoints only.

When to use it

Partitioning fits when several small models each need a fraction of a GPU and you can predict the mix. It fits less well when one model needs the whole GPU: the mig-7g.40gb type on P4d, for instance, is a single partition per GPU [2], so you gain isolation but no consolidation.

Consider the smallest partition that holds each model. On G7, the largest GPU-memory partition is 32 GB with one per GPU, and the smaller type gives two 16 GB partitions per GPU [2]. A model that needs more than 16 GB therefore consumes a full G7 GPU in practice.

Architect's checklist

  • List each model's GPU memory need and choose partition types whose memory fits, using the per-family tables [2].
  • Confirm your instance family and Region are among the supported ones, and that you hold endpoint usage quota for the instance type [2].
  • Design one layout per endpoint, since it applies to every accelerator on every instance and cannot be changed in place [2][3].
  • Verify that requested partitions fit on one accelerator and avoid the unsupported profile suffixes [3].
  • On P4d, decide whether you need mig-1g.10gb, and if so set the newer InferenceAmiVersion [2].
  • Set AcceleratorPartitionType and NumberOfAcceleratorDevicesRequired of 1 on every inference component [4].
  • Plan monitoring around invocation and latency metrics, because utilization metrics are not published [2].
  • Remember that host CPU, memory, storage and networking are shared, and size the instance for the sum of the models [2].
  • Delete idle partitioned endpoints, because the instance is billed whether or not components run [3].

Sources

Numbers in square brackets refer to these sources, checked on 2026-11-02.

  1. Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints - AWS
  2. Using GPU partitions on SageMaker AI real-time endpoints - Amazon SageMaker AI
  3. Create an endpoint with GPU partitions - Amazon SageMaker AI
  4. Deploy models to GPU partitions with inference components - Amazon SageMaker AI
  5. Using GPU partitions in Amazon SageMaker HyperPod - Amazon SageMaker AI
Tags:sagemakergpuinferencemig
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.