NVIDIA Run:ai Distributed Inferences API
Distributed inference enables running inference workloads across multiple pods, typically to scale model serving beyond a single container or node. This approach is useful when a single instance cannot meet resource requirements.NVIDIA Run:ai supports this model using Leader Worker Set (LWS). Each pod plays a specific role, either as a leader or worker, and together they form a coordinated service. NVIDIA Run:ai manages the orchestration and configuration of these pods to ensure efficient and scalable inference execution
NVIDIA Run:ai Distributed Inferences API is one of 62 APIs that NVIDIA Run:ai publishes on the APIs.io network, described by a machine-readable OpenAPI specification.
Tagged areas include Distributed Inferences. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, and authentication docs.
This API exposes 4 operations across 2 paths, and defines 116 schemas. It is described by OpenAPI 3.0.3, at version latest.
Requests are made against a single base URL, https://app.run.ai.
4 operations
2 paths
116 schemas
1 DELETE1 GET1 PATCH1 POST
Metadata
The identity and technical contract details declared by the specification.
Authentication & Security 1
NVIDIA Run:ai Distributed Inferences API declares
1 security scheme
for authenticating requests.
It accepts HTTP bearer tokens (JWT) (bearerAuth).
By default, every request must be authenticated.
bearerAuth— Bearer authentication
Paths & Operations 4
Across 2 paths, the API surfaces 4 operations — 1 DELETE, 1 GET, 1 PATCH, 1 POST. Each is listed below with its method, path, parameters, and response codes.
Distributed Inferences 4
Distributed inference enables running inference workloads across multiple pods, typically to scale model serving beyond a single container or node. This approach is useful when a…
POST
/api/v1/workloads/distributed-inferences
Create a distributed inference. [Experimental]
DELETE
/api/v1/workloads/distributed-inferences/{workloadId}
Delete a distributed inference.
GET
/api/v1/workloads/distributed-inferences/{workloadId}
Get a distributed inference data.
PATCH
/api/v1/workloads/distributed-inferences/{workloadId}
Update distributed inference spec.
Schemas 116
The contract defines 116 schemas that model the data the API accepts and returns. The most detailed are DistributedInferenceLeaderWorkerSpec1 (18 properties), WorkloadMeta1 (11 properties), EnvironmentVariable (8 properties), Toleration (7 properties). Each schema is shown below with its type and property counts.
Category
string
Specify the workload category assigned to the workload. Categories are used to classify and monitor different types of workloads within the NVIDIA Run:ai platf…
PvcFieldsNonUpdatable
object
6 properties
Probe
object
6 properties
WorkloadId2
string
A unique ID of the workload.
DistributedInference
NodeAffinityRequired
object
If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements…
1 property
UidGidSource
string
Indicate the way to determine the user and group ids of the container. The options are a. fromTheImage - user and group ids are determined by the docker image…
SecretInstance2
object
Labels
array
Set of labels to populate into the container running the workload.
Error
object
3 properties
2 required
ExcludeField
object
1 property
CpuMemoryLimit
string
Limitations on the CPU memory to allocate for this workload (1G, 20M, .etc). The system guarantees that this workload will not be able to consume more than thi…
PvcClaimSize
string
Requested size for the PVC. Mandatory when existingPvc is false. Recommended sizes: TB/GB/MB/TIB/GIB/MIB
PvcAddedAttrValues
array
an optional array of key-values pairs that are written as annotations on the created PVC. the allowed attributes are determined according to the storage class…
DistributedInferenceSpec
object
The specifications of the distributed inference to be created.
1 property
Toleration
object
Toleration details.
7 properties
ClusterId
string
The id of the cluster.
SecretFieldsNonUpdatable
object
1 property
Pvc
Capabilities
array
Add POSIX capabilities to running containers. Defaults to the default set of capabilities granted by the container runtime.
GpuMemoryRequest
string
Required if and only if gpuRequestType is memory. States the GPU memory to allocate for the created workload, per GPU device. Note that the workload will not b…
DistributedInferenceCommonSpec
object
SecretFieldsUpdatable
object
2 properties
EnvironmentVariable
object
Details of an environment variable which is populated into the container.
8 properties
LargeShmRequest
boolean
A large /dev/shm device to mount into a container running the created workload. An shm is a shared file system mounted on RAM.
DistributedInferenceWorkersField
object
1 property
RunAsGid
integer
The group id to run the entrypoint of the container which executes the workspace. Default to the value specified in the environment asset runAsGid field (optio…
EnvironmentVariableSecret
object
Details of the secret and key use to populate the environment variable
2 properties
Annotations
array
Set of annotations to populate into the container running the workload.
WorkloadDesiredPhase
string
The desired phase of the workload.
ProbeHandler
object
The action taken to determine the health of the container. (mandatory)
1 property
DistributedInferenceLeaderWorkerSpec1
object
18 properties
TolerationEffect
string
The taint effect to match. (mandatory)
MatchExpressionOperator
string
Represents a key's relationship to a set of values (mandatory).
ImagePullPolicy
string
Image pull policy. Defaults to Always if :latest tag is specified, otherwise it is IfNotPresent.
WorkloadName
string
The name of the workload.
ClaimInfo
object
Claim information for the newly created PVC. The information should not be provided when attempting to use existing PVC.
5 properties
UpdateSpec
object
The specifications of the inference to be updated.
1 property
DistributedInferenceServingPort
object
Defines the configuration for the inference serving endpoint. This determines how applications or services can send inference requests to the workload.
DepartmentId2
string
The id of the department.
DistributedInferenceRestartPolicy
string
Determines the behavior when a pod fails. - RecreateGroupOnPodRestart: Restarts all pods in the group if any pod fails. - None: No automatic restart behavior i…
StorageInstanceName
object
1 property
EnvironmentVariables
array
Set of environment variables to populate into the container running the workload.
CpuCoreRequest
number
CPU units to allocate for the created workload (0.5, 1, .etc). The workload will receive at least this amount of CPU. Note that the workload will not be schedu…
Preemptibility
string
Specifies whether the workload can be preempted by higher-priority workloads. Valid values are preemptible and non-preemptible. If explicitly set, this value t…
DistributedInferenceWorkerSpecFields
object
1 property
SupplementalGroups
string
Comma separated list of groups that the user running the container belongs to, in addition to the group indicated by runAsGid. Use only when the source uid/gid…
DistributedInferenceSpecSpec
ConfigMapItems
array
Set of config map volumes to use in the workload
DistributedInferenceServingPortAccess
object
5 properties
ImagePullSecrets
array
A list of references to Kubernetes secrets in the same namespace used for pulling container images.
Image
string
Docker image name. For more information, see [Images](https://kubernetes.io/docs/concepts/containers/images). The image name is mandatory for creating a worklo…
EnvironmentVariableUserCredential
object
Defines a reference to a user-created credential and a specific key within that credential whose value will populate the environment variable. User credentials…
2 properties
Annotation
object
Annotation details to be populated into the container.
3 properties
DistributedInferenceStartupPolicy
string
Determines when the worker pods should start during workload initialization. - LeaderCreated: Workers start after the leader pod is created. - LeaderReady: Wor…
CreateHomeDir
boolean
When set to true, creates a home directory for the container.
GpuMemoryLimit
string
Limitation on the memory consumed by the workload, per GPU device. The system guarantees The gpuMemoryLimit must be no less than gpuMemoryRequest.
EnvironmentVariableConfigMap
object
Details of the configMap and key use to populate the environment variable
2 properties
CpuCoreLimit
number
Limitations on the number of CPUs consumed by the workload (0.5, 1, .etc). The system guarantees that this workload will not be able to consume more than this…
DistributedInferenceServingPortAccessAuthorizationTypeEnum
string
Specifies who can send inference requests to the serving endpoint: Possible values: - public: No authorization is required. (Default) - authenticatedUsers: Any…
Capability
string
DistributedInferenceReplicasField
object
1 property
WorkloadCreationMeta
object
4 properties
3 required
GpuDevicesRequest
integer
Requested number of GPU devices. Currently if more than one device is requested, it is not possible to provide values for gpuMemory or gpuPortion.
NodePools
array
A prioritized list of node pools for the scheduler to run the workload on. The scheduler will always try to use the first node pool before moving to the next o…
PvcAddedAttrValue
object
2 properties
1 required
DistributedInferenceLeaderSpecFields
object
1 property
SecretItems1
array
Set of secret volumes to use in the workload
PodAffinity
object
Pod affinity scheduling rules (e.g. co-locate this workload in the same node, zone, etc. as some other workloads).
2 properties
EnvironmentVariablePodFieldReference
object
Details of the field-reference and key use to populate the environment variable
1 property
MatchExpression
object
A selector that contains values, a key, and an operator that relates the key and values.
3 properties
2 required
GpuRequestType
string
Sets the unit type for GPU resources requests. Stated in terms of portion or memory. Sets the unit type for other GPU request fields. If gpuDevicesRequest 1, o…
EmptyDir
object
3 properties
PvcItems
array
Set of pvc persistent volume claims to use in the workload.
DistributedInferenceServingPortProtocol
string
The protocol used to access the port.
PodAffinityType
string
The affinity type, required or preferred. (mandatory)
ReadOnlyRootFileSystem
boolean
If true, mounts the container's root filesystem as read-only.
ConfigMapInstance
object
Command
string
A command to the server as the entry point of the container running the workload.
ProbeHandlerScheme
string
Scheme to use for connecting to the host, defaults to HTTP.
RunAsNonRoot
boolean
Force the container to run as a non-root user.
SubmitWithTemplateId
object
1 property
Args
string
Arguments to the command that the container running the workload executes.
PriorityClass
string
Specifies the priority class for the workload, which determines its scheduling behavior. Valid values are: very-low, low, medium-low, medium, medium-high, high…
Phase
string
NodeSelectorTerm
object
A null or empty node selector term matches no objects. The requirements of them are ANDed.
1 property
WorkingDir
string
Container's working directory. If not specified, the container runtime default will be used. This may be configured in the container image.
PvcVolumeMode
string
Default volume mode for the PVC. Choose between Filesystem (default) or Block.
DistributedInferenceCreationRequest
Label
object
Label details to be populated into the container.
3 properties
DistributedInferenceStartupPolicyField
object
1 property
SeccompProfileType
string
Indicates which kind of seccomp profile will be applied to the container. The options are a. RuntimeDefault - the container runtime default profile should be u…
DefaultMode
string
File permission mode in octal string format. This value must be a 4-digit octal number, representing the default file mode when mounting a Secret or ConfigMap…
Tolerations
array
Set of tolerations to apply to the workload.
PvcInstance
object
WorkloadMeta1
object
11 properties
8 required
ProjectId
string
The id of the project.
CpuMemoryRequest
string
The amount of CPU memory to allocate for this workload (1G, 20M, .etc). The workload will receive at least this amount of memory. Note that the workload will n…
ExtendedResource
object
Quantity of an extended resource.
3 properties
HttpResponse
object
2 properties
2 required
EmptyDirItems
array
A list of emptyDir volumes to mount in the workload.
GpuPortionRequest
number
Required if and only if gpuRequestType is portion. States the portion of the GPU to allocate for the created workload, per GPU device, between 0 and 1. The def…
GpuPortionLimit
number
Limitations on the portion consumed by the workload, per GPU device. The system guarantees The gpuPotionLimit must be no less than the gpuPortionRequest.
Probes
object
Probes are used to determine if the container is healthy and ready to accept traffic.
1 property
EmptyDirInstance
object
UpdateRequest
NodeType3
string
Nodes (machines), or a group of nodes on which the workload will run. To use this feature, your Administrator will need to label nodes. For more information, s…
ExtendedResources
array
Extended resources and their quantity.
DistributedInferenceServingPortContainerAndProtocol
object
2 properties
ConfigMap
object
4 properties
Secret5
ImagePullSecret
object
A reference to a secret in the same namespace used to pull container images.
3 properties
TolerationOperator
string
A key's relationship to the value. Equal uses key and value. Exists is equivalent to wildcard for value, so that a workload can tolerate all taints of a partic…
RunAsUid
integer
The user id to run the entrypoint of the container which executes the workspace. Default to the value specified in the environment asset runAsUid field (option…
PvcFieldsUpdatable
object
1 property
PvcAccessModes
object
Default access mode(s) applied to newly created PVCs unless explicitly overridden.
3 properties
Specification
The full machine-readable OpenAPI contract behind this narrative.
Source
More from NVIDIA Run:ai 12
Other APIs NVIDIA Run:ai publishes across the network.
NVIDIA Run:ai Access Keys API
NVIDIA Run:ai Access rules API
NVIDIA Run:ai Administrator Command Line Interface API
NVIDIA Run:ai AI Applications API
NVIDIA Run:ai Applications API
NVIDIA Run:ai AuditLogs API
NVIDIA Run:ai Clusters API
NVIDIA Run:ai Compute API
NVIDIA Run:ai ConfigMap API
NVIDIA Run:ai Credentials API
NVIDIA Run:ai Datasources API
NVIDIA Run:ai Datavolumes API
#Distributed Inferences