NVIDIA Run:ai Inferences API
Inference workloads deploy trained models into a production environment to generate predictions from live data. These workloads are prioritized over Trainings and Workspaces during scheduling. NVIDIA Run:ai Inference workloads support auto-scaling to maintain service-level agreements (SLAs) by dynamically adjusting resources as demand changes.
NVIDIA Run:ai Inferences API is one of 62 APIs that NVIDIA Run:ai publishes on the APIs.io network, described by a machine-readable OpenAPI specification.
Tagged areas include Inferences. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, and authentication docs.
This API exposes 6 operations across 4 paths, and defines 156 schemas. It is described by OpenAPI 3.0.3, at version latest.
Requests are made against a single base URL, https://app.run.ai.
6 operations
4 paths
156 schemas
1 DELETE3 GET1 PATCH1 POST
Metadata
The identity and technical contract details declared by the specification.
Authentication & Security 1
NVIDIA Run:ai Inferences API declares
1 security scheme
for authenticating requests.
It accepts HTTP bearer tokens (JWT) (bearerAuth).
By default, every request must be authenticated.
bearerAuth— Bearer authentication
Paths & Operations 6
Across 4 paths, the API surfaces 6 operations — 1 DELETE, 3 GET, 1 PATCH, 1 POST. Each is listed below with its method, path, parameters, and response codes.
Inferences 6
Inference workloads deploy trained models into a production environment to generate predictions from live data. These workloads are prioritized over Trainings and Workspaces durin…
POST
/api/v1/workloads/inferences
Create an inference.
DELETE
/api/v1/workloads/inferences/{workloadId}
Delete an inference.
GET
/api/v1/workloads/inferences/{workloadId}
Get inference data.
PATCH
/api/v1/workloads/inferences/{workloadId}
Update inference spec. [Experimental]
GET
/api/v1/workloads/inferences/{workloadId}/metrics
Get inference metrics data.
GET
/api/v1/workloads/inferences/{workloadId}/pods/{podId}/metrics
Get inference pod's metrics data.
Schemas 156
The contract defines 156 schemas that model the data the API accepts and returns. The most detailed are WorkloadMeta1 (11 properties), ExposedUrl (9 properties), EnvironmentVariable (8 properties), Toleration (7 properties). Each schema is shown below with its type and property counts.
Category
string
Specify the workload category assigned to the workload. Categories are used to classify and monitor different types of workloads within the NVIDIA Run:ai platf…
PvcFieldsNonUpdatable
object
6 properties
AutoScalingMetricField
object
1 property
Probe
object
6 properties
ExposedUrl
object
A URL for accessing the workload.
9 properties
WorkloadId2
string
A unique ID of the workload.
NodeAffinityRequired
object
If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements…
1 property
UidGidSource
string
Indicate the way to determine the user and group ids of the container. The options are a. fromTheImage - user and group ids are determined by the docker image…
SecretInstance2
object
Labels
array
Set of labels to populate into the container running the workload.
InitializationTimeoutField
object
1 property
Error
object
3 properties
2 required
MetricThresholdField
object
1 property
ExcludeField
object
1 property
InferenceSpecSpec
CpuMemoryLimit
string
Limitations on the CPU memory to allocate for this workload (1G, 20M, .etc). The system guarantees that this workload will not be able to consume more than thi…
GitInstance
object
PvcAddedAttrValues
array
an optional array of key-values pairs that are written as annotations on the created PVC. the allowed attributes are determined according to the storage class…
PvcClaimSize
string
Requested size for the PVC. Mandatory when existingPvc is false. Recommended sizes: TB/GB/MB/TIB/GIB/MIB
Ports
array
Set of container ports that the workload exposes.
InferenceFields
object
InferenceUpdateSpec
object
The specifications of the inference to be updated.
1 property
Toleration
object
Toleration details.
7 properties
ClusterId
string
The id of the cluster.
SecretFieldsNonUpdatable
object
1 property
Pvc
GpuMemoryRequest
string
Required if and only if gpuRequestType is memory. States the GPU memory to allocate for the created workload, per GPU device. Note that the workload will not b…
Capabilities
array
Add POSIX capabilities to running containers. Defaults to the default set of capabilities granted by the container runtime.
HostPath
object
4 properties
InferencesMaxReplicasField
object
1 property
SecretFieldsUpdatable
object
2 properties
EnvironmentVariable
object
Details of an environment variable which is populated into the container.
8 properties
LargeShmRequest
boolean
A large /dev/shm device to mount into a container running the created workload. An shm is a shared file system mounted on RAM.
EnvironmentVariableSecret
object
Details of the secret and key use to populate the environment variable
2 properties
RunAsGid
integer
The group id to run the entrypoint of the container which executes the workspace. Default to the value specified in the environment asset runAsGid field (optio…
MeasurementResponse
object
3 properties
2 required
Annotations
array
Set of annotations to populate into the container running the workload.
AutoScalingMetric
string
The metric to use for autoscaling. Mandatory if minReplicas < maxReplicas, except for the special case where minReplicas is set to 0 and maxReplicas is set to…
WorkloadDesiredPhase
string
The desired phase of the workload.
ProbeHandler
object
The action taken to determine the health of the container. (mandatory)
1 property
MatchExpressionOperator
string
Represents a key's relationship to a set of values (mandatory).
TolerationEffect
string
The taint effect to match. (mandatory)
GitPassword
object
3 properties
ImagePullPolicy
string
Image pull policy. Defaults to Always if :latest tag is specified, otherwise it is IfNotPresent.
HostPathItems
array
Set of host paths to use in the workload.
WorkloadName
string
The name of the workload.
ClaimInfo
object
Claim information for the newly created PVC. The information should not be provided when attempting to use existing PVC.
5 properties
DepartmentId2
string
The id of the department.
ServingPort
object
A port for accessing the inference service
InferenceSpec
object
The specifications of the inference to be created.
1 property
HostPathMountPropagation
string
Share this volumes mount with other containers. If set to HostToContainer, this volume mount will receive all subsequent mounts that are mounted to this volume…
StorageInstanceName
object
1 property
EnvironmentVariables
array
Set of environment variables to populate into the container running the workload.
ServingPortContainerAndProtocol
object
2 properties
CpuCoreRequest
number
CPU units to allocate for the created workload (0.5, 1, .etc). The workload will receive at least this amount of CPU. Note that the workload will not be schedu…
Preemptibility
string
Specifies whether the workload can be preempted by higher-priority workloads. Valid values are preemptible and non-preemptible. If explicitly set, this value t…
RelatedUrls
array
Set of URLs that are related to the workload.
SupplementalGroups
string
Comma separated list of groups that the user running the container belongs to, in addition to the group indicated by runAsGid. Use only when the source uid/gid…
DataVolume
object
2 properties
ConfigMapItems
array
Set of config map volumes to use in the workload
InitialReplicasField
object
1 property
AutoScalingField
object
1 property
ImagePullSecrets
array
A list of references to Kubernetes secrets in the same namespace used for pulling container images.
AutoScalingMetricFields
object
ServingPortProtocol
string
The protocol used by the port, defaults to http
GitSecretRef
object
4 properties
Image
string
Docker image name. For more information, see [Images](https://kubernetes.io/docs/concepts/containers/images). The image name is mandatory for creating a worklo…
EnvironmentVariableUserCredential
object
Defines a reference to a user-created credential and a specific key within that credential whose value will populate the environment variable. User credentials…
2 properties
Annotation
object
Annotation details to be populated into the container.
3 properties
CreateHomeDir
boolean
When set to true, creates a home directory for the container.
GpuMemoryLimit
string
Limitation on the memory consumed by the workload, per GPU device. The system guarantees The gpuMemoryLimit must be no less than gpuMemoryRequest.
EnvironmentVariableConfigMap
object
Details of the configMap and key use to populate the environment variable
2 properties
CpuCoreLimit
number
Limitations on the number of CPUs consumed by the workload (0.5, 1, .etc). The system guarantees that this workload will not be able to consume more than this…
Capability
string
AutoScalingCommonFields
object
WorkloadCreationMeta
object
4 properties
3 required
GpuDevicesRequest
integer
Requested number of GPU devices. Currently if more than one device is requested, it is not possible to provide values for gpuMemory or gpuPortion.
NodePools
array
A prioritized list of node pools for the scheduler to run the workload on. The scheduler will always try to use the first node pool before moving to the next o…
PvcAddedAttrValue
object
2 properties
1 required
InferencesMinReplicasField
object
1 property
NfsInstance
object
SecretItems1
array
Set of secret volumes to use in the workload
MetricThresholdPercentageField
object
1 property
PodAffinity
object
Pod affinity scheduling rules (e.g. co-locate this workload in the same node, zone, etc. as some other workloads).
2 properties
NfsItems
array
Set of nfs volumes to use in the workload.
EnvironmentVariablePodFieldReference
object
Details of the field-reference and key use to populate the environment variable
1 property
ServingPortAccessAuthorizationTypeEnum
string
Specifies who can send inference requests to the serving endpoint: Possible values: - public: No authorization is required. (Default) - authenticatedUsers: Any…
MatchExpression
object
A selector that contains values, a key, and an operator that relates the key and values.
3 properties
2 required
GpuRequestType
string
Sets the unit type for GPU resources requests. Stated in terms of portion or memory. Sets the unit type for other GPU request fields. If gpuDevicesRequest 1, o…
EmptyDir
object
3 properties
PvcItems
array
Set of pvc persistent volume claims to use in the workload.
DataVolumeItems
array
Set of data volumes to use in the workload.
PodAffinityType
string
The affinity type, required or preferred. (mandatory)
ReadOnlyRootFileSystem
boolean
If true, mounts the container's root filesystem as read-only.
ConfigMapInstance
object
Command
string
A command to the server as the entry point of the container running the workload.
ExposedUrls
array
Set of container ports that the workload exposes via URLs.
ActivationReplicasField
object
1 property
ProbeHandlerScheme
string
Scheme to use for connecting to the host, defaults to HTTP.
RunAsNonRoot
boolean
Force the container to run as a non-root user.
SubmitWithTemplateId
object
1 property
Args
string
Arguments to the command that the container running the workload executes.
ServingConfiguration
object
The inference workload serving configuration.
2 properties
DataVolumeInstance
object
PriorityClass
string
Specifies the priority class for the workload, which determines its scheduling behavior. Valid values are: very-low, low, medium-low, medium, medium-high, high…
Phase
string
NodeSelectorTerm
object
A null or empty node selector term matches no objects. The requirements of them are ANDed.
1 property
WorkingDir
string
Container's working directory. If not specified, the container runtime default will be used. This may be configured in the container image.
PvcVolumeMode
string
Default volume mode for the PVC. Choose between Filesystem (default) or Block.
Label
object
Label details to be populated into the container.
3 properties
GitItems
array
Set of git repositories to use in the workload.
SeccompProfileType
string
Indicates which kind of seccomp profile will be applied to the container. The options are a. RuntimeDefault - the container runtime default profile should be u…
ServingConfigurationField
object
1 property
Inference1
DefaultMode
string
File permission mode in octal string format. This value must be a 4-digit octal number, representing the default file mode when mounting a Secret or ConfigMap…
Tolerations
array
Set of tolerations to apply to the workload.
Nfs
object
4 properties
InferencePodMetricType
string
Specifies what data to request.
PvcInstance
object
WorkloadMeta1
object
11 properties
8 required
GitAuthenticationMethod
string
Specifies the authentication method to use when accessing the Git repository. This is required for private repositories - password - Authenticate using a usern…
InferenceUpdateRequest
AutoScaling
object
ProjectId
string
The id of the project.
CpuMemoryRequest
string
The amount of CPU memory to allocate for this workload (1G, 20M, .etc). The workload will receive at least this amount of memory. Note that the workload will n…
ExtendedResource
object
Quantity of an extended resource.
3 properties
HttpResponse
object
2 properties
2 required
InferenceUpdateSpecServingConfiguration
object
1 property
EmptyDirItems
array
A list of emptyDir volumes to mount in the workload.
PortServiceType
string
The service type of the port (mandatory).
ConcurrencyHardLimitField
object
1 property
GpuPortionRequest
number
Required if and only if gpuRequestType is portion. States the portion of the GPU to allocate for the created workload, per GPU device, between 0 and 1. The def…
RelatedUrl
object
A URL that is related to the workload. For example, a URL to an external server providing statistics or logging about the workload.
4 properties
MetricsResponse
object
1 property
1 required
GpuPortionLimit
number
Limitations on the portion consumed by the workload, per GPU device. The system guarantees The gpuPotionLimit must be no less than the gpuPortionRequest.
HostPathInstance
object
Probes
object
Probes are used to determine if the container is healthy and ready to accept traffic.
1 property
EmptyDirInstance
object
ScaleDownDelayField
object
1 property
NodeType3
string
Nodes (machines), or a group of nodes on which the workload will run. To use this feature, your Administrator will need to label nodes. For more information, s…
ExtendedResources
array
Extended resources and their quantity.
ConfigMap
object
4 properties
GitCommon
object
4 properties
InferenceCreationRequest
ScaleToZeroRetentionField
object
1 property
Secret5
Port
object
A port for accessing the workload.
7 properties
InferenceUpdateSpecAutoscaling
object
1 property
ServingPortAccess
object
4 properties
ImagePullSecret
object
A reference to a secret in the same namespace used to pull container images.
3 properties
AuthorizationType
string
Specifies who can access the connection URL: - authenticatedUsers: Any authenticated user or service account can access the URL; the authorizedUsers and author…
TolerationOperator
string
A key's relationship to the value. Equal uses key and value. Exists is equivalent to wildcard for value, so that a workload can tolerate all taints of a partic…
InferenceWorkloadMetricType
string
Specify which data to request.
RunAsUid
integer
The user id to run the entrypoint of the container which executes the workspace. Default to the value specified in the environment asset runAsUid field (option…
PvcFieldsUpdatable
object
1 property
PvcAccessModes
object
Default access mode(s) applied to newly created PVCs unless explicitly overridden.
3 properties
Specification
The full machine-readable OpenAPI contract behind this narrative.
Source
More from NVIDIA Run:ai 12
Other APIs NVIDIA Run:ai publishes across the network.
NVIDIA Run:ai Access Keys API
NVIDIA Run:ai Access rules API
NVIDIA Run:ai Administrator Command Line Interface API
NVIDIA Run:ai AI Applications API
NVIDIA Run:ai Applications API
NVIDIA Run:ai AuditLogs API
NVIDIA Run:ai Clusters API
NVIDIA Run:ai Compute API
NVIDIA Run:ai ConfigMap API
NVIDIA Run:ai Credentials API
NVIDIA Run:ai Datasources API
NVIDIA Run:ai Datavolumes API
#Inferences