Runloop Benchmark API is one of 24 APIs that Runloop publishes on the APIs.io network, described by a machine-readable OpenAPI specification.
This API exposes 19 JSON Schema definitions.
Tagged areas include Benchmarks. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, a JSON-LD context, and 19 JSON Schemas.
This API exposes
28 operations
across 25 paths,
and defines 73 schemas.
It is described by OpenAPI 3.2.0, at version 0.1.
Requests are made against a single base URL, https://api.runloop.ai.
The identity and technical contract details declared by the specification.
ScenarioEnvironment
object
ScenarioEnvironmentParameters specify the environment in which a Scenario will be run.
4 properties
ScenarioRunListView
object
3 properties
2 required
BenchmarkJobView
object
A BenchmarkJobView represents a benchmark job that runs a set of scenarios entirely on runloop.
9 properties
4 required
ScenarioOutcomeView
object
Outcome data for a single scenario execution, including its final state and scoring results.
7 properties
3 required
ScenarioDefinitionListView
object
3 properties
2 required
ScoringContract
object
InputContextView specifies the problem statement along with all additional context for a Scenario.
1 property
1 required
LaunchParameters
object
LaunchParameters enable you to customize the resources available to your Devbox as well as the environment set up that should be completed before the Devbox is…
13 properties
ScoringContractResultView
object
A ScoringContractResultView represents the result of running all scoring functions on a given input context.
2 properties
2 required
FileMount
object
3 properties
3 required
BrokerMount
object
6 properties
2 required
InputContext
object
InputContextView specifies the problem statement along with all additional context for a Scenario.
2 properties
1 required
LifecycleConfigurationParameters
object
Lifecycle configuration for Devbox idle and resume behavior. Configure idle policy via afteridle and resume triggers via resumetriggers.
2 properties
FailureReason
object
Information about why a scenario execution failed
2 properties
2 required
ScenarioDefinitionView
object
A ScenarioDefinitionView represents a repeatable AI coding evaluation test, complete with initial environment and scoring contract.
13 properties
6 required
BenchmarkRunListView
object
3 properties
2 required
AgentMount
object
5 properties
3 required
AstGrepScoringFunction
object
AstGrepScoringFunction utilizes structured coach search for scoring.
4 properties
3 required
BenchmarkDefinitionListView
object
3 properties
2 required
BenchmarkDefinitionJobSpec
object
Specifies a benchmark definition with runtime configuration. The benchmark definition's scenarios will be executed using the provided agent and orchestrator co…
4 properties
3 required
JobAgentConfig
object
Configuration for an agent in a benchmark job
7 properties
2 required
ScenarioDefinitionJobSpec
object
Specifies a set of scenarios with runtime configuration. The scenarios will be executed using the provided agent and orchestrator configurations.
4 properties
3 required
IdleConfigurationParameters
object
2 properties
2 required
TestBasedScoringFunction
object
TestBasedScoringFunction writes test files to disk and executes a test command to verify the solution.
3 properties
1 required
JobAgentEnvironment
object
Environment configuration for an agent in a benchmark job
2 properties
BenchmarkDefJobSource
object
Benchmark definition job source
3 properties
2 required
CustomScoringFunction
object
CustomScoringFunction is a custom, user defined scoring function.
3 properties
2 required
ScenarioRunView
object
A ScenarioRunView represents a single run of a Scenario on a Devbox. When completed, the ScenarioRun will contain the final score and output of the run.
13 properties
5 required
HarborJobSpec
object
Harbor-based job specification with inline YAML configuration.
2 properties
2 required
CodeMount
object
6 properties
3 required
RunProfile
object
5 properties
PythonScriptScoringFunction
object
PythonScriptScoringFunction will run a python script in the context of your environment as a ScoringFunction.
4 properties
2 required
MetadataKeysView
object
1 property
ScoringFunction
object
ScoringFunction specifies a method of scoring a Scenario.
3 properties
3 required
IdleAction
string
Action to take after Devbox idle timer is triggered. shutdown: Shutdown the Devbox. suspend: Suspend the Devbox.
BenchmarkCreateParameters
object
BenchmarkCreateParameters contain the set of parameters to create a Benchmark.
7 properties
1 required
BenchmarkJobListView
object
3 properties
2 required
BenchmarkRunView
object
A BenchmarkRunView represents a run of a complete set of Scenarios, organized under a Benchmark or created by a BenchmarkJob.
11 properties
4 required
BenchmarkJobCreateParameters
object
BenchmarkJobCreateParameters contain the set of parameters to create a BenchmarkJob.
2 properties
JobOrchestratorConfig
object
Orchestrator configuration for benchmark job execution
4 properties
BenchmarkUpdateParameters
object
BenchmarkUpdateParameters contain the set of parameters to update a Benchmark. All fields are optional - null fields preserve existing values, provided fields…
7 properties
ScenarioDefinitionStatus
string
BrokerMountProtocol
string
TestFile
object
2 properties
ScenariosJobSource
object
Scenarios job source with a list of scenario definition IDs
2 properties
2 required
BashScriptScoringFunction
object
BashScriptScoringFunction is a scoring function specified by a bash script that will be run in the context of your environment.
2 properties
1 required
CommandScoringFunction
object
CommandScoringFunction executes a single command and checks the result.The output of the command will be printed. Scoring will passed if the command returns st…
2 properties
1 required
UserParameters
object
Configuration for the Linux user in the Devbox environment.
2 properties
2 required
ScoringFunctionResultView
object
A ScoringFunctionResultView represents the result of running a single scoring function on a given input context.
4 properties
4 required
ResumeTriggers
object
Triggers that can resume a suspended Devbox.
2 properties
BenchmarkDefinitionView
object
A BenchmarkDefinitionView represents a grouped set of Scenarios that together form a Benchmark.
10 properties
5 required
JobSpec
object
Job specification describing scenarios and execution configuration
3 properties
2 required
StartBenchmarkRunParameters
object
4 properties
1 required
BenchmarkOutcomeView
object
Outcome data for a single benchmark run within a benchmark job, representing results for one agent configuration.
9 properties
6 required
BenchmarkScenarioUpdateParameters
object
2 properties
HarborJobSource
object
Harbor job source with inline YAML configuration
2 properties
2 required
ExternalApiAgentConfig
object
Configuration for externally-driven benchmark runs via API
2 properties
1 required
MetadataValuesView
object
2 properties
InProgressRunView
object
A lightweight view of a benchmark run currently in progress, showing basic execution details without full outcome data.
5 properties
3 required
ScoringFunctionResultViewState
string
ResourceSize
string
The size of the Devbox resources for Runloop to allocate. XSMALL: 0.5 cpu x 1GiB memory x 4GiB disk SMALL: 1 cpu x 2GiB memory x 4GiB disk MEDIUM: 2 cpu x 4GiB…
ObjectMount
object
3 properties
3 required
The full machine-readable OpenAPI contract behind this narrative.
Other APIs Runloop publishes across the network.
