List experiments
List out all experiments. The experiments are sorted by creation date, with the most recently-created experiments coming first
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Query Parameters
limit
integerLimit the number of objects to return
0
starting_after
stringPagination cursor id.
For example, if the final item in the last page you fetched had an id of foo
, pass starting_after=foo
to fetch the next page. Note: you may only pass one of starting_after
and ending_before
"uuid"
ending_before
stringPagination cursor id.
For example, if the initial item in the last page you fetched had an id of foo
, pass ending_before=foo
to fetch the previous page. Note: you may only pass one of starting_after
and ending_before
"uuid"
ids
Any properties in string, array<string>Filter search results to a particular set of object IDs. To specify a list of IDs, include the query param multiple times
experiment_name
stringName of the experiment to search for
project_name
stringName of the project to search for
project_id
stringProject id
"uuid"
org_name
stringFilter search results to within a particular organization
curl -X GET "https://api.braintrust.dev/v1/experiment?limit=0&starting_after=497f6eca-6276-4993-bfeb-53cbbbba6f08&ending_before=497f6eca-6276-4993-bfeb-53cbbbba6f08&ids=497f6eca-6276-4993-bfeb-53cbbbba6f08&experiment_name=string&project_name=string&project_id=497f6eca-6276-4993-bfeb-53cbbbba6f08&org_name=string" \
-H "Authorization: Bearer <token>"
Returns a list of experiment objects
{
"objects": [
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"created": "2019-08-24T14:15:22Z",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"commit": "string",
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"deleted_at": "2019-08-24T14:15:22Z",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
"metadata": {
"property1": null,
"property2": null
}
}
]
}
Create experiment
Create a new experiment. If there is an existing experiment in the project with the same name as the one specified in the request, will return the existing experiment unmodified
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Request Body
application/json
OptionalAny desired information about the new experiment object
project_id
Unique identifier for the project that the experiment belongs under
"uuid"
name
stringName of the experiment. Within a project, experiment names are unique
1
description
stringTextual description of the experiment
repo_info
objectMetadata about the state of the repo when the experiment was created
base_exp_id
stringId of default base experiment to compare against when viewing this experiment
"uuid"
dataset_id
stringIdentifier of the linked dataset, or null if the experiment is not linked to a dataset
"uuid"
dataset_version
stringVersion number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
public
booleanWhether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
metadata
objectUser-controlled metadata about the experiment
ensure_new
booleanNormally, creating an experiment with the same name as an existing experiment will return the existing one un-modified. But if ensure_new
is true, registration will generate a new experiment with a unique name in case of a conflict.
curl -X POST "https://api.braintrust.dev/v1/experiment" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"metadata": {
"property1": null,
"property2": null
},
"ensure_new": true
}'
Returns the new experiment object
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"created": "2019-08-24T14:15:22Z",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"commit": "string",
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"deleted_at": "2019-08-24T14:15:22Z",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
"metadata": {
"property1": null,
"property2": null
}
}
Get experiment
Get an experiment object by its id
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X GET "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08" \
-H "Authorization: Bearer <token>"
Returns the experiment object
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"created": "2019-08-24T14:15:22Z",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"commit": "string",
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"deleted_at": "2019-08-24T14:15:22Z",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
"metadata": {
"property1": null,
"property2": null
}
}
Partially update experiment
Partially update an experiment object. Specify the fields to update in the payload. Any object-type fields will be deep-merged with existing content. Currently we do not support removing fields or setting them to null.
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Request Body
application/json
OptionalFields to update
name
stringName of the experiment. Within a project, experiment names are unique
description
stringTextual description of the experiment
repo_info
objectMetadata about the state of the repo when the experiment was created
base_exp_id
stringId of default base experiment to compare against when viewing this experiment
"uuid"
dataset_id
stringIdentifier of the linked dataset, or null if the experiment is not linked to a dataset
"uuid"
dataset_version
stringVersion number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.
public
booleanWhether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization
metadata
objectUser-controlled metadata about the experiment
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X PATCH "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"name": "string",
"description": "string",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"metadata": {
"property1": null,
"property2": null
}
}'
Returns the experiment object
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"created": "2019-08-24T14:15:22Z",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"commit": "string",
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"deleted_at": "2019-08-24T14:15:22Z",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
"metadata": {
"property1": null,
"property2": null
}
}
Delete experiment
Delete an experiment object by its id
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X DELETE "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08" \
-H "Authorization: Bearer <token>"
Returns the deleted experiment object
{
"id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"name": "string",
"description": "string",
"created": "2019-08-24T14:15:22Z",
"repo_info": {
"commit": "string",
"branch": "string",
"tag": "string",
"dirty": true,
"author_name": "string",
"author_email": "string",
"commit_message": "string",
"commit_time": "string",
"git_diff": "string"
},
"commit": "string",
"base_exp_id": "4838cee2-a665-4545-aa9f-483678c01a6b",
"deleted_at": "2019-08-24T14:15:22Z",
"dataset_id": "8c4c51f1-f6f3-43bc-b65d-7415e8ef22c0",
"dataset_version": "string",
"public": true,
"user_id": "a169451c-8525-4352-b8ca-070dd449a1a5",
"metadata": {
"property1": null,
"property2": null
}
}
Insert experiment events
Insert a set of events into the experiment
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Request Body
application/json
OptionalAn array of experiment events to insert
events
A list of experiment events to insert
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X POST "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08/insert" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"events": [
{
"input": null,
"output": null,
"expected": null,
"error": null,
"scores": {
"property1": 1,
"property2": 1
},
"metadata": {
"model": "string",
"property1": null,
"property2": null
},
"tags": [
"string"
],
"metrics": {
"start": 0,
"end": 0,
"prompt_tokens": 0,
"completion_tokens": 0,
"tokens": 0,
"caller_functionname": null,
"caller_filename": null,
"caller_lineno": null,
"property1": 0,
"property2": 0
},
"context": {
"caller_functionname": "string",
"caller_filename": "string",
"caller_lineno": 0,
"property1": null,
"property2": null
},
"span_attributes": {
"name": "string",
"type": "llm",
"property1": null,
"property2": null
},
"id": "string",
"created": "2019-08-24T14:15:22Z",
"origin": {
"object_type": "experiment",
"object_id": "463a83d0-a816-4902-abba-2486e0c0a0bb",
"id": "string",
"_xact_id": "string",
"created": "string"
},
"_object_delete": true,
"_is_merge": true,
"_merge_paths": [
[
"string"
]
],
"_parent_id": "string",
"span_id": "string",
"root_span_id": "string",
"span_parents": [
"string"
]
}
]
}'
Returns the inserted row ids
{
"row_ids": [
"string"
]
}
Fetch experiment (GET form)
Fetch the events in an experiment. Equivalent to the POST form of the same path, but with the parameters in the URL query rather than in the request body. For more complex queries, use the POST /btql
endpoint.
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Path Parameters
experiment_id
Experiment id
"uuid"
Query Parameters
limit
integerlimit the number of traces fetched
Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest _xact_id
. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier _xact_id
. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by id
) from your combined result set.
The limit
parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.
0
max_xact_id
stringDEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.
Together, max_xact_id
and max_root_span_id
form a pagination cursor
Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id)
. See the documentation of limit
for an overview of paginating fetch queries.
max_root_span_id
stringDEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.
Together, max_xact_id
and max_root_span_id
form a pagination cursor
Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id)
. See the documentation of limit
for an overview of paginating fetch queries.
version
stringRetrieve a snapshot of events from a past time
The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id
returned by a past fetch as the version to reproduce that exact fetch.
curl -X GET "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08/fetch?limit=0&max_xact_id=string&max_root_span_id=string&version=string" \
-H "Authorization: Bearer <token>"
Returns the fetched rows
{
"events": [
{
"id": "string",
"_xact_id": "string",
"created": "2019-08-24T14:15:22Z",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
"input": null,
"output": null,
"expected": null,
"error": null,
"scores": {
"property1": 1,
"property2": 1
},
"metadata": {
"model": "string",
"property1": null,
"property2": null
},
"tags": [
"string"
],
"metrics": {
"start": 0,
"end": 0,
"prompt_tokens": 0,
"completion_tokens": 0,
"tokens": 0,
"caller_functionname": null,
"caller_filename": null,
"caller_lineno": null,
"property1": 0,
"property2": 0
},
"context": {
"caller_functionname": "string",
"caller_filename": "string",
"caller_lineno": 0,
"property1": null,
"property2": null
},
"span_id": "string",
"span_parents": [
"string"
],
"root_span_id": "string",
"span_attributes": {
"name": "string",
"type": "llm",
"property1": null,
"property2": null
},
"is_root": true,
"origin": {
"object_type": "experiment",
"object_id": "463a83d0-a816-4902-abba-2486e0c0a0bb",
"id": "string",
"_xact_id": "string",
"created": "string"
}
}
],
"cursor": "string"
}
Fetch experiment (POST form)
Fetch the events in an experiment. Equivalent to the GET form of the same path, but with the parameters in the request body rather than in the URL query. For more complex queries, use the POST /btql
endpoint.
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Request Body
application/json
OptionalFilters for the fetch query
limit
integerlimit the number of traces fetched
Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest _xact_id
. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier _xact_id
. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by id
) from your combined result set.
The limit
parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.
0
cursor
stringAn opaque string to be used as a cursor for the next page of results, in order from latest to earliest.
The string can be obtained directly from the cursor
property of the previous fetch query
max_xact_id
stringDEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.
Together, max_xact_id
and max_root_span_id
form a pagination cursor
Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id)
. See the documentation of limit
for an overview of paginating fetch queries.
max_root_span_id
stringDEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.
Together, max_xact_id
and max_root_span_id
form a pagination cursor
Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id)
. See the documentation of limit
for an overview of paginating fetch queries.
version
stringRetrieve a snapshot of events from a past time
The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id
returned by a past fetch as the version to reproduce that exact fetch.
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X POST "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08/fetch" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"limit": 0,
"cursor": "string",
"max_xact_id": "string",
"max_root_span_id": "string",
"version": "string"
}'
Returns the fetched rows
{
"events": [
{
"id": "string",
"_xact_id": "string",
"created": "2019-08-24T14:15:22Z",
"project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
"experiment_id": "916afd89-cac5-4339-9c59-dd068abdfa69",
"input": null,
"output": null,
"expected": null,
"error": null,
"scores": {
"property1": 1,
"property2": 1
},
"metadata": {
"model": "string",
"property1": null,
"property2": null
},
"tags": [
"string"
],
"metrics": {
"start": 0,
"end": 0,
"prompt_tokens": 0,
"completion_tokens": 0,
"tokens": 0,
"caller_functionname": null,
"caller_filename": null,
"caller_lineno": null,
"property1": 0,
"property2": 0
},
"context": {
"caller_functionname": "string",
"caller_filename": "string",
"caller_lineno": 0,
"property1": null,
"property2": null
},
"span_id": "string",
"span_parents": [
"string"
],
"root_span_id": "string",
"span_attributes": {
"name": "string",
"type": "llm",
"property1": null,
"property2": null
},
"is_root": true,
"origin": {
"object_type": "experiment",
"object_id": "463a83d0-a816-4902-abba-2486e0c0a0bb",
"id": "string",
"_xact_id": "string",
"created": "string"
}
}
],
"cursor": "string"
}
Feedback for experiment events
Log feedback for a set of experiment events
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Request Body
application/json
OptionalAn array of feedback objects
feedback
A list of experiment feedback items
Path Parameters
experiment_id
Experiment id
"uuid"
curl -X POST "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08/feedback" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"feedback": [
{
"id": "string",
"scores": {
"property1": 1,
"property2": 1
},
"expected": null,
"comment": "string",
"metadata": {
"property1": null,
"property2": null
},
"source": "app",
"tags": [
"string"
]
}
]
}'
Returns a success status
{
"status": "success"
}
Summarize experiment
Summarize experiment
Authorization
Authorization
Most Braintrust endpoints are authenticated by providing your API key as a header Authorization: Bearer [api_key]
to your HTTP request. You can create an API key in the Braintrust organization settings page.
In: header
Path Parameters
experiment_id
Experiment id
"uuid"
Query Parameters
summarize_scores
booleanWhether to summarize the scores and metrics. If false (or omitted), only the metadata will be returned.
comparison_experiment_id
stringThe experiment to compare against, if summarizing scores and metrics. If omitted, will fall back to the base_exp_id
stored in the experiment metadata, and then to the most recent experiment run in the same project. Must pass summarize_scores=true
for this id to be used
"uuid"
curl -X GET "https://api.braintrust.dev/v1/experiment/497f6eca-6276-4993-bfeb-53cbbbba6f08/summarize?summarize_scores=true&comparison_experiment_id=497f6eca-6276-4993-bfeb-53cbbbba6f08" \
-H "Authorization: Bearer <token>"
Experiment summary
{
"project_name": "string",
"experiment_name": "string",
"project_url": "http://example.com",
"experiment_url": "http://example.com",
"comparison_experiment_name": "string",
"scores": {
"property1": {
"name": "string",
"score": 1,
"diff": -1,
"improvements": 0,
"regressions": 0
},
"property2": {
"name": "string",
"score": 1,
"diff": -1,
"improvements": 0,
"regressions": 0
}
},
"metrics": {
"property1": {
"name": "string",
"metric": 0,
"unit": "string",
"diff": 0,
"improvements": 0,
"regressions": 0
},
"property2": {
"name": "string",
"metric": 0,
"unit": "string",
"diff": 0,
"improvements": 0,
"regressions": 0
}
}
}