Setting up Braintrust in your AWS Account

This guide walks through the process of setting up Braintrust in your AWS account.

Before following these instructions, make sure you have account access, and are able to sign in to the app.

There are two methods you can follow:

1. Use the braintrust command-line (CLI) tool. This is the recommended method and requires just a single command.
2. Use the AWS console. If you need more control or cannot get AWS credentials for the CLI, this option allows you to run the whole process within the CLI.

Setting up the stack: CLI

pip install --upgrade 'braintrust[cli]'
 
# verify the installation worked
braintrust --help

braintrust install api --create <YOUR_STACK_NAME> \
  --org-name <YOUR_ORG_NAME> \
  --provisioned-concurrency 1

Stack with name braintrust has been updated with status: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
Universal URL: https://dfwhllz61x709.cloudfront.net

Setting up the stack: CloudFormation console

Verifying the stack

Run the following command to test that the stack is running. The first time you run it, AWS may do some setup work to provision your lambda function, and it can take up to 30 seconds to run.

curl -X GET '<YOUR_UNIVERSAL_URL>/ping'

You should see a response like {"id":"80c48a10-4888-4382-a55b-255018e70fe5","email":"___braintrust_anon_user___@braintrustdata.com","organizations":[]}.

Configure your organization's endpoint

Visit your organization's settings page and set the API URL captured above. You can skip the realtime URL and proxy URL, unless you have an advanced need that requires it (see the docker guide for more information).

Once you configure the URL, select Save. The page automatically attempts to test that you're authorized to access the URL.

Test the application end-to-end

Hooray! At this point you should be able to test the full application. The easiest way to do this is by using the Python SDK.

This simple Python script will run a full loop of using Braintrust and setting up an experiment.

import braintrust
 
# NOTE:
# * You can specify your API key in the `api_key` parameter, or set it as an environment variable
#   `BRAINTRUST_API_KEY`. If you don't specify it, the SDK will look for the environment variable.
# * You should not specify api_url in the SDK. It will automatically use the value you set in the
#   settings page.
experiment = braintrust.init(project="SetupTest", org_name="my_org.com", api_key="sk-****")
experiment.log(
    input={"test": 1},
    output="foo",
    expected="bar",
    scores={
        "n": 0.5,
    },
    metadata={
        "id": 1,
    },
)
print(experiment.summarize())

Maintaining your installation

Most new Braintrust releases do not require stack updates. Occasionally, however, you will need to update the stack to get access to new features and performance enhancements. Like installation, you can update the stack through either the CLI or AWS console.

Using the Braintrust CLI

To update your stack, simply run (replacing <YOUR_STACK_NAME>):

braintrust install api <YOUR_STACK_NAME> --update-template

You can also use this command to change parameters, with or without template updates. For example, if you want to allocate provisioned concurrency 8 to your lambda functions, run

braintrust install api <YOUR_STACK_NAME> --provisioned-concurrency 8

Using the AWS interface

You can also update the stack directly through AWS. Their docs walk through how to update through the console and the aws cli.

The rest of the guide covers topics only needed for advanced configurations.

Connect Braintrust's VPC to other internal resources

To permit incoming and outgoing traffic between Braintrust's lambda functions and other internal cloud resources, you can either run everything in the same VPC or setup VPC peering. This is necessary if you want to access resources like an LLM gateway or a database that is not publicly accessible.

VPC Peering

When you create your Braintrust CloudFormation, it automatically creates a VPC with the same name as your CloudFormation.

You can access the Braintrust VPC's name and ID from the CloudFormation's Outputs tab (named pubPrivateVPCID).

AWS has a comprehensive guide for configuring peering. Follow the instructions for

• Create a VPC peering connection
• Accept a VPC peering connection
• View your VPC peering connections
• Update your route tables for a VPC peering connection
  • Make sure to update the route tables in both VPCs.
• Update your security groups to reference peer security groups. We recommend allowing “All Traffic” from Braintrust’s VPC.

Troubleshooting

• If you continue to see errors after updating the VPC peering group, you may need to update your CloudFormation template (which will effectively reboot your Lambda functions). You can do this by triggering an update on the CloudFormation and letting it run. You may need to change a stack parameter and then change it back to trigger the updates.
• You can manually test network settings by booting up an EC2 machine in the Braintrust VPC to test connectivity. Make sure to assign a public IP to the instance and use the public subnet of the VPC while initializing.