Functions

Many of the advanced capabilities of Braintrust involve defining and calling custom code functions. Currently, Braintrust supports defining functions in Javascript/Typescript and Python, which you can use as custom scorers or callable tools.

This guide serves as a reference for functions, how they work, and some security considerations when working with them.

Accessing functions

Several places in the UI, for example the custom scorer menu in the playground, allow you to define functions. You can also bundle them in your code and push them to Braintrust with braintrust push and braintrust eval --push. Technically speaking, functions are a generalization of prompts and code functions, so when you define a custom prompt, you are technically defining a "prompt function".

Every function supports a number of common features:

• Well-defined parameters and return types
• Streaming and non-streaming invocation
• Automatic tracing and logging in Braintrust
• Prompts can be loaded into your code in the OpenAI argument format
• Prompts and code can be easily saved and uploaded from your codebase

See the API docs for more information on how to create and invoke functions.

Sandbox

Functions are executed in a secure sandbox environment. If you are self-hosting Braintrust, then you must:

• Set EnableQuarantine to true in the Cloudformation stack
• Set ALLOW_CODE_FUNCTION_EXECUTION to 1 in the Docker configuration

If you use our managed AWS stack, custom code runs in a quarantined-VPC in lambda functions which are sandboxed and isolated from your other AWS resources. If you run via Docker, then the code runs in a sandbox but not a virtual machine, so it is your responsibility to ensure that malicious code is not uploaded to Braintrust.

For more information on the security architecture underlying code execution, please reach out to us.