Python scorer fails with an import error on self-hosted deployments

Applies to:

Plan - Enterprise
Deployment - Self-hosted (GCP, Azure)
Use case - Running Python scorers with native dependencies on a self-hosted data plane

Summary

Issue: A Python scorer fails with an import error when you invoke it on a self-hosted GCP or Azure data plane. One example is cannot import name 'cygrpc' from 'grpc._cython'. Cause: The scorer bundle contains a native Python extension built for a different Python version or CPU architecture than the API runtime that executes the scorer. Resolution: Build and push the scorer with a Python interpreter and architecture that match the self-hosted API runtime.

Symptoms

The scorer pushes successfully, but invocation returns an HTTP 500 error. The error includes details similar to:

python=3.11.x platform=linux grpc import failed: cannot import name 'cygrpc' from 'grpc._cython'

The bundle can contain the expected native extension and still fail. For example, a bundle containing cygrpc.cpython-312-x86_64-linux-gnu.so cannot load that extension in Python 3.11. This issue affects packages with compiled extensions.

Cause

On self-hosted GCP and Azure deployments, Python scorer code runs in the API container. The Python version is determined by the OS image. The Python interpreter used by bt functions push determines which wheel or native extension is placed in the scorer bundle. The --runner option controls the interpreter used to build the bundle. It does not change the interpreter in the API container. If you build with Python 3.12 and invoke on a Python 3.11 API runtime, Python looks for a cpython-311 extension. A cpython-312 extension does not load. CPU architecture must also match. For example, an x86_64 extension does not load in an aarch64 runtime.

Resolution steps

Step 1: Check the API runtime

Check the Python version and architecture in an API pod:

kubectl exec -n <namespace> deploy/braintrust-api -- python3 --version
kubectl exec -n <namespace> deploy/braintrust-api -- python3 -c \
  'import platform; print(platform.machine())'

If your deployment uses separate API workload pools, run the commands against the pod that handles function invocation.

Step 2: Check the bundle build interpreter

Run these commands from the scorer’s development environment:

<path-to-python> --version
<path-to-python> -c 'import platform; print(platform.machine())'

The Python major and minor version and CPU architecture must match the API runtime. Patch versions within the same Python major and minor version are generally compatible, but use the same patch version when possible.

Step 3: Recreate the environment with the matching Python version

Create a virtual environment with the Python version used by the API runtime. For example, if the API runtime reports Python 3.11:

python3.11 -m venv .venv311

Install the scorer’s dependencies using that environment and use the same interpreter to push the function:

.venv311/bin/python -m pip install -r requirements.txt

bt functions push scorer.py \
  --requirements requirements.txt \
  --if-exists replace \
  --no-input \
  --runner .venv311/bin/python

If the API runtime uses a different architecture than your development machine, build the bundle in a matching environment, such as a container or CI runner for that architecture.

Step 4: Invoke the scorer again

bt functions invoke <scorer-name> \
  -p <project-name> \
  --input '{"output":"hello","expected":"hello"}'

How to confirm it worked

The scorer should invoke without the native-extension import error. To verify the exact versions used by the scorer, temporarily include diagnostic information in the scorer output or error:

import platform
import sys

runtime = (
    f"machine={platform.machine()} "
    f"python={sys.version} "
    f"platform={sys.platform}"
)

try:
    import grpc
    from grpc._cython import cygrpc

    details = f"{runtime} grpc={grpc.__file__} cygrpc={cygrpc.__file__}"
except Exception as error:
    details = f"{runtime} grpc import failed: {error}"

raise RuntimeError(details)

The Python version and architecture reported by the scorer must match the API runtime. Remove diagnostic code after troubleshooting.

Notes

This issue is specific to native dependencies. Pure-Python packages generally do not contain interpreter-specific .so extensions.
AWS Lambda deployments use separately selected Lambda runtimes and do not use this Kubernetes API-container execution path.

​Summary

​Symptoms

​Cause

​Resolution steps

​Step 1: Check the API runtime

​Step 2: Check the bundle build interpreter

​Step 3: Recreate the environment with the matching Python version

​Step 4: Invoke the scorer again

​How to confirm it worked

​Notes

Summary

Symptoms

Cause

Resolution steps

Step 1: Check the API runtime

Step 2: Check the bundle build interpreter

Step 3: Recreate the environment with the matching Python version

Step 4: Invoke the scorer again

How to confirm it worked

Notes