Skip to main content
Applies to:
  • Plan -
  • Deployment -

​
Summary

Issue: The Moderation AutoEval appears in the experiment scorer picker under LLM-as-a-Judge but is absent from the Select functions picker when creating an automation rule. Cause: The automation rule dropdown filters AutoEvals to only those with a prompt template (({ template }) => template); Moderation is template-free because it calls the OpenAI Moderation API directly rather than using an LLM-as-a-Judge prompt, so it is excluded. Resolution: Wrap autoevals.Moderation in a custom code scorer via the SDK; the custom scorer then appears under This project in the automation rule Select functions picker.

​
Resolution steps

​
Workaround: wrap Moderation in a custom scorer

​
Step 1: Create a custom scorer that calls autoevals.Moderation

import autoevals
import braintrust

scorer = braintrust.scorer(
    name="Moderation",
    fn=lambda output, **kwargs: autoevals.Moderation()(output=output),
)

​
Step 2: Push the scorer to your project

braintrust.login(api_key="YOUR_BRAINTRUST_API_KEY")

project = braintrust.projects.get("your-project-name")
project.scorers.push(scorer)

​
Step 3: Select the scorer in your automation rule

  1. Open your project and go to Automations.
  2. Create or edit an automation rule.
  3. In the Select functions picker, select This project.
  4. Select the Moderation scorer you pushed in the previous step.

​
Notes

  • This is a known product inconsistency. The experiment scorer picker uses a less restrictive filter that includes template-free AutoEvals like Moderation; the automation rule picker does not.
  • Other template-free AutoEvals (ListContains, ValidJSON, EmbeddingSimilarity) may be similarly absent from the automation picker. Use the same custom scorer approach for those if needed.
  • A permanent fix to relax the automation filter is tracked internally.