FlowManner
Chat
Sign InGet Started
Flowmanner
Menu
  • Chat
  • Agents
  • Pricing
  • Docs
  • About
Products
  • Workflows
  • Templates
  • Changelog
  • Missions
Resources
  • Blog
  • API Reference
More
  • Careers
  • Contact
  • Security
  • Start
    • Quickstart
    • Your First Mission
    • Your First Eval
  • How-to
    • Connect Your Own Keys (BYOK)
    • Switch Models Per Task
    • Publish a Template
    • Run Evaluations
    • Chat with Artifacts & Previews
    • Automate via the API
    • Webhooks & Triggers
  • Concepts
    • Glossary
    • Execution Model
    • Models & Routing
    • Sandboxes & Previews
    • Memory & Privacy
    • Trust Model
  • Reference
    • Reference · API
      • Authentication
      • Missions & Runs
      • Chat & Sandboxes
      • Blueprints, Graphs & Templates
      • Evaluations & Feedback
      • Marketplace & Community
      • Files & Exports
      • Integrations & Webhooks
      • Agents & Orchestration
      • Auth, Workspaces & Billing
      • Memory & Knowledge
      • Notifications
      • Platform Services
    • Errors
    • Models
    • Limits

Run Evaluations

Datasets, test cases, runs, and comparisons — measure template quality instead of guessing.

  1. Create a dataset — POST /api/v1/datasets (evaluation.py:80).
  2. Add test cases individually (…/{dataset_id}/test-cases, evaluation.py:165) or in bulk (…/test-cases/bulk, evaluation.py:188). Each case pairs an input with expected outcomes.
  3. Launch a run against your template — POST /runs (evaluation.py:258). Runs execute the template per case and score outputs; costs accrue like normal missions.
  4. Compare variants head-to-head with POST /compare (evaluation.py:285) before promoting changes.
  5. Shortcuts: seed a dataset from a template via POST /templates/{template_id}/create-dataset (evaluation.py:661), import traces from Langfuse (POST /import/langfuse).

Cross-user privacy: community eval features share scores and metadata, not your raw payloads.

Last updated 2026-08-25 (git-derived)