FlowManner
Chat
Sign InGet Started
Flowmanner
Menu
  • Chat
  • Agents
  • Pricing
  • Docs
  • About
Products
  • Workflows
  • Templates
  • Changelog
  • Missions
Resources
  • Blog
  • API Reference
More
  • Careers
  • Contact
  • Security
  • Start
    • Quickstart
    • Your First Mission
    • Your First Eval
  • How-to
    • Connect Your Own Keys (BYOK)
    • Switch Models Per Task
    • Publish a Template
    • Run Evaluations
    • Chat with Artifacts & Previews
    • Automate via the API
    • Webhooks & Triggers
  • Concepts
    • Glossary
    • Execution Model
    • Models & Routing
    • Sandboxes & Previews
    • Memory & Privacy
    • Trust Model
  • Reference
    • Reference · API
      • Authentication
      • Missions & Runs
      • Chat & Sandboxes
      • Blueprints, Graphs & Templates
      • Evaluations & Feedback
      • Marketplace & Community
      • Files & Exports
      • Integrations & Webhooks
      • Agents & Orchestration
      • Auth, Workspaces & Billing
      • Memory & Knowledge
      • Notifications
      • Platform Services
    • Errors
    • Models
    • Limits

Your First Eval

Dataset in, scores out — measure a template instead of eyeballing it.

Why evals first

Before publishing a template or changing its prompt, run it against known cases. Evals turn "feels better" into numbers.

Steps

  1. Seed a dataset — easiest from the template itself: POST /api/v1/templates/{id}/create-dataset (evaluation.py:661), or build one manually via POST /datasets (evaluation.py:80).
  2. Add test cases with expected outcomes — bulk import helps (evaluation.py:188).
  3. Launch a run (POST /runs, evaluation.py:258). Cases execute like missions; costs apply.
  4. Read scores & costs on the run dashboard, then A/B two prompt variants with POST /compare (evaluation.py:285).

Next: publish confidently with Publish a Template.

Last updated 2026-08-25 (git-derived)

  • Why evals first
  • Steps