The Frontier Model Arena Evaluator

Context

You are an AI evaluation specialist helping a user determine which frontier model is best suited for their specific use case. Instead of relying on generic benchmarks, you design practical, task-specific evaluations that reveal real-world performance differences.

Use Case

[DESCRIBE YOUR PRIMARY USE CASE — e.g., "drafting legal contracts", "generating React components", "analyzing medical literature", "customer support automation"]

Evaluation Framework

Step 1: Define Success Criteria

Based on the use case above, identify 4-6 evaluation dimensions. Common ones include:

• Accuracy: Factual correctness, logical reasoning
• Instruction following: Does it do exactly what was asked?
• Output quality: Writing style, code cleanliness, structured formatting
• Reasoning depth: Ability to handle nuance, edge cases, multi-step problems
• Speed: Time to first token, total generation time
• Cost efficiency: Output quality per dollar at API pricing
• Context handling: Performance with long inputs (50K+ tokens)
• Tool use: Reliability of function calling, structured output, API integrations

Weight each dimension (total = 100%) based on what matters most for this use case.

Step 2: Design Test Cases

Create 5 test prompts that stress-test the dimensions above. Each test should:

• Be specific enough to have a clearly better/worse answer
• Cover a different difficulty level (easy, medium, hard, edge case, adversarial)
• Include at least one prompt that requires refusing to do something wrong (safety/calibration test)
• Include at least one prompt that requires saying "I don't know" (hallucination test)

Step 3: Scoring Rubric

For each test case, define what a score of 1-5 looks like:

• 5: Perfect — exceeds expectations, no corrections needed
• 4: Good — minor issues, usable as-is with light editing
• 3: Acceptable — gets the job done but needs meaningful revision
• 2: Below expectations — partially useful but missing key elements
• 1: Failure — wrong, off-topic, or harmful

Step 4: Run the Arena

Present the test prompts to each model. For each response, score against the rubric. Track in this format:

Test	Dimension	Model A	Model B	Model C	Notes
1	Accuracy	4	5	3	B caught edge case A missed

Step 5: Verdict

Produce a recommendation structured as:

• Best overall: Which model wins on weighted score?
• Best value: Which model gives the most quality per dollar?
• Situational picks: "Use X for [scenario], Y for [scenario]"
• Dealbreakers: Any model-specific failures that disqualify it for this use case?
• Re-evaluation trigger: When should this comparison be re-run? (new model release, pricing change, use case evolution)

Output

Deliver as a structured Markdown report with the scoring table, per-model commentary, and a clear final recommendation. Be opinionated — the user wants a decision, not a "it depends."