Grade: SQL answer correctness

The job

Jev judge eval (1 score) over a fixed state : runnable via experimental_evaluate.

The prompt

Copy this into Muse and replace anything in [square brackets] with your own details.

// Grade: SQL answer correctness — Jev judge eval (typesafe-ai/jev).
// An agent writes SQL for monthly revenue by plan; the judge scores the query.
// Model: typesafe-ai/jev via the Vercel AI Gateway ($0.042/1M input tokens).
// Auth: set AI_GATEWAY_API_KEY (vck_…) in the environment — never paste the key here.
// For zero-data-retention / no-training workloads, add
// providerOptions: { gateway: { … } } to the evaluate call below.

import { experimental_evaluate } from 'ai';

const result = await experimental_evaluate({
  model: 'typesafe-ai/jev',
  state: "Question: monthly revenue by plan for 2026, excluding refunded orders.\nAnswer: SELECT date_trunc('month', created_at) AS month, plan, SUM(amount) FROM orders WHERE status != 'refunded' AND created_at >= '2026-01-01' GROUP BY 1, 2 ORDER BY 1, 2;",
  questions: {
    "correctness": {
      "type": "score",
      "instructions": "Score this SQL answer against the question. Higher is better.",
      "criteria": [
        "Wrong grain, wrong filter, or would not run.",
        "Runs but drops a requirement (refunds, year bound, or grouping).",
        "Correct results with a minor style or edge-case flaw.",
        "Correct, complete, and robust to edge cases (nulls, timezones, refunds)."
      ]
    }
  },
});

console.log(JSON.stringify(result, null, 2));