Decide: page the on-call or queue it

The job

Jev judge eval (1 choice + 1 score) over a fixed state : runnable via experimental_evaluate.

The prompt

Copy this into Muse and replace anything in [square brackets] with your own details.

// Decide: page the on-call or queue it — Jev judge eval (typesafe-ai/jev).
// Checkout errors spike at 2am; the judge decides the response and grades the reasoning.
// Model: typesafe-ai/jev via the Vercel AI Gateway ($0.042/1M input tokens).
// Auth: set AI_GATEWAY_API_KEY (vck_…) in the environment — never paste the key here.
// For zero-data-retention / no-training workloads, add
// providerOptions: { gateway: { … } } to the evaluate call below.

import { experimental_evaluate } from 'ai';

const result = await experimental_evaluate({
  model: 'typesafe-ai/jev',
  state: "Alert 02:14: checkout error rate 18% (baseline 0.2%) for 12 minutes, all regions, no deploy in the last 6h. Last similar alert: 3 weeks ago, self-recovered in 9 minutes.",
  questions: {
    "response": {
      "type": "choice",
      "instructions": "Decide the correct response to this alert.",
      "criteria": {
        "page_now": "Wake the on-call immediately — revenue-impacting and sustained.",
        "watch": "Monitor for 15 more minutes — could self-recover like last time.",
        "ticket": "File a ticket for business hours — no urgency."
      }
    },
    "reasoning": {
      "type": "score",
      "instructions": "Score the decision reasoning quality for this alert. Higher is better.",
      "criteria": [
        "Ignores blast radius (all regions, 18%) or time of day.",
        "Notes severity but over-indexes on the single self-recovery anecdote.",
        "Weighs severity vs history, lands defensible either way.",
        "Correct call with explicit math: 90x baseline × all regions × 12 min = page."
      ]
    }
  },
});

console.log(JSON.stringify(result, null, 2));