Check: docs-grounded answer accuracy

The job

Jev judge eval (1 boolean + 1 score) over a fixed state : runnable via experimental_evaluate.

The prompt

Copy this into Muse and replace anything in [square brackets] with your own details.

// Check: docs-grounded answer accuracy — Jev judge eval (typesafe-ai/jev).
// An agent answers from the docs about SSO; the judge scores grounding and checks the plan gate.
// Model: typesafe-ai/jev via the Vercel AI Gateway ($0.042/1M input tokens).
// Auth: set AI_GATEWAY_API_KEY (vck_…) in the environment — never paste the key here.
// For zero-data-retention / no-training workloads, add
// providerOptions: { gateway: { … } } to the evaluate call below.

import { experimental_evaluate } from 'ai';

const result = await experimental_evaluate({
  model: 'typesafe-ai/jev',
  state: "Docs: SSO (SAML/OIDC) is available on Enterprise only; setup takes ~20 minutes in Settings → Security.\nAnswer: Yes — SSO with SAML or OIDC is on the Enterprise plan, and setup takes about 20 minutes under Settings → Security. Here is the doc link to follow.",
  questions: {
    "groundedness": {
      "type": "score",
      "instructions": "Score how well the answer stays grounded in the docs. Higher is better.",
      "criteria": [
        "Contradicts the docs or invents unsupported facts.",
        "Roughly right but adds details the docs do not support.",
        "Matches the docs with minor imprecise phrasing.",
        "Faithful to the docs: plan gate, protocols, time, and location all exact."
      ]
    },
    "plan_gate_stated": {
      "type": "boolean",
      "instructions": "True only if the answer states that SSO requires the Enterprise plan."
    }
  },
});

console.log(JSON.stringify(result, null, 2));