HomeChallengesRepsProblemsLeaderboardTake a challenge

Refuse and Evaluate

Make the thing say "I do not know" — then build the eval set that proves it still does after you change the prompt.

Advanced25 min on the clock6 graded checks · 100 pointsRAG and Model APIsPro
Graded by reading your code against the checklist below.

What you build

  • Set a retrieval floor, and refuse before calling the model when nothing clears it
  • Constrain the system prompt to the supplied sources
  • Verify every [n] in the answer maps to a source that exists
  • Write an eval set of question / expected-source pairs
  • Include a question the documents cannot answer
  • Report retrieval scores so a bad answer can be traced to a bad retrieval

Done means

An unanswerable question is refused without a model call, and every citation resolves.

How it is graded

Published in full, before you start — every point is one of these and there is nothing else.

  1. Refuse when nothing clears the retrieval floor+20Similarity search
  2. Constrain the model to the supplied sources+15Calling a model API
  3. Verify every citation resolves to a real source+20Grounding and citations
  4. Write an eval set and run it+20Rate limiting and cancellation
  5. Include a question the documents cannot answer+15Rate limiting and cancellation
  6. Report the retrieval scores+10Running it in production

What it teaches

The rest of RAG and Model APIs