Jev Confidence Explained: Thresholds, Probability & Human Review
Separate probability from confidence and choose action thresholds using labeled data, risk, and review capacity.
On this page
Probability vs confidenceWhy Noul is differentHigh, medium, and low routingThresholds depend on riskHow to test thresholdsHuman review needs useful contextLogging and common mistakesThe answer tells your code what Jev selected. Confidence helps your code decide whether it should trust that selection enough to act.
Probability vs confidence
Choice and Score provide probability distributions over options or levels. Confidence is a statistic derived from that distribution, between zero and one. Concentrated distributions generally indicate a clearer selection; diffuse distributions indicate ambiguity. A vendor probability claim is not an independent accuracy measurement on your workload.
| Answer | Probability information | Separate confidence |
|---|---|---|
| Choice | Per option | Yes |
| Score | Per level | Yes |
| Noul | Probability of yes | No |
Why Noul is different
Noul already represents a binary proposition as a probability. Read its noul field. Do not search for a missing confidence value or substitute Choice-specific logic without deciding what uncertainty means for your proposition.
High, medium, and low routing
High confidence can permit a low-risk automated action after normal authorization checks. Medium confidence might request confirmation or more evidence. Low confidence should have a safe fallback such as review. The exact boundaries belong to your application.
Thresholds depend on risk
Showing the wrong help article is reversible. Approving a money-moving operation has different consequences. Even a high-confidence judgment should not bypass verified balances, account ownership, policy checks, or confirmation for a consequential action.
0.8 is only an example threshold. Validate it against labeled data from your workflow. A single number copied from a tutorial is not a safety guarantee.
How to test thresholds
- Collect representative inputs and correct outcomes, including uncommon and ambiguous cases.
- Freeze the model version, state construction, instructions, and criteria.
- Compare decisions with labels; group errors by class, language, and confidence band.
- Sweep candidate thresholds and measure false actions, missed actions, and review volume.
- Pick a policy that fits the cost of mistakes and the capacity of the review team.
- Re-evaluate whenever a model, rubric, provider, or input distribution changes.
Human review needs useful context
Show reviewers the relevant evidence, selected answer, probability distribution, and question revision. Record their correction, not just an approval checkbox. Review backlog and delayed actions are product behavior and need an owner.
Logging and common mistakes
Log the returned model ID and structured decision without storing sensitive input unnecessarily. Monitor confidence distributions, class frequencies, and override rates. Never optimize “automatic action rate” alone: an easy way to increase it is to accept wrong answers.
If confidence stays low, investigate overlapping options, missing evidence, and poor rubrics before lowering the threshold. Confidence cannot reveal every confidently wrong decision.