A decision model that publishes where it is wrong

Kev is three models, at 0.8B, 4B and 9B, each a rank-16 LoRA adapter and a pointer head on a Qwen3.5 base. You hand it a piece of text and a set of yes/no, multiple-choice or rating questions, and it returns probabilities rather than a label. The repository publishes accuracy and Brier scores on datasets the models were not trained on: Kev-4B at 0.794 development and 0.832 test accuracy, Kev-9B at 0.812 and 0.837, against 0.857 development accuracy for the hosted Jev service the project benchmarks itself against. It also publishes where that breaks. On the new-source development set, the readme states, Kev-4B "assigns at least 0.9 probability to a wrong answer on 8.2% of questions (Kev-9B: 7.5%)". On records whose deciding evidence had been removed, Kev-9B still answers with at least 0.9 confidence 5% of the time, Jev 9%, and the previous-generation Kev-8B 26%. The evaluation data is frozen with recorded checksums and the benchmark command is in the readme, so those numbers can be rerun rather than taken. Palmer is explicit about what the comparison is not: "We don't know which datasets Jev was trained on, so this isn't a controlled comparison of the two architectures." The repository credits itself as "Built with Devin". The thread stood at 134 points and 58 comments at 07:10 EDT.

Why it matters: an LLM judge sitting inside a verification loop is a component almost nobody can characterize. This one can be. It reports a Brier score, a per-question-type calibration error, the share of decisions you could automate at a 5% error budget, and its own failure rate at high confidence, all against a frozen suite you can run on your own data. You can ask a judge like that for evidence instead of taking its word, and it fits on a 32GB Mac. The caveat is in the readme too: probabilities are poorly calibrated on sources the model has not seen, so the threshold has to be set against your own labelled examples rather than borrowed from the table.