When we were building the review pipeline, the easiest thing to do would have been to post everything the model found. More comments means more coverage, right? On paper, yes. In practice, it means your developers spend 20 minutes dismissing irrelevant feedback before they get to the one thing that actually matters.
We chose a different trade-off. Codzee only posts a comment if the model's confidence in that finding is 75% or above. Everything below that threshold gets silently dropped. This post explains why we landed on 75%, what we considered, and what we gave up to get there.
What "confidence" actually means in this context
When a language model analyzes a code diff, it's not producing a binary "bug / no bug" output. It's producing a probability distribution over possible interpretations of the code. A high-confidence finding is one where the model has a strong, consistent signal that something is wrong, multiple independent features of the code all point in the same direction.
A low-confidence finding is one where the model sees something that might be a problem but isn't sure, it could be intentional, it could be fine in context, or it could be a pattern the model has seen go wrong before but which is harmless here. These findings are the hardest to act on, because even when they're technically correct, they require the developer to do significant reasoning to figure out whether they apply.
What happens at different thresholds
During calibration, we ran the same set of 1,200 real PRs through the model at five different confidence thresholds and measured what happened:
The 75% threshold is where the acceptance rate crosses 90% while still posting enough to be useful. At 85% you're getting near-perfect precision, but you're also missing real bugs that the model had at 80% confidence. At 50% you're catching more, but your developers are spending more time on false positives than on real issues.
Why 75% is configurable
Different teams have different risk profiles. A fintech team processing payments has a very different tolerance for missed security issues than a startup shipping a marketing landing page. A team that's already running strong static analysis might want to raise the threshold to 85% so Codzee's comments are purely the things their existing tools can't catch. A team with no other automated review might want to lower it to 65% to get broader coverage.
We made 75% the default because it's the point where most teams we tested with said comments felt trustworthy without feeling sparse. But we expose the setting per repo so teams can move it based on their own experience.
The cost of the threshold: what we're not telling you
Here's the honest version: at 75%, we're deliberately not posting some real bugs. If the model identifies an issue at 68% confidence, it doesn't get posted. That issue might be real. We chose to accept that cost because we believe the cost of posting too much, teams losing trust in the tool, comments getting skimmed or ignored, real issues getting buried, is higher than the cost of occasionally missing a borderline finding.
That's a value judgment, not a technical one. Different teams might draw that line differently. But it's a line that should be drawn consciously, not by defaulting to "post everything and let the developer sort it out."
What happens to the sub-threshold findings
They don't disappear. Codzee stores them internally. When the model learns from dismissals and feedback, the sub-threshold findings from previous PRs are used as training signal. And when the same pattern appears at higher confidence in a future PR, it gets a score boost because it's been seen before.
The threshold isn't a trash can. It's a delay, a way of saying "we're not confident enough yet, but we're paying attention."
Want to set your own threshold?
Codzee lets you configure the confidence threshold per repo. Free during beta.