Discuss a project
All insights

Technical Notes

Does the detector actually help the reviewer?

Evaluate the verified findings, missed issues and the work that still lands on the analyst.

Reference from 2026: NIST: Expanding the AI Evaluation Toolbox

Aerial location photography
Illustrative location photography

Decide what counts as one finding

An obstruction can span several tiles. Counting each tile as a separate alert may inflate both apparent performance and reviewer workload. Decide whether the unit is an event, asset, segment or site.

Specify how overlapping flags will be merged. Report coverage failures separately rather than scoring unassessable ground as a correct negative.

Build an evaluation the team can trust

Use independently reviewed examples, including ordinary sites with no issue and captures that cannot answer the question. Separate sites or dates where nearby imagery is likely to look alike. Decide what counts as a match before seeing the model output. Keep disputed labels visible rather than forcing them into a false certainty.

Example: a repeat inspection program

Compare assisted and manual review on comparable work. Count confirmed findings, missed findings and rejected flags. Measure how long it takes to reach a defensible decision, not just how quickly the model produces a list. If the assisted workflow finds more real issues but overwhelms the team with duplicates, tune the handoff and retest. Review missed events with the people affected by the decision.

Find the cause before changing the threshold

Was the area missing from the capture? Did the detector miss a visible feature? Did the reasoning step retrieve the wrong record? Those failures need different fixes.

Keep the same outcome definitions as sites and capture conditions change. Otherwise a later evaluation may not be comparable with the first.

Practical checklist

  • Define the counting unit and acceptable coverage.
  • Audit flagged and unflagged areas with independent reviewers.
  • Record reviewer time, correction reasons and the consequence of missed findings.

Sources and further reading

NIST: AI Risk Management Framework USGS: Guidelines for calibration of uncrewed aircraft systems imagery