Test Robot Vision Against the Errors Your Process Cannot Accept

Which error would cause the part to be accepted when the process should have stopped it? That is the question a single "accuracy" number hides. A vision system makes several kinds of decisions, and the harm from each is different. Splitting the errors apart, with explicit denominators, is what turns a vision test into something a buyer can judge.

Name the decision and the error that matters

Before counting anything, define what the vision system decides. Good-or-bad sorting, class recognition, localization, and graspability judgement are different tasks. For the calculation below, the labels are "truly good" and "truly defective" on one side, and "auto-accept," "auto-reject" and "abstain" on the other. Two different tasks must not share one success rate.

Count false accepts and false rejects with explicit denominators

The table below is a synthetic illustration, not measured performance for AIsirRobot, a customer, or any device.

Total 1000 items; the system decided automatically on 986; of those, 970 were correct.

• False accept rate, with all truly defective items as the denominator = 4 / 200 = 2.00%.
• False reject rate, with all truly good items as the denominator = 12 / 800 = 1.50%.
• Abstention share = 14 / 1000 = 1.40%.
• Automatic correct yield across the entire trial = 970 / 1000 = 97.00%. This counts only correct automatic decisions; it does not assume that the human-resolved abstentions were correct.
• Correct share among automatic decisions only = 970 / 986 ≈ 98.38%, and this figure cannot be quoted without the condition that 14 abstentions were removed.
• Defective share inside the final auto-accepted set = 4 / 784 ≈ 0.51%, which answers a different question from the false accept rate and cannot replace it.

Read the two figures with their denominators attached: 98.38% is the correct share among automatic decisions only, and 0.51% is the defective share inside the set that was accepted. Neither is an overall accuracy for the whole 1000-item run, and quoting either one alone would misstate the result. The point of the arithmetic is that the denominator changes the answer. An unqualified "accuracy" figure usually hides which denominator was used.

Keep abstentions in the report

An abstention is a decision not to decide, and it must not disappear. Report its count and share alongside both the decided-only result and automatic correct yield across the full trial. Ask what the maker's confidence score represents and whether it has been calibrated for the relevant data. Guo and colleagues' calibration study explains why confidence and correctness probability must not simply be equated; it is not a performance finding for the proposed robot vision system.

The 14 abstentions in the example are 1.40% of the whole set. Dropping them silently would inflate the success figure and hide the workload the abstentions push to a person.

Measure localization separately from classification

A correct class does not imply a correct position. Define the coordinate frame, the ground-truth reference source, the translation and angle errors, the process window, the sample conditions and the no-result record. Results may include distributions grouped by condition and a look at the tails, but no universal error ceiling is set here. This section does not become a dimensional metrology scheme or a lens calculation.

Choose the dimensions the process actually needs; do not impose one universal metric set. A correct classification is not a localization pass.

Freeze a validation set the tuning has not seen

A report is only as honest as the samples behind it. Design an independent set across batches, surfaces, lighting, variants, difficult samples and unknown classes. Keep tuning data and validation data separate, and check the label review and the version.

Do not reuse tuning images or near-duplicates as independent validation evidence. A held-out set from the same production batch can test within-batch performance, but it does not establish generalization to unseen batches; use a batch-level holdout when that is the intended claim. Report difficult samples separately rather than averaging them away. Record the statistical method used for any uncertainty statement: zero observed errors in a finite sample do not prove zero risk.

Write an acceptance statement that preserves exceptions

The numbers land in an approvable scope. Output: task and version, sample groups, counts and denominators, abstention handling, localization distributions, anomaly records, and the conditions not covered. Compare this against the project threshold, which comes from the project rather than from this article.

Connect the results to project-level pilot acceptance criteria, and assign factory and site acceptance responsibilities. Include the relevant lighting conditions in the vision test when they affect the required decision. For a review of available manufacturer material, bring the report version, sample scope and missing fields to documentation and compliance support; AIsirRobot can coordinate those document questions without presenting them as a new test report.

For the synthetic 1,000-item trial above, the result statement is: 4 of 200 defective items were auto-accepted, 12 of 800 good items were auto-rejected, and 14 items were referred to a person. Correct automatic decisions were 970 of 986 automatic decisions and 970 of all 1,000 trial items. No human-resolution outcome or localization performance is supplied. These counts demonstrate the reporting method; approval against a real process threshold requires the project's frozen version, sample scope and acceptance criteria.