The report from BixeLab on accuracy differentials in New Zealand’s biometric identity verification platform for public service access has been published, showing the system is not biased, but does have slightly higher false positive rates for Māori and Pasifika people.
The IdentityX cloud platform and liveness detection software were found in the “Level B Evaluation Assurance — Identity Check Bias Evaluation” to have “no significant bias” in liveness or matching performance “among the tested ethnic groups.” Those groups include Māori, Pasifika and “other” New Zealanders. The initial findings of the report were revealed last month.
New Zealand’s app did not wrongly classify any genuine live submissions as presentation attacks, for a 0 percent Bona Fide Presentation Classification Error Rate (BPCER) across all participant groups. The rejection rate was 0.7 percent, and did not significantly vary between any groups.
The biometric matching engine was found to be effective for all groups, with an overall false match rate (FMR) of 0.09 percent and a false non-match rate (FNMR) of 0.9 percent was noted. FMR was slightly higher for Māori and Pasifika participants in the back-end test, however.
A review by the Department of Internal Affairs suggests that this is partly caused by problems with image quality.
The liveness detection check worked in 87 percent of selfie checks, but all participants were able pass the check within three tries. The system matched subjects’ face biometrics successfully in 162 of 161 attempts, with a single attempt by a roughly middle-aged male non-indigenous user delivering a non-match.
The overall positive finding “is supported by the Chi-Squared Test conducted at a 95 percent confidence level, which showed consistent performance between Māori and non-Māori (‘Others’). However, the Kolmogorov-Smirnov (KS) Test did identify some variations in the underlying distributions across the tested groups. While these variations do not impact on the user experience, they suggest that any future adjustments to system configurations should be made cautiously to ensure continued equitable performance,” the report says.
The test report explains the scope of the evaluation and how it was conducted in detail, and breaks down the demographics of participants.
Daon, as the face biometrics provider for the Identity Check service, provided face image quality metrics and metadata from its matching subsystem.
The test involved selfies submitted by 148 people, but did not include any subjects with Tā moko, or facial tattoos.
Article Topics
accuracy | biometric liveness detection | biometric testing | biometric-bias | biometrics | Daon | demographic fairness | face biometrics | Identity Check | New Zealand
