Facial recognition systems have an endless appetite for training data. But using real people’s faces without their consent raises ethical and privacy concerns. Synthetic faces help, but even those are often based on the biometrics of real people. Faces generated using a graphics pipeline and no real data – such as those contained in Microsoft’s open source DigiFace-1M dataset – don’t look real enough.
How to feed the facial recognition machine without accidentally making privacy rights part of the meal? This is the question researchers from the Idiap Research Institute set out to answer with their project, “Digi2Real: Bridging the Realism Gap in Synthetic Data Face Recognition via Foundation Models.”
The work introduces a “novel framework for realism transfer aimed at enhancing the realism of synthetically generated face images” using a large-scale face foundation model. It starts with the DigiFace-1M dataset, a collection of over one million diverse synthetic face images for facial recognition.
“By integrating the controllable aspects of the graphics pipeline with our realism enhancement technique, we generate a large amount of realistic variations – combining the advantages of both approaches,” says the abstract. “Our empirical evaluations demonstrate that models trained using our enhanced dataset significantly improve the performance of face recognition systems over the baseline.
The resulting Digi2Real synthetic face dataset contains images of 20,000 unique synthetic identities, with the realism transfer technique applied to procedurally generated identities from a graphics pipeline “to produce photorealistic images, which are more effective for training face recognition models than the original DigiFace dataset.”
The process involves “interpolating between multiple images of an identity within the embedding space,” then using the pre-trained Arc2Face model to “synthesize identity-consistent images from these interpolated embeddings.” It then further enhances them by reducing the domain gap in the intermediate CLIP encoder space.
“By combining the controllable features of the graphics pipeline with our realism enhancement technique, we present a new approach for creating attribute-controllable face recognition datasets,” the researchers say. And they have numbers to back it up: their testing shows that “Face Recognition performance with Digi2Real dataset significantly improves over the DigiFace and achieves better performance than many other synthetic datasets.”
The Digi2Real dataset, containing 399,355 images of 20,000 unique individuals, is publicly available.
Article Topics
Arc2Face | biometrics | biometrics research | dataset | Digi2Real | facial recognition | Idiap | synthetic data | synthetic faces
