Head Poses and Grimaces: Challenges for automated face identification algorithms?

In today's biometric and commercial settings, state-of-the-art image processing relies
solely on artificial intelligence and machine learning which provides a high level of
accuracy. However, these principles are deeply rooted in abstract, complex "black-box
systems". When applied to forensic image identification, concerns about transparency
and accountability emerge. This study explores the impact of two challenging factors in
automated facial identification: facial expressions and head poses. The sample
comprised 3D faces with nine prototype expressions, collected from 41 participants (13
males, 28 females) of European descent aged 19.96 to 50.89 years. Pre-processing
involved converting 3D models to 2D color images (256x256 px). Probes included a set
of 9 images per individual with head poses varying by 5° in both left-to-right (yaw) and
up-and-down (pitch) directions for neutral expressions. A second set of 3,610 images
per individual covered viewpoints in 5° increments from -45° to 45° for head
movements and different facial expressions, forming the targets. Pair-wise
comparisons using ArcFace, a state-of-the-art face identification algorithm yielded
54,615,690 dissimilarity scores. Results indicate that minor head deviations in probes
have minimal impact. However, the performance diminished as targets deviated from
the frontal position. Right-to-left movements were less influential than up and down,
with downward pitch showing less impact than upward movements. The lowest
accuracy was for upward pitch at 45°. Dissimilarity scores were consistently higher for
males than for females across all studied factors. The performance particularly
diverged in upward movements, starting at 15. Among tested facial expressions,
happiness and contempt performed best, while disgust exhibited the lowest AUC
values.