Face-off: The Challenges of Facial Recognition Technology in an era of Cosmetic Surgical Modification

Facial Recognition Technology (FRT) is type of biometric identifier that allows for human identification via tracking physical characteristics of the face, and is being increasingly used in bureaucratic, legal, and commercial applications due to its ease of implementation and its ability to survey a large population at once (Cole, 2012). As a form of image analysis and pattern recognition, this technology compares a probe, or a facial image of interest, against a gallery of images until a threshold of similarity is met (Parks & Monson, 2008). Because facial recognition software can be executed from a distance without any physical contact, and can discretely survey a large set of information of once, it has often been employed widely despite having more technical challenges than other biometric identifiers such as fingerprints (Cole, 2012). As a result, over the past few decades facial recognition technologies and software have emerged as one of the most active fields of research in computer vision and pattern recognition (Adjabi et. Al., 2020).

Simultaneously, however, there has been a recent boom in cosmetic alterations to the face: there has been an increase of over 160% in the total number of plastic surgeries from 1997 to 2008 (Singh et. Al., 2010). In 2012, more than 14.6 million plastic surgery procedures were performed in the United States, of which almost ten million pertained only to facial treatments (Nappi et. Al., 2016). The popularity of cosmetic plastic surgery in recent decades comes from many factors, including the availability and advancement of surgical technologies, the speed with which these procedures are performed, and their increasing affordability to the mass (Singh et. Al., 2010). Consequently, there is a cat-and-mouse chase that has developed between cosmetic surgery and facial recognition technology: as the procedures behind plastic surgery become more refined in their capacity to drastically alter a person’s facial structures, the mechanisms of accurate face detection must in turn develop novel methods of identifying individuals based on the both existing and new facial landmarks. In addition to the technological challenges of developing new methods of accurate facial recognition software, the “technological race” presents an existential concern: What is the value of a face in contemporary times on how we self-identify? In an era where the face is both so malleable yet so affixed to our identity, how does this race between plastic surgery and facial recognition affect our perception of self? What is the face as an object but also from an ontological framework?

The first near-real-time facial tracking software was developed by Matthew Turk and Alex Pentland in 1991 at the Massachusetts Institute of Technology (Turk & Pentland, 1991). The design of their algorithm came from the realization that their program could exploit the fact that faces share common basic structures, called principal components. By analyzing the principal components of different faces, researchers were able to create eigenfaces — the distinguishing characteristics of each face that were revealed when similar correlations between faces were removed (Tsao & Livingstone, 2008). Modern facial recognition technology requires deep learning: the algorithms mimic the neural networks akin to human cognition and use successively complex layers of information to become increasingly specific and accurate at recognizing facial features (Martinez, 2017). These programs often start by creating a template of the target’s face, which is created by measuring facial landmarks called nodal points — characteristics like the distance between the eyes or the width of the nose are often used (Hamann & Smith, 2019). The template is then compared to a database of faces for a match: a threshold of successive verifications in extracted facial features must be made for a positive identification. However, the challenges in current facial recognition technology stems from the fact that these facial features used in positive identifications are the very ones often modified in cosmetic plastic surgery.

Plastic surgery is defined as a medical alteration which restructures human body parts (Sabharwal & Gupta, 2019). The type of plastic surgery can largely be defined as global or local when transforming facial characteristics. Plastic surgery that is local in nature is texture-based; that is, it is often used to rectify defects and improve skin consistency, and any surgical treatment is performed locally on a region (Sabharwal & Gupta, 2019). Procedures that are global in nature, on the other hand, are structure-based: surgery involves entire variation of facial characteristics and outward manifestation of a person (Sabharwal & Gupta, 2019). As a result, in global plastic surgery, facial recognition technology poses a challenge since the entire face geometry has been altered. The ability to evade facial recognition and detection has profound implications in government, industrial and commercial spaces: a group of women were stopped at Hongqiao International Airport’s customs after undergoing plastic surgery in South Korea because their new faces were unrecognizable compared to the pictures on their passports (Shanhaiist, 2017). On a more sinister level, however, plastic surgery provides an opportunity for a person to conceal their identity with the intent to commit fraud or evade law enforcement (Singh et. al., 2010).

Research by Singh et. al., in 2010 showed that current models of facial recognition were unable to handle plastic surgery classified as global features, as they altered the nodal points of the face necessary to match an identity pre- and post-operation (Singh et. al., 2010). In analyzing faces before and after surgery, two critical questions about their classification need to be addressed: whether one can consider an altered face to be an intra-class variation of the original face, and whether these changes cause the new face to be closer to the original face than another one (Nappi et. al., 2016). There have been methods to create algorithms for specific regions of the face, but the computational time for matching a full face renders them inconvenient and not practical for real-time use. Some of the more promising research takes advantage of the fact that there are few features on the face which remain unaltered during facial plastic surgery and are not easily doctored (Dadure et. al., 2018). Research by Sabharwal and Gupta showed that combining regional cues based on the local textures of the face with more robust Principal Component Analysis (PCA) yielded promising recognition for both local and global surgical treatment; however, the percentage of positive matching was still only 87% (Sabharwal and Gupta, 2019). As such, there is still much advancement necessary in facial recognition technology to accurately and consistently identify faces that have undergone facial plastic surgery.

The implications for improving facial recognition technology and the algorithms for image recognition in light of expanding plastic surgery go beyond worries of sinister and fraudulent activities: the association between our face and our identity has been solidified by social constructs, cultural influences, and the institutions we live in (Cole, 2012). An investigation of faces invariably leads to the concept of self; researchers and theorists have long argued the important of the face to self-identity, and the extent to which the face of an individual is a public or private phenomenon, situation-specific or context-independent (Spencer-Oatey, 2007). Simon (2004) identifies a number of functions of identity; that identity helps to provide people with a sense of belonging and uniqueness; it helps people locate themselves in their social worlds; and many facets of identity help provide people with positive self-evaluations. But where does the face tie into our identity? Many differentiate identity and the face as individual versus relational; that is, the identity is contained within an individual, whereas the face is a relational phenomenon (Spencer-Oatey, 2007). As Goffman (1967) argues “the term face may be defined as the positive social value a person effectively claims for himself by the line others assume he has taken during a particular contact.” As a result, while face and identity are both similar in their contribution to the “self-image” and having multiple attributes, the face is only associated with positively evaluated attributes that the individual wants others to acknowledge (and conversely, with negatively evaluated attributes that the individual does not want others to label to themselves (Spencer-Oatey, 2007)).

The relationship between a person’s face and their sense of self has long been explored by artists and activists, who contemplate not only the social but also political implications of tying our face with our identities. Artist and specialty mask maker Kamenya Omote produces three-dimensional face masks of strangers’ faces to sell (Hyperallergic, 2020). These realistic masks were created by printing onto a three-dimensional model created from the model’s face. Omote’s “That Face” series tunes into society’s preoccupation and obsession with our likeness, and creates masks in an era where you can, through plastic surgery, literally buy and sell a face. He sees the long-standing implication of the developing technologies that enable us to create an identical replication of someone’s face: he said in an interview with Hyperallergic (2020) that “if face replication becomes common, it would be interesting to be able to save a face from your youth, for example, or be able to change your face at will. But I’m more interested in what happens to people’s bodies, because the face and the body are inseparable.” Likewise, industrial designer Jing-Cai Liu developed a wearable projector that projects different faces onto one’s own face as a response to the use of mass surveillance by the retail and advertisement industries in tailoring targeted marketing towards customers they track. The project explored themes of the dystopian future, provoking debates about the “emerging future” and creating solutions to maintaining the privacy of our identities (Liu, 2017). By superimposing another person’s face on top of your own, Liu claims their identity is guarded and the wearer is protected from “privacy violations” (Liu, 2017).

Although Omote and Liu are concerned with hiding the actual physical characteristics of the face, Adam Harvey looks to avoid facial detection software by altering the very landmarks of the face often modified in plastic surgery (Harvey, 2010). Particularly in the United States, there has been an increase in surveillance in society over the past few decades, and these algorithms are noted for their bias towards the Black and brown communities (Benjamin, 2019). As a result, many activists and artists have sought to use CV Dazzle as a form of protection against the surveillance state that they see their countries becoming. The inspiration behind CV Dazzle, or Computer Vision Dazzle, came from a type of WWI ship camouflage called Dazzle that broke the visual continuity of a battleship with blocks of forms and patterns to conceal the ship’s orientation and size to enemies (Harvey, 2018). By drawing striking and bold patterns on the face, CV Dazzle aims to confuse facial recognition software by altering the expected dark and light areas of the face that most computer algorithms look for. Similarly, Dutch designer Jip van Leeuwenstein’s “Surveillance Exclusion” project is a series of fitted clear masks that are formed like a lens (van Leeuwenstein, 2016). These masks are able to fracture the image of the face to avoid certain facial recognition software from detecting them — as van Leeuwenstein says “mega databanks and high resolution cameras stock hundreds of exabytes a year. But who has access to this data? / By wearing this mask formed like a lens it is possible to become unrecognizable for facial recognition software.” However, he understands the significance of our face in forming an identity: the masks are transparent so that all the facial features, while distorted, can still be visible to those around the wearer. Although, like CV Dazzle, this work renders the user more conspicuous, van Leeuwenstein’s work maintains to a certain degree the integrity of the wearer’s face so that they will “not lose [their] identity and facial expressions. (van Leeuwenstein, 2016)”

We now have the ability to change our identity by the alteration of our faces. In an era of surveillance states, where our faces are being turned into scannable barcodes, this is a way to avoid current methods of tracking individuals’ activities/movements and locations. But for many people on a personal level, it’s a way to start over; to change aspects of themselves they find undesirable and recreate their ideal vision of themselves. The idea of making oneself anew has interesting implications: it asserts the necessity of the face in our current society as one of the most important markers of someone’s identity. As a result, is plastic surgery a method of making oneself invisible or developing a new identity entirely?

Although the artists mentioned above are all concerned with the increasing role surveillance technology has on our lives, and the implications that has for our own relationship with our faces, their approach to tackling this problem comes down to the matter of invisibility. Omote and Liu’s works are an attempt at invisibility; that is, to cover their faces entirely as to render it mysterious and avoid being tracked. However, Harvey’s approach with CV Dazzle is the exact opposite: to confuse facial detection by being ironically more visible outside in modern society. Indeed, in the case of both CV dazzle and the original Dazzle camouflage, their intention in developing the elaborate and bold patterns is not to hide but rather be un-trackable. In the case of cosmetic surgery, however, procedures that change the face will make the individual remain invisible in society while disrupting facial detection — but at the cost of their former identity.

For bureaucratic, commercial and governmental frameworks, our faces are a number — in the case of Apple’s new FaceID technology, scans of the user’s faces are processed and stored as numerical data as templates for comparing against new faces. But the link between our faces and our identity has been cemented throughout history, civilizations, and through rituals. And the process of facial modification — whether it is permanent, in the case of cosmetic surgery, or temporary in the case of make-up and props — is largely an attempt at changing the way we look in hopes that it will change some aspect of ourselves that is more intangible. There is a duality that exists as members of contemporary society, holding fiercely to our individuality (and our image of self) while engaging in uniformity (everyone using their face as biometric tools). What is the value of a face when it becomes a number? As the reliance on the face as an accurate biometric identifier gains widespread acceptance, it is important to consider the implications this notion has on our sense of self and how we navigate our identities in society.

Citations:

Adjabi, I., Ouahabi, A., Benzaoui, A., & Taleb-Ahmed, A. (2020). Past, Present and Future of Face Recognition: A Review. Electronics, 9(8), 1–52. Retrieved from https://www.mdpi.com/2079-9292/9/8/1188/htm

Ali, A.S.O., Sagayan, V., Malik, A. and Aziz, A. (2016), Proposed face recognition system after plastic surgery. IET Comput. Vis., 10: 344–350. https://doi.org/10.1049/iet-cvi.2014.0263

Benjamin, R. (2019). Race after Technology. Medford, MA: Polity Press

Harvey, Adam. (2010) CV Dazzle. https://cvdazzle.com/

Harvey, Adam (2018) CV Dazzle, Camouflage from face-detection https://ahprojects.com/cvdazzle/

Hyperallergic (2020). Japanese Shop Sells Hyperrealistic 3-D printed face masks. https://hyperallergic.com/610180/japanese-shop-sells-hyperrealistic-3d-printed-face-masks/

Cole, S. A. (2012). The face of biometrics. Technology and Culture, 53(1), 200–203. Retrieved from https://gold.idm.oclc.org/login?url=https://www-proquest-com.gold.idm.oclc.org/scholarly-journals/face-biometrics/docview/929039973/se-2?accountid=11149

Dadure, P., S. Sikder and K. Sambyo, “Plastic Surgery Face Recognition Using LBP and PCA Algorithm,” 2018 2nd International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech), Kolkata, India, 2018, pp. 1–5, doi: 10.1109/IEMENTECH.2018.8465379.

Goffman, Erving, 1967. Interaction Ritual. Essays on Face-to-Face Behaviour. Pantheon, New York.

Hamann, K., & Smith, R. (2019). Facial recognition technology. Criminal Justice, 34(1), 9–13. Retrieved from https://gold.idm.oclc.org/login?url=https://www-proquest-com.gold.idm.oclc.org/trade-journals/facial-recognition-technology/docview/2246857277/se-2?accountid=11149

Jones, W.D. “Plastic surgery 1, face recognition 0,” in IEEE Spectrum, vol. 46, no. 9, pp. 17–17, Sept. 2009, doi: 10.1109/MSPEC.2009.5210052.

Liu, Jing-Cai. Wearable Face Projector (2017). Retrieved from http://jingcailiu.com/wearable-face-projector/

Martinez A. M. (2017). Computational Models of Face Perception. Current directions in psychological science, 26(3), 263–269. Retrieved from https://doi.org/10.1177/0963721417698535

Nappi, M., Ricciardi, S. and Tistarelli, M. (2016). Deceiving faces: When plastic surgery challenges facial recognition. Image and Vision Computing. 54: 71–82. https://doi.org/10.1016/j.imavis.2016.08.012

Parks, C. & Monson. K. (2018). Recognizability of computer-generated facial approximations in an automated facial recognition context for potential use in unidentified persons data repositories: Optimally and operationally modeled conditions. Forensic Science International, 291, 272–278. Retrieved from https://doi.org/10.1016/j.forsciint.2018.07.024

Sabharwal, T., and Gupta, R. (2019), Human identification after plastic surgery using region based score level fusion of local facial features. Journal of Information Security and Applications. 48. https://doi.org/10.1016/j.jisa.2019.102373

Simon, Bernd, 2004. Identity in Modern Society. A Social Psychological Perspective. Blackwell, Oxford.

Singh, R., M. Vatsa, H. S. Bhatt, S. Bharadwaj, A. Noore and S. S. Nooreyezdan, “Plastic Surgery: A New Dimension to Face Recognition,” in IEEE Transactions on Information Forensics and Security, vol. 5, no. 3, pp. 441–448, Sept. 2010, doi: 10.1109/TIFS.2010.2054083.

South Korean Plastic Surgery Trips Headaches for Customs Officers, [online at Shainghaiist] Available: http://shanghaiist.com/2009/08/04/south_korean_plastic_surgery_trips.php.

Spencer-Oatey, H. (2007). Theories of identity and the analysis of the face. Journal of Pragmatics, 39, 639–656.

Tsao, D. Y., & Livingstone, M. S. (2008). Mechanisms of face perception. Annual review of neuroscience, 31, 411–437. https://doi.org/10.1146/annurev.neuro.30.051606.094238

Turk, M. & Pentland, A. (1991). Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1), 71–86. Retrieved from https://doi.org/10.1162/jocn.1991.3.1.71

Van Leeuwenstein, Jip (2016). Surveillance Exclusion. http://www.jipvanleeuwenstein.nl/