Facebook Facial Recognition: Its Quiet Rise and Dangerous Future

In early April, Engadget posted a short article confirming a rumor that Facebook would be using facial recognition to suggest the names of friends who appeared in newly uploaded photos. You’d be allowed to opt out of tagging, and only friends would be able to tag each other in albums. Nevertheless, a commenter beneath the story quipped, “Awesome! Now I can take pictures of cute girls at the grocery store or at the park, upload them and Facebook will tell me who they are! (I’m pretty sure that’s not [how] it works but I’m sure it will get there.)”

The commenter’s confidence says a lot: Facial recognition may be just one more way for Facebook to push the visual part of the social graph (photos of us) toward being more public and far less private. Facebook has a history of asking for forgiveness after the fact instead of asking for permission in advance, and its new face-recognition feature could become the latest example of a seemingly innocuous development morphing into a serious threat to the privacy of our (visual) data. And as usual, some Facebook users will like the convenience of the new features so much that they will forget the privacy trade-off altogether, or just choose not to worry about it.

Features You Didn’t Know You Had

As it stands, Facebook’s current feature uses facial recognition technology to pick out faces in your photos. Once you’ve uploaded your album, Facebook will take you to a new screen where you can enter the name of each person below their face. Sometimes (depending on your privacy settings and the clarity of the photo), Facebook will go a step further: If a face matches one you previously tagged in another album, Facebook may suggest that person’s name for you. Facebook quietly added the feature to the Privacy Settings, allowing users to disable the peppy-sounding ‘Suggest photos of me to friends’ option. Most Facebook users probably don’t know that the extra privacy setting is there.

Technological advances in the last 10 years are making it possible for computers to match images and names with impressive accuracy. Though every company using the technology handles it a little differently, the president of Applied Recognition, Ray Ganong, shared some insight into how his company’s product Fotobounce gets the job done: “We scan each image as a bitmap and look for potential face images that qualify. We try to see the two eyes, and based on the eye location we reorient the face and then generate a digital signature, based on that face.” Many builders of facial recognition technology base their matches on “faceprints” of people, where an engine synthesizes information using many photos of the same person from different angles or with different lighting to make a more accurate match. Given that Facebook users had uploaded 60 billion photos by the end of 2010, the prospects for accurate facial recognition on the social network are better now than ever before.

Facial recognition in a social networking context is not particularly new. Third-party app builders have been offering face detection on Facebook since Face.com entered the scene in 2009 with its Photo Finder app, which scanned thousands of photos to find images in which the user appears but isn’t tagged. But the difference between third-party apps and Facebook’s new recognition feature is that the former have always required participants to actively opt in to the feature, whereas at Facebook the feature is turned on by default and requires the user first to learn that it’s in use, and then to expressly opt out. Even then, Facebook’s servers don’t lose the information they’ve acquired for associating your face with your name. They just comply with your request not to use it for the time being.

Despite the service’s need to make users feel at ease about these changes, some comments from Facebook’s management over the past few months have been confusing and a little defensive, adding to the impression that the company is easing in a feature that could generate negative reactions later. In September 2010, Facebook revealed that it would recognize and group similar faces together. During a public announcement regarding the new features, Sam Odio, the newly hired product manager of Facebook Photos, said “This isn’t face recognition […] Picasa and iPhoto–they’ll detect a face and say, ‘This is Sam,’ and they’ll suggest that it’s Sam. We’re not doing that. We’re not linking any faces to profiles automatically. Right now, we want to stay away from that because it’s a very touchy subject.” Apparently the subject wasn’t quite so touchy four months later, when Facebook started suggesting the names of friends in uploaded photos.

Some might argue that the facial recognition tagging feature actually gives users more privacy by increasing their chances of being tagged, and in that way discovering where their image is appearing and how it’s being used. But for some, the worry is less about how friends might use your photos and more about how Facebook could use your information–and give others access to it. Even if you choose to disable the ‘Suggest photos of me to friends’ option, Facebook will still have the technical ability to connect your name with your image. And even when Facebook doesn’t suggest the name of your friend, picking out a face and asking you to tag it is essentially the same thing as offering the name of your friend, except that it enlists you as a participant in the process. “Facebook is being really clever about it […] they’re not assigning names with it, but the minute you assign a name to it you’ve completed the recognition,” says Marisol MacGregor, head of marketing at Viewdle, a company that specializes in making lightweight facial recognition technology.

Safe Now, But What’s Coming?

In the hands of smaller developers like Viewdle and Fotobounce, which keep little if any personal information on their company’s servers, face recognition could be minimally worrisome. But in the hands of Facebook, which sits on a monster database filled with dense detail about the personal lives of more than 500 million people, the technology has the potential to be creepy.

Of course, as far as we know, the company is not going any farther with its current technology than suggesting that you tag people you are already friends with in newly uploaded photos. But could Facebook ever identify people you’re not friends with and suggest that you become friends with them? “Absolutely, it would be easy to do. All that data would be on that server farm. Technically, it’s totally possible to expand that,” says Applied Recognition’s Ganong.

It’s not hard to imagine Facebook’s “Suggest photos of me to friends” privacy setting becoming “Suggest photos of me to friends of friends” and then “Suggest photos of me to others”–essentially allowing you to take photos of strangers on the street and request a friendship. No other company except Google could realistically offer a feature that tells you the name of a complete stranger you’d seen in the park or at a concert. In 2008 the company offered face recognition on Picasa, Google’s photo-sharing site, but it recently removed face-finding tech from its Google Goggles app until privacy issues could be resolved. It turns out that algorithmically finding faces is cool, but sharing faces can get scary.

Though Google is a giant company, one reason why its facial recognition feature in Goggles may not work is that Google doesn’t have the kind of built-in emphasis on friendship and sharing that Facebook has. Facebook’s ubiquity, and its financial interest in getting people to connect, makes it perhaps the only company in the world that could roll out the ability to recognize strangers, and get users to accept it. Some people might relish the idea of being spotted and of making connections with new people, but being able to identify a face with nothing more than a camera could have serious adverse consequences. “Facial recognition is especially troubling because cameras are ubiquitous and we routinely show our faces. And of course, one can take pictures of crowds, so it scales a bit better than, say, fingerprints,” says Lee Tien, Electronic Frontier Foundation senior staff attorney, via e-mail.

Misidentification is another problem. Gil Hirsch, CEO of Face.com, says that his company set up a very high threshold of recognition to confirm face matches on its Photo finder app. “We don’t want to send you a message saying ‘Hey Megan we found a photo of you’ and it’s not really you,” he explained. But that threshold of recognition will be different with every system, Facebook’s included. Nevertheless, better and faster algorithms are slowly whittling down the likelihood of erroneous identifications. Compared to being accurately identified to a stranger, misidentification may register as a lesser concern. Tien of EFF notes, “If Facebook misidentifies someone, the consequences are not the same as when a police video-camera misidentifies you as a suspect.” True, unless a misidentification implicates you in dubious activities. The imagination reels.

From a business perspective, it’s important to Facebook that its users tag themselves and each other in as many photos as possible. These tags create more page views, which is valuable to Facebook’s advertisers. But it could go much further. If you are tagged in a photo with three friends, advertisers could tailor information to what they think you might want based on your friend’s preferences. Though perhaps not at the level of an infringement of legal privacy rights, facial recognition in the hands of Facebook does permit advertisers an unprecedented level of information about how to get a message across to you.

Facial recognition is a cool technology that Facebook is using to add more convenience to the act of tagging people in photos. The technology may indeed create more connections between friends, and so far it seems to pose little real threat to privacy–because for now it’s all among friends. But that could change. If you are uncomfortable with facial recognition, pay a visit to your Facebook privacy settings and opt out of the feature. In the broader view, it’s important that we all keep a close eye on Facebook’s use of this powerful technology, and that we let tech privacy groups and lawmakers know if the technology is being abused to enrich social networking sites and their advertisers, at the expense of our privacy.