face-recognition-phone

Samsung’s and Apple’s Face Recognition Technologies and How To Fool Them

Posted on by Zbigatron

In September 2017 Apple announced iPhone X with a very neat feature called Face ID. This feature is used to recognise your face to allow you to unlock your phone. Samsung, however, has had facial recognition since the release of Android Ice Cream Sandwich way back in 2011. What is the difference between the two technologies? And how can either of them be fooled? Read on to find out.

Samsung’s Face Recognition

Samsung’s Face Unlock feature works by using the regular front camera of your phone to take a picture of your face. It analyses this picture for facial features such as the distance between the eyes, facial contours, iris colour, iris size, etc. This information is stored on your phone so that next time you try to unlock it, the phone takes a picture of you, processes it for the aforementioned data and then compares it to the information it has stored on your phone. If everything matches, your phone is unlocked.

The only problem is that all processing is done using 2D images. So, as you may have guessed, a simple printed photo of your face or even one displayed on another phone will fool the system. Need proof? Here’s a video of someone unlocking a Galaxy Note 8, which was released in April 2017, with a photo shown on another phone. It’s quite amusing.

There was a “liveness check” added to Face Unlock with the release of Android Jelly Bean in 2012. This works by attempting to detect blinking. I haven’t tried this feature but from what I’ve read on forums, it isn’t very accurate and requires a longer time to process your face – hence probably why the feature isn’t turned on by default. And yes, it could also be fooled by a close-up video of you, though this would be much harder to acquire.

Note: Samsung is aware of the security flaws of Face Unlock, which is why it does not allow identity verification for Samsung Pay to be made using it. Instead it advocates for the use of its iris recognition technology. But is that technology free from flaws? No chance, as a security researcher from Berlin has shown. He took a photo of his friend’s eye from a few metres away (!) in infrared mode (i.e. night mode), printed it out on paper, and then stuck a contact lens on the printed eye. Clever.

Apple’s Face ID

This is where the fun begins. Apple really took this feature seriously. In a nutshell, Face ID works by firstly illuminating your face with IR light (IR = infrared light that is not visible to the naked eye) and then projecting a further 30,000 (!) IR points onto your face to build a super-detailed 3D map of your facial features. Quite impressive.

This technology, however, has been in use for a very long time. If you’re familiar with the Kinect camera/sensor (initially released in 2010), it uses the same concept of infrared point projection to capture and analyse 3D motion.

So, how do you fool the ‘TrueDepth camera system’, as Apple calls it? It’s not easy because this technology is quite sophisticated. But successful attempts have already been documented in 2017.

To start off with, here’s a video showing identical twins unlocking each other’s phones. Also quite amusing. How about relatives that look similar? It’s been done! Here’s a video showing a 10-year-old boy unlocking his mother’s phone. Now that’s a little more worrisome. However, it shows that iPhone Xs can be an alternative to DNA paternity/maternity tests 🙂 Finally, in November 2017, Vietnamese hackers posted a video documenting how their 3D-printed face mask fooled Apple’s technology. Some elements, like the eyes, on this mask were printed on a standard colour printer. The model of the face was acquired in 5 minutes using a hand-held scanner.

Summary

Apple in September 2017 released a new facial recognition feature with their iPhone X called ‘FaceID’. It works by projecting IR light onto your face to build a detailed 3D map of it. It is hard to fool but successful attempts have been documented in 2017. Samsung’s facial recognition system called Face Unlock has been around since 2011. It, however, only analyses 2D images and hence can be duped easily with printed photos or another phone showing the phone owner’s face.

To be informed when new content like this is posted, subscribe to the mailing list:

Reflection-in-the-eye

Extracting Images from Reflections in the Eye

Posted on by Zbigatron

Ever thought about whether you could zoom in on someone’s eye in a photo and analyse the reflection on it? Read on to find out what research has done in this respect!

A previous post of mine discussed the idea of enhancing regions in an image for better clarity much like we often see in Hollywood films. While researching for that post I stumbled upon an absolutely amazing academic publication from 2013.

The publication in question is entitled “Identifiable Images of Bystanders Extracted from Corneal Reflections” (R. Jenkins & C. Kerr, PloS one 8, no. 12, 2013). In the experiments that Jenkins & Kerr performed, passport-style photographs were taken of volunteers while a group of bystanders stood behind the camera watching. The volunteers’ eyes were then zoomed in on and the faces of the onlookers reflected in the eyes were extracted, as shown in the figure below:

Zooming-face
(Image adapted from the original publication)

Freaky stuff, right!? Despite the fact that these reflections comprised only 0.5% of the initial image size, you can quite clearly make out what is reflected in the eye. The experiments that were performed also showed that the bystanders were not only visible but identifiable. Unfortunately, with a small population size for the experiments, this technically makes the results statistically insignificant (the impact factor of the journal in 2016 was 2.8, which speaks for itself) – but who cares?! The coolness factor of what they did is through the roof! Just take a look at a row of faces that they managed to extract from a few reflections captured by the cameras. Remember, these are reflections located on eyeballs:

reflections-eye-row-faces
(Image taken from the original publication)

With respect to interesting uses for this research the authors state the following:

our findings suggest a novel application of high-resolution photography: for crimes in which victims are photographed, corneal image analysis could be useful for identifying perpetrators.

Imagine a hostage taking a photo of their victim and then being recognised from the reflection in the victim’s eye!

But it gets better. When discussing future work, they mention that 3D reconstruction of the reflected scene could be possible if stereo images are combined from reflections from both eyes. This is technically possible (we’re venturing into work I did for my PhD) but you would need much higher resolution and detailed data of the outer shape of a person’s eye because, believe it or not, we each have a differently shaped eyeball.

Is there a catch? Yes, unfortunately so. I’ve purposely left this part to the very end because most people don’t read this far down a page and I didn’t want to spoil the fun for anyone 🙂 But the catch is this: the Hasselblad H2D camera used in this research produces images at super-high resolution: 5,412 x 7,216 pixels. That’s a whopping 39 megapixels! In comparison, the iPhone X camera takes pictures at 12 megapixels. And the Hasselblad camera is ridiculously expensive at US$25,000 for a single unit. However, as the authors state, the “pixel count per dollar for digital cameras has been doubling approximately every twelve months”, which means that sooner or later, if this trend continues, we will be sporting such 39 megapixel cameras on our standard phones. Nice!

Summary

Jenkins and Kerr showed in 2013 that extracting reflections on eyeballs from photographs is not only possible but faces on these reflections can be identifiable. This can prove useful in the future for police trying to capture kidnappers or child sex abusers who frequently take photos of their victims. The only caveat is that for this to work, images need to be of super-high resolution. But considering how our phone cameras are improving at a regular rate, we may not be too far away from the ubiquitousness of such technology. To conclude, Jenkins and Kerr get the Noble Prize for Awesomeness from me for 2013 – hands down winners.

To be informed when new content like this is posted, subscribe to the mailing list:

research-image

How to Find a Good Thesis Topic in Computer Vision

Posted on by Zbigatron

“What are some good thesis topics in Computer Vision?”

This is a common question that people ask in forums – and it’s an important question to ask for two reasons:

  1. There’s nothing worse than starting over in research because the path you decided to take turned out to be a dead end.
  2. There’s also nothing worse than being stuck with a generally good topic but one that doesn’t interest you at all. A “good” thesis topic has to be one that interests you and will keep you involved and stimulated for as long as possible.

For these reasons, it’s best to do as much research as you can to avoid the above pitfalls or your days of research will slowly become torturous for you – and that would be a shame because computer vision can truly be a lot of fun 🙂

So, down to business.

The purpose of this post is to propose ways to find that one perfect topic that will keep you engaged for months (or years) to come – and something you’ll be proud to talk about amongst friends and family.

I’ll start the discussion off by saying that your search strategy for topics depends entirely on whether you’re preparing for a Master’s thesis or a PhD. The former can be more general, the latter is (nearly always) very fine-grained specific. Let’s start with undergraduate topics first.

Undergraduate Studies

I’ll propose here three steps you can take to assist in your search: looking at the applications of computer vision, examining the OpenCV library, and talking to potential supervisors.

Applications of Computer Vision

Computer Vision has so many uses in the world. Why not look through a comprehensive list of them and see if anything on that list draws you in? Here’s one such list I collected from the British Machine Vision Association:

  • agriculture
  • augmented reality
  • autonomous vehicles (big one nowadays!)
  • biometrics
  • character recognition
  • forensics
  • industrial quality inspection
  • face recognition
  • gesture analysis
  • geoscience
  • image restoration
  • medical image analysis
  • pollution monitoring
  • process control
  • remote sensing
  • robotics (e.g. navigation)
  • security and surveillance
  • transport

Go through this list and work out if something stands out for you. Perhaps your family is involved in agriculture? Look up how computer vision is helping in this field! The Economist wrote a fascinating article entitled “The Future of Agriculture” in which they discuss, among other things, the use of drones to monitor crops, create contour maps of fields, etc. Perhaps Computer Vision can assist with some of these tasks? Look into this!

OpenCV

OpenCV is the best library out there for image and video processing (I’ll be writing a lot more about it on this blog). Other libraries do exist that do certain specific things a little better, e.g. Tracking.js, which performs things like tracking inside the browser, but generally speaking, there’s nothing better than OpenCV.

On the topic of searching for thesis topics, I recall once reading a suggestion of going through the functions that OpenCV has to offer and seeing if anything sticks out at you there. A brilliant idea. Work down the list of the OpenCV documentation. Perhaps face recognition interests you? There are so many interesting projects where this can be utilised!

Talk to potential supervisors

You can’t go past this suggestion. Every academic has ideas constantly buzzing around his head. Academics are immersed in their field of research and are always to talking to people in the industry to look for interesting projects that they could get funding for. Go and talk to the academics at your university that are involved in Computer Vision. I’m sure they’ll have at least one project proposal ready to go for you.

You should also run any ideas of yours past them that may have emerged from the two previous steps. Or at least mention things that stood out for you (e.g. agriculture). They may be able to come up with something themselves.

PhD Studies

Well, if you’ve reached this far in your studies then chances are you have a fairly good idea of how this all works now. I won’t patronise you too much, then. But I will mention three points that I wish someone had told me prior to starting my PhD adventure:

  • You should be building your research topic around a supervisor. They’ve been in the field for a long time and know where the niches and dead ends are. Use their experience! If there’s a supervisor who is constantly publishing in object tracking, then doing research with them in this area makes sense.
  • If your supervisor has a ready-made topic for you, CONSIDER TAKING IT. I can’t stress this enough. Usually the first year of your PhD involves you searching (often blindly) around various fields in Computer Vision and then just going deeper and deeper into one specific area to find a niche. If your supervisor has a topic on hand for you, this means that you are already one year ahead of the crowd. And that means one year saved of frustration because searching around in a vast realm of publications can be daunting – believe me, I’ve been there.
  • Avoid going into trending topics. For example, object recognition using Convolutional Neural Networks is a topic that currently everyone is going crazy about in the world of Computer Vision. This means that in your studies, you will be competing for publications with big players (e.g. Google) who have money, manpower, and computer power at their disposal. You don’t want to enter into this war unless you are confident that your supervisor knows what they’re doing and/or your university has the capabilities to play in this big league also.

Summary

Spending time looking for a thesis topic is time worth spending. It could save you from future pitfalls. With respect to undergraduate thesis topics looking at Computer Vision applications is one place to start. The OpenCV library is another. And talking to potential supervisors at your university is also a good idea.

With respect to PhD thesis topics, it’s important to take into consideration what the fields of expertise of your potential supervisors are and then searching for topics in these areas. If these supervisors have ready-made topics for you, it is worth considering them to save you a lot of time and stress in the first year or so of your studies. Finally, it’s usually good to avoid trending topics because of the people you will be competing against for publications.

But the bottom line is, devote time to finding a topic that truly interests you. It’ll be the difference between wanting to get out of bed to do more and more research in your field or dreading each time you have to walk into your Computer Science building in the morning.

To be informed when new content like this is posted, subscribe to the mailing list:

Enhancing image meme hollywood

Is image enhancing possible? Yes, in a way…

Posted on by Zbigatron

I was rewatching “Bourne Identity” the other day. Love that flick! Heck, the scene at the end is one of my favourites. Jason Bourne grabs a dead guy, jumps off the top floor landing, and while falling shoots a guy square in the middle of the forehead. He then breaks his fall on the dead body he took down with him. That has to be one of the best scenes of all time in the action genre.

But there’s one scene in the film that always causes me to throw up a little in my mouth. It’s the old “Just enhance it!” scene (minute 31 of the movie) and something we see so often in cinemas: people scanning security footage and zooming in on a face; when the image becomes blurry they request for the blur to dissipate. The IT guy waves his wand and presto!, we see a full resolution image on the screen. No one stands a chance against magic like that.

But why is enhancing images as shown in movies so ridiculous? Because you are requesting the computer to create new information for the extra pixels that you are generating. Let’s say you zoom in on a 4×4 region of pixels and want to perform facial recognition on it. You then request for this region to enhance. This means you are requesting more resolution, say 640×480. How on earth is the computer supposed to infer what the additional 307,184 pixels are to contain?

Enhancing image example

The other side to the story

However! Something happened at work that made me realise that the common “Enhance” scenario may not be as far-fetched as one would initially think. A client came to us a few weeks ago requesting that we perform some detailed video analytics of their security footage. They had terabytes of the stuff – but, as is so often the case, the sample video provided to us wasn’t of the best quality. So, we wrote back to the client stating the dilemma and requested that they send us better quality footage. We haven’t heard back from them yet, but you know what? It’s well possible that they will provide us with what we need!

You see, they compressed the video footage in order for it to be sent over the Internet quickly. And here is where the weak link surfaces: transferring of data. If they could have sent the full uncompressed video easily, they would have.

Quality vs transmission restraints

So, back to Hollywood. Let’s say your security footage is recording at some mega resolution. NASA has released images from its Hubble Space Telescope at resolutions of up to 18,000 x 18,000. That’s astronomical! (apologies for the pun). At that resolution, each image is a whopping 400MB (rounded up) in size. This, however, means that you can keep zooming in on their images until the cows come home. Try it out. It’s amazing.

But let’s assume the CIA, those bad guys chasing Bourne, have similar means at their disposal (I mean, who knows what those people are capable of, right!?). Now, let’s say their cameras have a frame rate of 30 frames/sec, which is relatively poor for the CIA. That means that for each second of video you need 12GB of storage space. A full day of recording would require you to have 1 petabyte of space. And that’s just footage from one camera!

It’s possible to store video footage of that size – Google cloud storage capacities are through the roof. But, the bottleneck is the transferring of such data. Imagine if half a building was trying to trawl through security footage in its original form from across the other side of the globe.

The possible scenario

See where I’m going with this? Here is a possible scenario: initially, security footage is sent across the network in compressed form. People scan this footage and then when they see something interesting, they zoom in and request the higher resolution form of the zoomed in region. The IT guy presses a few keys, waits 3 seconds, and the image on the screen is refreshed with NASA quality resolution.

Boom! 

Of course, additional infrastructure would be necessary to deal with various video resolutions but that is no biggie. In fact, we see this idea being utilised in a product all of us use on a daily basis: Google Maps. Each time you zoom in, the image is blurry and you need to wait for more pixels to be downloaded. But initially, low resolution images are transferred to your device to save on bandwidth.

So, is that what’s been happening all these years in our films? No way. Hollywood isn’t that smart. The CIA might be, though. (If not, and they’re reading this: Yes, I will consider being hired by you – get your people to contact my people).

Summary

The old “enhance image” scene from movies may be annoying as hell. But it may not be as far-fetched as things may initially seem. Compressed forms of videos could be sent initially to save on bandwidth. Then, when more resolution is needed, a request can be sent for better quality.

To be informed when new content like this is posted, subscribe to the mailing list: