The Halide app is one of the most famous third-party software cameras on iOS. Its developers are thoroughly studying the source code, as well as all available APIs distributed by Apple.
The new iPhone SE was the first smartphone that relies only on artificial intelligence to blur the background when shooting portraits.
So what?
Any portrait mode is obtained using at least a pair of "eyes", that is, lenses. Thus, the camera receives information about the object and its distance from the camera.
The iPhone XR also has one camera, but there is another hardware trick – pixel autofocus. This is also a kind of “pair of eyes,” which gives an idea of the volume of space.
Thus, the iPhone SE was the first iPhone to generate deep blur through only machine learning, relying only on the picture. But, like the iPhone XR, the same limitation applies: blurring is only possible if you are taking a portrait of people.
The developers tried to find out what was the matter.
According to the API distributed by Apple, there is no reason to think that the camera does not distinguish, for example, dogs. She is able to do this and sometimes copes with this task very, very well. For example, a fellow dog named June:
And then an illustration of what depth data the iPhone SE and iPhone XR received respectively:
The darker the farther. Based on this data, the camera understands what needs to be washed out and what not. And according to the data obtained from the camera API, we see that the iPhone SE did much better than the XR. And that's damn cool: the dog’s left ear will become more blurry! Just unbelieveble.
Unfortunately, the camera does not always cope. For example, here the iPhone just goes crazy and does not understand what is happening:
Ben Sandofsky of Halide believes that the smartphone was embarrassed by the tree at the back. It looks so much like horns. As a result, the depth map of this image is as follows:
It looks more like shawarma in a hand than a dog.
Further more complicated is the scene where the camera does not understand what is happening at all. Neural networks will not help here:
There is already iPhone SE compared with 11 Pro (left). SE just smeared everything:
In this case, the portrait would not have come out.
From this we can conclude why Apple allows you to take portraits of only people: they have a well-trained neuroprocessor. Of course, the company could give the opportunity to blur any object, but then we would very often get a bad result. So it's better not to give something than to give and receive something like this:
Natalya Artamonova. Photo: Ilya Kichaev / Wylsacom Media
Do not worry, this picture was not taken on iPhone SE, but on VSmart Live – a smartphone with a dedicated lens for the “portrait mode”.
Apple would be laughed at for that, which would be a huge blow to the company's reputation. Therefore, one of the basic principles of the company is this: either the thing works, or it simply does not exist.
Will one smartphone camera ever be enough to make a good portrait?
Halide experts do not think so. And this is not only a technical problem – in a few years the neuroprocessor will process photos so cool that errors will be minimal.
But there are other difficulties that even a perfect human eye is difficult to solve. It is illustrated by some illusion of Maurits Escher:
Even our eyes find it difficult to understand what is happening in reality: is the object moving away or approaching?
And in which direction the girl is spinning:
People will be divided into two camps. As is the case with the optical illusion about the color of the dress.
All this happens because even our pair of perfect eyes is not enough. But technology is still very interesting to follow.
. (tagsToTranslate) articles
Chief editor of the blog – Erika J. Wells .
