This week I’d like to give you all a slightly more technical look at our eye tracking implementation in the SOLUS VR demo.

It was a lot of fun to work with, and is an interesting case study to look at when designing eye tracking interaction in VR games.

In SOLUS we implemented five different features, and I will go through each of them in a series of three different posts. This is part one;

Fredrik Lindh,
Game Developer Tobii Eye Tracking

Foveated Depth of field

The first thing you might notice when trying out the demo is the foveated depth of field effect. Depth of field is a term most commonly found in photography where it denotes the distance band at which objects in a scene as perceived by a physical camera end up sharp in the final image. In general, objects in front of and beyond the depth of field conversely experience some amount of blurring depending on different factors such as what kind of camera aperture are used, the focal length and other factors.

Interestingly enough, this effect is also present in the human eye. The best way to test this yourself is to hold one of your hands just in front of your face and then look just next to and behind it. If you did it right, you should see your hand blur in your peripheral vision. Then look at your hand and see that the background blurs.

Another interesting thing about depth of field is that virtual cameras, such as those that can be found in games, do not have this focus problem since they don’t actually focus light rays to hit some sensor, but instead use linear algebra to project our game scene onto a plane in front of our virtual camera. This does not mean that achieving depth of field is impossible though. There are many techniques that can let a game designer add blur on top of the rendered image. Of these techniques there are several, offering advantages in either performance, or visual fidelity. All of them however, require the designer to specify some focal distance at which the depth of field should be located. This is all well and good if you want to mimic how a movie or a photograph leverages the technique. That is, by drawing attention to a specific character or object in your scene… but if you instead want to replicate how a human experiences the phenomenon you’re out of luck, since the designer cannot know what object in the current camera scene the user is actually looking at, and consequently what distance to choose for the depth of field - unless he has an eyetracker of course!

By using an eye tracker, it is possible to sample the scene depth of the pixel/pixel area that the user is looking at and figure out what the distance should be at this particular moment.

It sounds like this should be an easy task, but in reality, it is quite difficult. For scenes that have very smooth distance gradients, it works really well, but as soon as you start getting many objects that are close in screen-space, but very far apart in z (distance), that’s where you start getting problems since the eyetracker has small errors in both precision and accuracy. In these cases then, it becomes necessary to use some heuristic to determine which object the user is in fact looking at. At Tobii we call these heuristics GTOM (Gaze-To-Object-Mapping) techniques. Using GTOM, we can guess which object in the scene the user is focusing at, instead of sampling the pixel depth, which leads to better and more convincing results.

And of course; the better the GTOM algorithm, the better it feels!

→ Read Part Two

→ Read Part Three