How Eye Tracking Will TOTALLY Change the Way You Game.

How Eye Tracking Will TOTALLY Change the Way You Game.

They’re just kids, really. Standing around the table, arms poking in and out of my field of view as they excitedly babble in French-scented English, their youthful enthusiasm can’t help but be infectious. This one wants me to try something new in their game, that one wants me to know the game isn’t finished. Their voices rise and fall as each takes a turn. But none of it distracts me from what’s happening on the screen.

By Russ Pitts (Polygon)

devs-tobii-hq
The kids are all right — their youthful enthusiasm in the corner of Tobii’s HQ is an understatement.

I’m playing a very early build of a video game in which the player navigates through a maze of caverns, encountering monsters and spirits, and trying to solve a (hopefully) spooky mystery. It will be a horror game, like many others. But unlike any other, you will play it with your eyes.

As I turn a corner in the maze, my eyes flick up and left and the view of the screen shifts with them. It’s disorienting at first, but within seconds I’m used to it. I’m holding a video game controller in my hands. My left thumb is controlling the direction I am moving. My right thumb is controlling a flashlight, which allows me to see better in the darkened cave. And my eyes … my eyes are controlling where the game’s “camera” points — what would normally be controlled with the right thumbstick.

And my eyes … my eyes are controlling where the game’s “camera” points — what would normally be controlled with the right thumbstick.

It is therefore possible for the light to shine in a completely different direction from where I am looking, just as would happen in real life. And the effect is startling and new.

I turn a corner and see something out of the corner of my eye. I glance at it, the game refocuses on that point, but my hand moves more slowly and I am looking at something too dark to see until the thumb controlling the flashlight catches up. And in that moment of disoriented darkness … anything can happen. Boo!

Is it a ghost I see before me, or merely shadow? That’s what will make for a (hopefully) scary, fun horror game.(Now available for download) And what makes its unique take on the horror genre possible is an eye tracker, by Tobii Tech.

An Overnight Success, 15 Years Later

Tobii may be one of the most successful technology companies you’ve never heard of. If I’m being honest, I hadn’t heard of them before I spoke with them about this story either.

Arriving at Stockholm’s Arlanda airport, I explained to customs officials I was in Sweden on business. They asked for whom I was working. I said “Tobii. It’s a technology company — “

“We know,” they interrupted.

OK. They know. But that’s in Sweden, where Tobii employs over 300 people and has attracted upwards of $50 million in investment. In the rest of the world? You would probably have never heard of them. Unless you’re one of hundreds of university researchers, or thousands of people with disabilities who have already been using Tobii’s line of eye trackers.

tobii-hq
Tobii’s HQ in Stockholm, Sweden

This company coming seemingly out of nowhere with a stand-alone eye-tracking device in a partnership with Steel Series, an eye-tracker-enabled gaming laptop in partnership with MSI, and a growing number software partnerships with companies like Ubisoft and Avalanche might be a new force in gaming, but it’s been studying eyeballs — and tracking them — for a long, long time. When you play Assassin’s Creed: Rogue, The Hunter, The Hunter: Primal and many more games to come (they showed me some, but I can’t talk about them), you will experience what data researchers and ability engineers have known for over a decade: Your eyes know what you’re thinking before your hands do.

You will experience what data researchers and ability engineers have known for over a decade: Your eyes know what you’re thinking before your hands do.

My time at Tobii is full of interesting experiences, but one thing sticks out — and it happens multiple times — the people here are almost supernaturally aware of what I’m thinking. In multiple interviews, Tobii employees will comment on how I am interacting with them, how I maintain eye contact while they are speaking, to encourage them to continue; how I nod at them to indicate that I am listening; and smile to influence them to expand on what they might be saying. Subtle interview tricks I learned so long ago I’m no longer aware of doing them. But the people I talk to at Tobii are aware. They’re aware and responding. Because they’re watching my eyes. And they’re programming computers to do the exact same thing.

Tobii was founded in 2001 by a Henrik Eskilsson, John Elvesjo and Marten Skogö. John was working in a research lab on technology to detect bubbles in various liquids then someone had the bright idea: You know what’s also like a bubble? And then came the idea: Let’s build a device that can track the human eye, then put an eye tracker into every computer. Tobii was born.

tobii-eye-tracking-gen-1
Tobii’s very 1st eye tracker (circa 2001)

“The dream is for us make every computer know what you want to interact with before you even touch any controller,” says Tobii Tech’s vice president of product, Henrik Johansson. “We are making steady progress, we have computers in our labs where you can just look at an icon and place a finger on a touch pad to click, a similar interaction to on a phone or a tablet.”

Currently, Tobii is the market leader. In fact, no one else is even close.

Part of that has to do with a divergence in the evolution of computing technology. When Tobii was founded, no one (except maybe Steve Jobs) expected tablets and phones to become the most common computing devices. But they did, and so Tobii changed course.

future-of-eye-tracking-mulitple-devices
The future of Eye Tracking will be in multiple devices

The new goal, says Johansson: “To be in every laptop and tablet and mobile and car. That’s what we’re working on. It’s Tobii Tech’s task, to make that happen.”

The Window to Your Mind

Tobii Tech is one of three divisions at Tobii. The first focuses on professional researchers. Scientists. Marketers.

“It started out as academic research and broadened throughout to different types of consultants or professionals that want to interpret where a person is looking,” says Tobii’s Johan Bouvin. “It can be research in psychology, neurology, media design, usability, web design … a lot of marketing.”

The idea is that if you can tell what person is looking at, then you can most likely tell a little about what they’re thinking. What you look at is a good approximation of what you think. In marketing especially this is crucial. When you spend thousands of dollars on advertising, you want to be sure people are paying attention.

Eye trackers attached to computer screens, or in the case of some research applications, worn on the face, keep track of where a person’s eyeballs are pointed, and then that hard data is used to decide how effective a given whatever risk at capturing someone’s attention.

From there Tobii started working with assistive technology, spawning a new division, Tobii Dynavox. Eye-tracking speech computers, communication software, computer access tools — Tobii Dynavox makes almost everything you could imagine that might help someone with disabilities use technology. And even some devices that go beyond, to improving quality of life.

johan-bouvin-product-manager
Johan Bouvin, Product Manager at Tobii

Bouvin shows me a large tablet that can be mounted onto, say a wheelchair. It has inputs for a variety of sensors that can be attached anywhere on the body. Even if someone is paralyzed from the neck down TobiiDyanvox device can track the user’s eye movements with the integrated eye tracking, and just like that someone who may not even have use of a hand can use a computer. And if you can imagine all of what might be done with a computer, suddenly you’ve allowed for an almost limitless variety of interactions.

“Being able to use your eyes to browse the web, send emails, create artwork, music, call friends and family and sometimes even get back to work opens up so many doors that previously was closed for people with disabilities.”

The main word they use is that they get some of their independence back.

And then there’s Tobii Tech. Tobii’s third division was founded to do one thing, and one thing only (and has been granted leeway by the corporate headquarters to take as much as four or five years to do it): own the consumer market for eye-tracking. This literally translates into creating that market. Which is what Tobii Tech, in its steadfast, matter-of-factly Swedish way is currently doing.

The Eye Tracker.

tobii-eyex
The Tobii EyeX.

Tobii’s eye tracking works like this: infrared illuminators “light up” the area in front of the computer screen, making it easier for the sensors to lock in on the “bubbles” of your eyes.

Without being able to accurately predict all of the different types of environments a computer might be in, it’s hard to program something to be able to “see” in every possible scenario. Changes in lighting, temperature, humidity, distance, eyewear and a plethora of other variables mean the eye tracker has to function in an almost infinitely unpredictable variety of circumstances. The infrared illuminators level the playing field, and making those emitters smaller and powerful is one of the biggest tricks in adapting Tobii’s technology for consumer devices.

The other big trick is offloading the processing to the eye tracker itself. Tobii’s newest generation device does just that. With an onboard processor and infra-red sensors capable of reading eye movements at 60–90 frames per second, the newest eye tracker will be smaller, lighter, more powerful and uses less power than any eye-tracking device ever made. And Tobii Tech thinks it’s the version that will finally unlock eye-tracking for consumer devices.

Get the latest Tobii Eye Tracker 4C

“Basically it’s a sensor technology that’s pretty much what you and I do” says Bouvin. He’s giving me a tour of Tobii Tech’s demo room, a space filled with all of the devices Tobii designs and manufactures (although most of the manufacturing is now done offsite).

tobii-demo-room
Part of the Tobii demo room with tons of cool devices

Sitting on a pedestal in front of Bouvin is a laptop made by gaming computer maker MSI.

It looks for eyes. It interprets where the eyes are and where they’re directed. Once you know where my eyes are and where they’re directed and you know about the objects that are here, it’s not that difficult to guess what I’m looking at.”

The eye tracker knows, generally, what eyes look like. It knows that an eye is a small, black circle surrounded by a larger circle of color. And it knows that what the smaller circle is pointing at is what you’re seeing.

“It’s not even a metaphor or analogy. That’s what the algorithm looks for. You can see the reflection of light in my eyes as lines. When I move my eyes, those reflections will be in different places, compared to the pupil. That’s part of the information you get from the system.

“Then it’s pure trigonometry from that point on. If this system knows where it is with respect to the screen and it can calculate where the eyes are and where they’re directed, that’s enough information to get a point in space. Those vectors cross each other.”

Which is engineer speak for “the computer is reading your mind”.

“It makes the device more aware of you. That experience of having a machine that starts to understand not only your commands, but also your attention and your state.” — Johan Bouvin

If you own an iPhone, you already own a device that’s aware of you. Someone calls you. The phone’s screen lights up. You look at the screen to see who’s calling. Let’s say you decide to answer. You touch the green button on the screen and raise the phone to your ear. The phone, using it’s optical sensors, can tell that it is being held close to an ear, and shuts off the screen to save power. It knows you’re not looking at the screen, because the screen is pressed to the side of your head, next to your ear. This phone knows you can’t see from your ear, so it stops showing you things on the screen. That’s responsive technology.

Tobii eye tracking will take this further.

“Our eyes give so many hints about what we’re about to do, our cognitive processes,” says Bouvin. “Capturing that interaction in the man-machine interface is so exciting, finding ways where you can get the logic, based on those eye movements, to behave in a way that feels natural for the user.

“Any motor action you take, anything you touch, anything you move, your eyes will be there before your hand.”

natural-targeting-in-real-life
Natural Targeting in real life!

Bouvin demonstrates by picking up a pencil from the table in front of us. First he looks at the pencil, then he moves his hand to pick it up. In the computing world, we’ve become used to this type of interaction, but everyone who is currently alive who knows how to use a computer has had to train their mind to add a step between what they see and interacting with it.

We’ve had to learn the motor control of seeing, and then moving a cursor with a mouse or a trackpad, and then interacting.

Tobii eye tracking will remove that inter-evolutionary step, making it possible for us to interact with computers in the same way we interact with the world. Look at a thing. Interact with the thing. No cursor required.

“What you get with eye tracking is you can substitute pretty much all of that positioning, all that directional input, because your eyes are there before you touch,” says Bouvin. “You’re looking there. The movement and directional thing becomes unnecessary. Same with a mouse. You’re already there with your eyes. Moving the mouse cursor there is a step you can more or less do without. The only thing that’s left is the actual action.”

Bouvin expresses this as a hope for the future, but here in Tobii’s headquarters I’m seeing it in action, using a device that’s available to purchase now, playing a game that’s already in stores. It’s not the future: it’s now.

The Four Immersions

“There’s a fundamental problem with games,” Olsson tells me. “You walk around their world like you’re a dog with a cone around your head. And that’s a major problem. No matter how good you make the graphics you have no peripheral vision.”

The Tobii eye tracking can solve that problem. And I soon get a demonstration in another game, this one the recently released Assassin’s Creed: Rogue. The first AAA title to ship with support for eye tracking.

“There’s a fundamental problem with games, You walk around their world like you’re a dog with a cone around your head. And that’s a major problem. No matter how good you make the graphics you have no peripheral vision.” — Anders Olsson VP of Software at Tobii

I’m walking around a room in a laboratory, one of the places in the fictional world of Assassin’s Creed where the memories of your long-dead assassin ancestors are being pulled from your DNA or whatever (I don’t play a lot of Assassin’s Creed games). I’m controlling the game as I normally would, but something curious happens when I glance at the edge of the screen: The Tobii eye tracker detects my gaze and begins panning the screen to the left or right, depending on where I’m looking. It’s a feeling not unlike suddenly focusing attention on what’s in your peripheral vision. The game world in Rogue, like in the Frenchmen’s game, is responding to what I’m looking at.

assassins-creed-rogue-eye-tracking
See the game through the character’s eyes in Assassin’s Creed. The game intuitively pans to where you are looking across the screen.

The second immersion is eye contact, and it’s a harder immersion to implement. Tobii prides itself on providing an SDK and development libraries that make it possible to implement most eye-tracking features in a minimal amount of time. The fact that the eye tracking practically sits on top of the existing game engine, tapping into the HID and camera controls that already require external connections for the game engine, whatever engine a developer may be using, possible to implement features that look and feel like magic in practically no time at all, in game development terms.

All of the eye tracking features implemented in Assassin’s Creed: Rogue were implemented in “about two weeks.” Something almost unheard of in game development.

Eye contact, in Tobii terms, will mean a number of things for the games that might implement it. The simplest will make objects in the game world aware they’re being looked at. Similar to how an object might change color if a mouse cursor is hovering over it. You can choose to interact with a game object simply by looking at it, and then use the controller to do whatever the game might allow you to do. More complex (but seemingly inevitable) is a game in which characters you meet in the world will hold your gaze, interacting with you as a human might: by looking you in the eye.

engagement-eye-tracking
Engage an opponent with a lingering look or enjoy cool little exchanges like a friendly nod or wink.

Nothing so complex is happening right now in Assassin’s Creed: Rogue, but the game nevertheless manages to surprise me. My demo soon moves into the traditional realm of Assassin’s Creed: a historical world in which I can navigate obstacles and then kill people. In true Assassin’s Creed fashion, I press X on the controller as I am moving and am able to leap over things, climb walls and trees and so forth.

But with the eye tracker I am suddenly, and almost without realizing it, moving across obstacles even before I move the controller. The game, supplementing the right thumbstick with the Tobii eye tracker, is responding to my gaze faster than I am moving the thumbstick where I want to look. And like the Frenchmen’s little horror game, it begins reading my mind; anticipating my moves, perfectly.

The third immersion is described to me as “natural targeting”, but it’s basically shooting at two things at once.

Eye tracking can make that happen. I did it. in a game called Son of Nor I can use mouse as normal to control the direction of my character, and at the same time shoot/throw objects in another direction — sideways, upwards, backwards.

son-of-nor-eye-tracking
Pick up a rock at the bottom right of the screen and throw it to the top left — just by looking right to left.

In fact, it makes Son of Nor so much fun I had a hard time stepping away from it, even though it’s a type of game I wouldn’t ordinarily enjoy playing. With Tobii eye tracking, you can move around the world with the controller or keyboard, and pick up objects and then throw them simply by looking at them. It feels like having super powers.

Imagine playing Call of Duty and shooting at whatever your crosshair is aimed at with your gun, but being able to throw a grenade somewhere else on the screen — simultaneously — just by looking at it.

Which makes Tobii’s fourth immersion sound like fantasy. They call it “Immersive Graphics and Sound”. And it amounts to nothing less than making a video game show you the world in the same way you would see and hear the real world.

Look at something in the background and the foreground falls out of focus, and that object becomes clearer. Look left, and the same thing happens. Your peripheral vision suddenly takes focus, and whatever you were looking at become harder to see.

fables-in-motion-eye-tracking
A simple look at a particular detail within the game will enhance its graphic or evoke a situational sound

Graphics cards are powerful enough now, Olsson believes, to make this happen, but no game in the world can manage it without eye tracking.

And Tobii has announced a $1 million dollar development fund to help indie game developers around the world program for it.

“Tobii has announced a $1 million dollar development fund to help indie game developers around the world program for it.”

But all of the games currently available that utilize Tobii’s eye tracking only capture a part of the potential of the technology. The first game in the world that will manage to use all of the power of the Tobii technology is a little horror game being made in a corner of the Tobii headquarters, by a quintet of Frenchmen.

The Window to the World

The Frenchmen have a workspace, deep on a bottom floor of Tobii’s spacious headquarters. It was a last-minute addition to the deal that brought them here from France. It took them weeks to get an apartment. They have only a few weeks more to finish their game before heading back to school, to graduate. This summer fellowship, as it were, an odd, little opportunity that will likely fade away into their memories of their time at University. But right now they can’t believe how lucky they are.

dungeon-tobii
The “dungeon” where the Frenchmen worked & played

Interested in eye-tracking technology, the Frenchmen reached out to Tobii tech for advice, and maybe a prototype. This led to an opportunity to create a game for a showcase in Cannes. And that led to now: working at Tobii Tech, for Tobii Tech, to create a game Tobii can use to promote eye tracking.

Their game, if I’m being honest, wouldn’t be noteworthy without the eye-tracker. It’s also not finished when I get my eyes (literally) on it. But as far as horror games go, it already has a good grip on claustrophobia.

I have to imagine what it will be like when there are things to do and run away from, but I suspect, just from playing it in its current state, it will be intriguing. Or at least would be, just playing it with a controller.

sketches-glimpse-of-fear
Sketches for characters in Glimpse of Fear

Playing with the Tobii eye tracking, however, is an otherworldly experience. The game is responding to where I’m looking. Tobii’s technology (after a few minutes spent calibrating it to my eyes) is following my gaze, panning across the screen whenever my eyes move. It’s distracting — at first — not because the outcome is undesirable, but because I can’t stop feeling as if the machine is reading my mind. It’s as if I’m thinking the command and the game, without needing me to move a control, is responding. Instantly.

The Frenchmen’s game (Now available for download) is designed to illustrate what Tobii calls “the four immersions”, or the four ways in which eye tracking can enhance gameplay.

The first is controlling the screen. It’s what allows me to move the camera in the Frenchmen’s horror game.

I soon get stuck in the labyrinth the Frenchmen have created. Without monsters or anything to interact with, every tunnel looks just like every other, and I can’t find my way out.

One of the Frenchmen takes over (the one who made the maze) and he, too, soon gets lost. Which leads to lots of excited yelling (in French) and general frustration.

game-dev-glimpse-of-fear
Game Development of the labyrinth in Glimpse of Fear

I tune out. It doesn’t bother me that they built a maze so confusing they get lost in it themselves. In fact, that’s kind of exciting.

But I can’t help wondering, if they get this game of theirs to actually work, will there be any other experience in the world like it? Could a game made by a handful of French game designs students, working out of a tiny corner of a technology company in Sweden, change the world?

I’m generally skeptical about such things, But on this day, after using this piece of technology, it all seems too possible.


* You can now get the latest Tobii Eye Tracker here

Join us on Discord

Find other Eye Tracker gamers to play with and participate in exclusive giveaways on the Tobii Gaming Discord channel. Click to get your invite!

discord-cta
Tobii Gaming

Written by

Tobii Gaming

Hi, we are a bunch of developers, QA Engineers, customer support, product managers, and more (mostly gamers) aiming to combine the wonder, competitiveness, and creativity of gaming with the technology of eye tracking. Over the past decade, we have built eye trackers to revolutionize the way we play, creating a vital tool for competitive gaming, and empowering a new generation of content creators, as well as their followers. We love to game, we love eye tracking.