The Drones’ primary task is to patrol the surface for threats, then eliminate those threats. The drones are always on guard, responding swiftly and violently against anything they do perceive as a threat.
During his day-to-day maintenance, Jack often encounters active drones. Initially, the drones always regard him as a threat, and offer him a brief window of time speak his name and tech number (for example, “Jack, Tech 49”) to authenticate. The drone then compares this speech against some database, shown on their HUD as a zoomed-in image of Jack’s mouth and a vocal frequency.
Occasionally, we see that Jack’s identification doesn’t immediately work. In those cases, he’s given a second chance by the drone to confirm his identity.
Although never shown, it is almost certain that failing to properly identify himself would get Jack immediately killed. We never see any backup mechanism, and when Jack’s response doesn’t immediately work, we see him get very worried. He knows what happens when the drone detects a threat.
Zero Error Tolerance
This pattern is deadly because it offers very little tolerance for error. The Drone does show some desire to give Jack a second chance on his vocal pattern, but it is unclear how many total chances he gets.
On a website, if I enter my password wrong too many times it will lock me out. With this system, the wrong password too many times will get Jack killed.
There are many situations where Jack may not be able to immediately respond:
- Falling off his bike and knocking himself out
- Focus on repairing a drone, when a second drone swoops in to check the situation out
- Severe shock after breaking a limb
As we see in the crashed shuttle scene, the Drones have no hesitation in killing unconscious targets. This means that Jack has a strong chance of being killed by his Drone protector in some of the situations where he needs help the most.
A more effective method could be a passive recognition system. We already know that the drone can remotely detect Jack’s biosignature, and that the Tet has full access to the Drone’s HUD feed.
The Drone then could be automatically set to not attack Jack unless the Tet gives a very specific override. Or, alternatively, the Drone could be hard-wired to never attack Jack at all (though this would complicate the movie’s plot). In any situation where it looks like the Drone might attack anyways, the remote software Vika uses could act as a secondary switch, providing a backup confirmation message.
That said, we must acknowledge that this system excels at is keeping Jack nervous and afraid of active drones. While they help him, he knows that they can turn on him at any moment. This serves the TET by keeping Jack cowed, obedient, and always looking over his shoulder.
The Drones are built as autonomous sentries, able to protect extraordinarily expensive infrastructure against attack. They need to be able to eliminate that threat, quickly and efficiently. Current militaries are facing the exact same issues. Even though they have pledged (for now) to not build autonomous kill systems, modern military planners may find value in having a robot perform a drudging, dangerous task like patrolling remote infrastructure.
The question asked best in Oblivion is “What should constitute a threat?”
Drones fire mercilessly on unarmed civilians and armed enemy militia, but do not attack armed friendly soldiers (Jack). This already implies some level of advanced threat analysis, even if we abhor the choices the Drone makes.
Military Planners will need to answer the same question: How does the algorithm determine a threat? With human labor becoming more and more expensive both monetarily and emotionally, the push for autonomous drone systems will become even stronger for future conflicts.
There is still enough time to research and test potential concepts before we have to make a decision on autonomous drones.
Interaction Design Lessons:
- Don’t threaten civilians and non-combatants.
- Give clear feedback of limits and consequences if a deadly pattern is about to be activated.
- Give users a second chance.