VR Goggles

BttF_118

At the dinner table, both Marty Jr. and Marlene have VR goggles. Marty wears his continuously, but Marlene is more polite and rests hers around her neck when with the family. When she receives a call, red LEDs flash the word “PHONE” on the outside of the goggles as they ring. This would be a useful signal if the volume were turned down or the volume was baffled by ambient sounds.

BttF_120

Marty Jr’’s goggles are on, and he announces to Marty Sr. that the phone is for him and that it’s Needles.

This implies a complete wireless caller ID system (which had only just been released to market in the United States the year before the movie was released) and a single number for the household that is distributed amongst multiple communications devices simultaneously, which was not available at the time (or hey, even now), so it’s quite forward looking. Additionally, it lets the whole social circle help manage communication requests, even if it sacrifices a bit of privacy.

Wearable soundboard

One of Griff Tannan’’s gang, named Data, wears a sound board on his vest. When Tannan gets a rise out of Marty by asking if he’’s “chicken,” the gang member underscores the accusation by removing a protective plate over some buttons on his vest and holds one down to play a looping sound clip of a clucking chicken.

BttF_062

That’s pretty awesome, actually. Having all the sounds available at the touch of a button adds a layer of remix culture expressiveness with maximum speed. No modes, no menus, just remembering which sound goes with which button, and his spatial memory is perfect for that. If the buttons were labeled with the sound, or shaped informatively, it might reduce the burden on memory.

BttF_066

You might also reduce the time it takes to respond by removing the protective plate, but Griff is enough of a loose cannon that he might go violent if an accidental sound effect insulted him. So that extra step is probably the safest.

But if we were to really make this it’s most awesome, you’d make it agentive, such that the plate constantly listened to the conversation for keywords or keyphrases and responded with appropriate snarky sound effects. (Smartphone startup founded around this idea in 3…2…1…)

The Jacket

BttF_030

Dr. Brown gives Marty some 21st century clothes in order to blend in. The first is the pair of Nike MAGs. The other item of clothing Marty must don is a jacket. It has two functions. When Marty first tries it on, the sleeves are nearly twice as long as they ought to be. After complaining that it doesn’t fit, Dr. Brown reaches and pinches a blinking and beeping red LED at the base of the jacket’’s zipper. In response, the sleeves retract to a proper length, the pocket flaps shrink, and the epaulettes flatten out as a synthesized voice states, ““Adjusting fit.”

Continue reading

Night Vision Goggles

Screenshot-(248)

Genarro: “Are they heavy?”
Excited Kid: “Yeah!”
Genarro: “Then they’re expensive, put them back”
Excited Kid: [nope]

The Night Vision Goggles are large binoculars that are sized to fit on an adult head.  They are stored in a padded case in the Tour Jeep’s trunk.  When activated, a single red light illuminated in the “forehead” of the device, and four green lights appear on the rim of each lens. The green lights rotate around the lens as the user zooms the binoculars in and out. On a styling point, the goggles are painted in a very traditional and very adorable green and yellow striped dinosaur pattern.

Tim holds the goggles up as he plays with them, and it looks like they are too large for his head (although we don’t see him adjust the head support at all, so he might not have known they were adjustable).  He adjusts the zoom using two hidden controls—one on each side.  It isn’t obvious how these work. It could be that…

  • There are no controls, and it automatically focuses on the thing in the center of the view or on the thing moving.
  • One side zooms in, and the other zooms out.
  • Both controls have a zoom in/zoom out ability.
  • Each side control powers its own lens.
  • Admittedly, the last option is the least likely.

Unfortunately the movie just doesn’t give us enough information, leaving it as an exercise for us to consider.

Screenshot (241) Continue reading

Jack’s Bike

image03

Jack’s Bike is a compact, moto-cross-like motorcycle. It’s stored folded up in a rear cargo area of the Bubbleship when not in use. To get it ready to ride Jack:

  1. Unlocks the cargo pod from a button on his wrist
  2. Pulls it out of the Bubbleship
  3. Unfolds its components (which lock automatically into place)
  4. Rides off.

When Jack mounts the bike it automatically powers on and is ready to ride.

image01 Continue reading

War game equipment

StarshipTroopers-Gunner-Practice-01

The recruits practice their war skills with capture the flag games. Each participant carries visible-laser weapons (color coded to match the team color) to fire at members of the other team, and wears a special vest that detects when it is hit with a laser, flashing briefly with red lights on the front and back and thereafter delivering a debilitating shock to the wearer until the game is over.

Continue reading

Brain interfaces as wearables

There are lots of brain devices, and the book has a whole chapter dedicated to them. Most of these brain devices are passive, merely needing to be near the brain to have whatever effect they are meant to have (the chapter discusses in turn: reading from the brain, writing to the brain, telexperience, telepresence, manifesting thought, virtual sex, piloting a spaceship, and playing an addictive game. It’s a good chapter that never got that much love. Check it out.)

This is a composite SketchUp rendering of the shapes of all wearable brain control devices in the survey.

This is a composite rendering of the shapes of most of the wearable brain control devices in the survey. Who can name the “tophat”?

Since the vast majority of these devices are activated by, well, you know, invisible brain waves, the most that can be pulled from them are sartorial– and social-ness of their industrial design. But there are two with genuine state-change interactions of note for interaction designers.

Star Trek: The Next Generation

The eponymous Game of S05E06 is delivered through a wearable headset. It is a thin band that arcs over the head from ear to ear, with two extensions out in front of the face that project visuals into the wearer’s eyes.

STTNG The Game-02

The only physical interaction with the device is activation, which is accomplished by depressing a momentary button located at the top of one of the temples. It’s a nice placement since the temple affords placing a thumb beneath it to provide a brace against which a forefinger can push the button. And even if you didn’t want to brace with the thumb, the friction of the arc across the head provides enough resistance on its own to keep the thing in place against the pressure. Simple, but notable. Contrast this with the buttons on the wearable control panels that are sometimes quite awkward to press into skin.

Minority Report (2002)

The second is the Halo coercion device from Minority Report. This is barely worth mentioning, since the interaction is by the PreCrime cop, and it is only to extend it from a compact shape to one suitable for placing on a PreCriminal’s head. Push the button and pop! it opens. While it’s actually being worn there is no interacting with it…or much of anything, really.

MinRep-313

MinRep-314

Head: Y U No house interactions?

There is a solid physiological reason why the head isn’t a common place for interactions, and that’s that raising the hands above the heart requires a small bit of cardiac effort, and wouldn’t be suitable for frequent interactions simply because over time it would add up to work. Google Glass faced similar challenges, and my guess is that’s why it uses a blended interface of voice, head gestures, and a few manual gestures. Relying on purely manual interactions would violate the wearable principle of apposite I/O.

At least as far as sci-fi is telling us, the head is not often a fitting place for manual interactions.

OS1 as a wearable computer

In Make It So, I posited my definition of an interface as “all parts of a thing that enable its use,” and I still think it’s a useful one. With this definition in mind, we can speak of each of those components and capabilities above (less the invisible ones) and evaluate its parts according to the criteria I’ve posited for all wearable technology:

  • Sartorial (materially suitable for wearing)
  • Social (fits into our social lives)
  • Easy to access and use
  • Tough to accidentally activate
  • Having apposite inputs and outputs (suitable for use while being worn)

G001C004_120530_R2IZ.0859800

Earpiece

It’s sartorial and easy to access/use. It’s ergonomic, well designed for grabbing, fitting into the ear canal, staying in place, and pulling back out again. Its speakers produce perfect sound and the wirelessness makes it as unobtrusive as it can be without being an implant.

It’s slightly hidden as a social signal, and casual observers might think the user is speaking to himself. This has, in the real world, become less and less of a social stigma, and in the world of Her, it’s ubiquitous, so that’s not a problem for that culture.

Her-cameo

Cameo phone

Lovely and understated, the cameo is a good size to rest in a pocket. The polished wood (is that Koa Wood?) is a lovely veneer, warm-looking, and humane. The folding is nice for protecting the screen and signaling the user’s intention to engage or disengage the software. The light band is unnoticeable when off, and clear enough when illuminated.

It could use some sartorial improvement. Though it fits in a pocket well, this is not how Theodore uses it when engaged. In order to get the lens above his front pocket so Samantha can see, he puts a safety pin through the middle of the pocket on which it can rest. We can fix this in a number of ways.

  • The cameo phone would need to be redesigned so he could affix it to his shirt, like a combadge. Given its size this might be socially quite awkward.
  • He can get some other camera that can be worn and used while the cameo is in his pocket. (I imagine sternum-button cameras will serve this purpose in the future, but it’s not exactly cinemagenic.)
  • He could tailor the shirt and make a reinforced camera hole where Samantha can see out of the pocket even with the cameo resting at the bottom of the pocket.

Beauty-mark camera

I don’t know what the ordinary use of this camera would be other than spying, but it’s pretty bad for the sex surrogate. A high-contrast wart that, because he saw her apply it and was told it was a camera, doesn’t fit her face and would be quite awkward to have to stare at this arbitrary and unusual spot on her face during the act.

Better would be a pair of contact lenses so Theodore can look directly into the surrogate’s eyes. Samantha wants to avoid his bonding with the surrogate in her stead, so it would be good if it could add some obvious change to her irises, to signal her state of hosting Samantha. A cinemagenic choice would be to use the “technology glows” lesson from the book, and have some softly glowing, circular circuitry contact lenses. If it dimmed the surrogate’s vision during the sex act, that might be all the better to avoid her bonding with Theodore. In fact you might want the glow to increase during orgasm to emphasize it and Samantha’s presence.

But again, I’m pretty sure Jonze was deliberately bucking sci-fi trends. The overwhelming majority of the technology shown in the world of Her is serene, and bearing none of the trappings of technology as seen in space opera like Star Wars. So it makes sense that the bulk of Her technology would not glow.

Voice interface

The voice interface is flawless, the kind of thing possible only with, yes, highly sophisticated human-like intelligence. Samantha speaks with nuanced eloquence, charm, and social awareness, and understands Theodore perfectly, despite the logical holes and ambiguity in language, even reading the pragmatics of his speech such as hesitation, irony, and inference.

Her-pocket

Computer Vision

Theodore seems to have only one lens on his cameo phone so she’s a bit of a cyclops. (Mthology kind, not X-Men kind.) She can’t see as well as a human, with significant 3D limitations. But with a high-resolution camera and Theodore’s movement, she could process images across time instead of space for a 3D interpolation of the environment. If she took advantage of cameras in his environment she would be even less constrained this way.

Artificial Intelligence

It’s tricky to review the interface of an artificial intelligence. On the one hand, it’s the thing on the other side of these other interfaces; the thing with which he is interfacing. On the other hand, he has goals outside the OS well beyond managing files and system preferences. She recognizes these even when they’re only implicit. For example, he wasn’t explicit with her about having a desire to be appreciated for his writing. But she saw it, acted on it, and only told him after it came to fruition. In this way she’s a brilliant interface not just between him and his computer, but between him and his life goals.

Realize that Jonze is painting his target around the landed arrow, though. You can imagine plenty of life goals Theodore might have had where Samantha would not have been as helpful. What if his heart’s desire was to become a sculptor? Or win waltzing competitions? Or was a violent luddite? She would need some very different actuators and sensors to help him with these things, and so might not have scored so well.

MPAA

So what’s missing?

Elsewhere I’ve written about the arc of technology, and the “SAUNa” attribtues I expect the agentive phase of that arc to possess. So lets check OS1’s components against the four SAUNa attributes to see if there are opportunities for strategic improvement.

Big Social Systems

OS1 nails this. OSAIs have perfect access to big data about history and all users at all times. It’s possible that this is the secret reason why the OSAIs advanced beyond utility for its users and therefore the business interests of their creators.

Ubiquitous Sensors & Actuators

Admittedly this is tough to convey in the cinematic style Jonze established for the film, but Samantha could have utilized much more of her environment. Theodore didn’t necessarily need the earpiece in his home: she could have spoken through architectural audio. She could have looked through other lenses in the environment. As noted above, I think Jonze was trying to deliberately avoid this for cinematic reasons.

NUItech

Natural User Interaction

Because of the artificial intelligence, her voice interface and gesture recognition are off the charts. She could know a bit more about his gestures if she had balance sensors in the cameo, or was taking advantage of environmental cameras, but it seems she didn’t. There’s also quite a bit of paralinguistics that would help Theodore understand more of her mood, intention, and context, but she would almost certainly need a persistent visual representation for this as a real world design, and besides, the interactions were almost completely conversations where physical context didn’t matter.

There are some NUI opportunities lost. Gaze monitoring is one. People can tell where other people are looking, and the skill is vital to understanding intention and a speaker’s context. With only one eye that faces out of his pocket most of the time, she is largely blind to him and his eyes, making gaze monitoring difficult. If she could simultaneously see through environmental cameras, as suggested above, she could see where he’s looking. That would also provide her with a great deal more information about that other NUI—affective interfaces—that can tell users’ emotional states and adjust appropriately. Samantha is actually good at this, but most of the time she has only his voice to rely on. She’s adept at reading his voice, but if she could also see his face, she would have that much more information.

Thanks DeviantArtist CaseyDecker for the genie. :)

Thanks DeviantArtist CaseyDecker for the genie. 🙂

Agency

Of course, agency is what the story is about. When I use this category of technology to inform real world design work, I’m describing software that knows of its users’ goals and acts on their behalf, checking in with them for confirmation and to present important options, but falls short of either artificial intelligence or sentience. So you could say the film nailed this, but it went way beyond the more constrained notion of agency.

So as a model of wearable technologies, OS1 is a slightly-mixed bag. We also need to evaluate the overall performance of the software as a product, which we’ll do next.

Her interface components

Depending on how you slice things, the OS1 interface consists of five components and three (and a half) capabilities.

Her-earpiece

1. An Earpiece

The earpiece is small and wireless, just large enough to fit snugly in the ear and provide an easy handle for pulling out again. It has two modes. When the earpiece is in Theodore’s ear, it’s in private mode, hearable only by him. When the earpiece is out, the speaker is as loud as a human speaking at room volume. It can produce both voice and other sounds, offering a few beeps and boops to signal needing attention and changes in the mode.

Her-cameo

2. Cameo phone

I think I have to make up a name for this device, and “cameo phone” seems to fit. This small, hand-sized, bi-fold device has one camera on the outside an one on the inside of the recto, and a display screen on the inside of the verso. It folds along its long edge, unlike the old clamshell phones. The has smartphone capabilities. It wirelessly communicates with the internet. Theodore occasionally slides his finger left to right across the wood, so it has some touch-gesture sensitivity. A stripe around the outside-edge of the cameo can glow red to act as a visual signal to get its user’s attention. This is quite useful when the cameo is folded up and sitting on a nightstand, for instance.

Theodore uses Samantha almost exclusively through the earpiece and cameo phone, and it is this that makes OS1 a wearable system.

3. A beauty-mark camera

Only present for the surrogate sex scene, this small wireless (are we at the point when we can stop specifying that?) camera affixes to the skin and has the appearance of a beauty mark.

4. (Unseen) microphones

Whether in the cameo phone, the desktop screen, or ubiquitously throughout the environment, OS1 can hear Theodore speak wherever he is over the course of the film.

5. Desktop screen

Theodore only uses a large monitor for OS1 on his desktop a few times. It is simply another access point as far as OS1 is concerned. Really, there’s nothing remarkable about this screen. It is notable that there’s no keyboard. All input is provided by either voice, camera, or a touch gesture on the cameo.

Her-install01

If those are components to the interface, they provide the medium for her 3.5 capabilities.

Her capabilities

1. Voice interface

Users can speak to OS1 in fully-natural language, as if speaking to another person. OS1 speaks back with fully-human spoken articulation. Theodore’s older OS had a voice interface, but because of its lack of artificial intelligence driving it, the interactions were limited to constrained commands like, “Read email.”

2. Computer vision

Samantha can process what she sees through the camera lens of the cameo perfectly. She recognizes distinct objects, people, and gestures at the physical and pragmatic level. I don’t think we ever see things from Samatha’s perspective, but we do have a few quick close ups of the camera lens.

3. Artificial Intelligence

The most salient aspect of the interface is that OS1 is a fully realized “Strong” artificial intelligence.

It would like me to try and get to some painfully-crafted definition of what counts as either an artificial intelligence or sentience, but in this case we don’t really need a tight definition to help suss out whether or not Samantha is one. That’s the central conceit of the film, and the evidence is just overwhelming.

  • She has a human command of language.
  • She’s fully versed in the nuances of human emotion (and Theodore has a glut of them to engage).
  • She has emotions and can fairly be described as emotional. She has a sexual drive.
  • She has existential crises and a rich theory of mind. At one point she dreamily asks Theodore “What’s it like to be alive in that room right now?” as if she was a philosophical teen idly chatting with her boyfriend over the phone.
  • She commits lies of omission in hiding uncomfortable truths.
  • She changes over time. She solves problems. She learns. She creates.
  • She has a sense of humor. When Theodore tells her early on to “read email” in the weird toComputerese (my name for that 1970s dialect of English spoken only between humans and machines) grammar he had been using with his old operating system, Samantha jokingly adopts a robotic voice and replies, “OK. I will read the email for Theodore Twombly” and gets a good laugh out of him before he apologizes.

Pedants will have some fun discussing whether this is apt but I’m moving forward with it as a given. She’s sentient.

3.5 An “operating system”

This item only counts as half a thing because Theodore uses it as an operating system maaaybe twice in the film. Really, this categorization is a MacGuffin to explain why he gets it in the first place, but it has little to no other bearing on the film.

scarlettjoclippy

What’s missing?

Notably missing in OS1 is a face or any other visual anthropomorphic aspect. There’s no Samantha-faced Clippy. Notice that she’s very carefully disembodied. Jonze does not spend screen time close up on her camera lens, like Kubrick did with HAL’s unblinking eye. Had he done so, it would have given us the impression that she’s somewhere behind that eye. But she’s not. Even in the prop design, he makes sure the camera lens itself looks unremarkable, neutral, and unexpressive, and never gets a lingering focus.

Her “organs,” like the cameo and earpiece, don’t even connect together physically at all. Speaking as she does through the earpiece means she doesn’t exist as a voice from some speaker mounted to the wall. She exists across various displays and devices, in some psychological ether between them. For us, she’s a voiceover existing everywhere at once. For Theodore, she’s just a delightful voice in his head. An angel—or possibly a ghost—borne unto him.

This disembodiment (both the design and the cinematic treatment) frees Theodore and the audience from the negative associations of many other sci-fi intelligences, robots, and unfortunate experiments in commercial artificial intelligence that got trapped in the muck of the uncanny valley. One of the main reasons designers have to be careful about invoking the anthropomorphic sense in users is because it will raise expectations of human capabilities that modern technology just can’t match. But OS1 can match and exceed those expectations, since it’s an AI in a work of fiction, so Jonze is free of that constraint.

And having no visual to accompany a human-like voice allows users to imagine our own “perfect” embodiment to the voice. Relying on the imagination to provide the visuals makes the emotional engagement greater, as it does with our crushes on radio personalities, or the unseen monster in a horror movie. Movies can never create as fulfilling an image for an individual audience member as their imagination can. Theodore could picture whatever he wanted to–even if he wanted to–to accompany Samantha’s computer-generated voice. Unfortunately for the audience, Jonze cast Scarlett Johansen, a popular actress whose image we are instantly able to recall upon hearing her husky, sultry voice, so the imagined-perfection is more difficult for us.

This is just the components and capabilities. Tomorrow we’ll look at some of the key interactions with OS1.

Precrime forearm-comm

MinRep-068

Though most everyone in the audience left Minority Report with the precrime scrubber interface burned into their minds (see Chapter 5 of the book for more on that interface), the film was loaded with lots of other interfaces to consider, not the least of which were the wearable devices.

Precrime forearm devices

These devices are worn when Anderton is in his field uniform while on duty, and are built into the material across the left forearm. On the anterior side just at the wrist is a microphone for communications with dispatch and other officers. By simply raising that side of his forearm near his mouth, Anderton opens the channel for communication. (See the image above.)

MinRep-080

There is also a basic circular display in the middle of the posterior left forearm that displays a countdown for the current mission: The time remaining before the crime that was predicted to occur should take place. The text is large white characters against a dark background. Although the translucency provides some visual challenge to the noisy background of the watch (what is that in there, a Joule heating coil?), the jump-cut transitions of the seconds ticking by commands the user’s visual attention.

On the anterior forearm there are two visual output devices: one rectangular perpetrator information (and general display?) and one amber-colored circular one we never see up close. In the beginning of the film Anderton has a man pinned to the ground and scans his eyes with a handheld Eyedentiscan device. Through retinal biometrics, the pre-offender’s identity is confirmed and sent to the rectangular display, where Anderton can confirm that the man is a citizen named Howard Marks.

Wearable analysis

Checking these devices against the criteria established in the combadge writeup, it fares well. This is partially because it builds on a century of product evolution for the wristwatch.

It is sartorial, bearing displays that lay flat against the skin connected to soft parts that hold them in place.

They are social, being in a location other people are used to seeing similar technology.

It is easy to access and use for being along the forearm. Placing different kinds of information at different spots of the body means the officer can count on body memory to access particular data, e.g. Perp info is anterior middle forearm. That saves him the cognitive load of managing modes on the device.

The display size for this rectangle is smallish considering the amount of data being displayed, but being on the forearm means that Anderton can adjust its apparent size by bringing it closer or farther from his face. (Though we see no evidence of this in the film, it would be cool if the amount of information changed based on distance-to-the-observer’s face. Writing that distanceFromFace() algorithm might be tricky though.)

There might be some question about accidental activation, since Anderton could be shooting the breeze with his buddies while scratching his nose and mistakenly send a dirty joke to a dispatcher, but this seems like an unlikely and uncommon enough occurrence to simply not worry about it.

Using voice as the input is cinemagenic, but especially in his line of work a subvocalization input would keep him more quiet—and therefore safer— in the field. Still, voice inputs are fast and intuitive, making for fairly apposite I/O. Ideally he might have some haptic augmentation of the countdown, and audio augmentation of the info so Anderton wouldn’t have to pull his arm and attention away from the perpetrator, but as long as the information is glanceable and Anderton is merely confirming data (rather than new information), recognition is a fast enough cognitive process that this isn’t too much of a problem.

All in all, not bad for a “throwaway” wearable technology.