mmWave radar, you won't see it coming

When I was a kid, Reebok released sneakers with a computer chip in the tongue that could tell you how fast you ran and how high you jumped. They were called the Reebok Traxtar and I thought they were the coolest thing ever, so I begged my parents to get me a pair.

After lacing them up and running and jumping around the neighborhood, the energetic child in me was dismayed to find that the shoe's beeping noises and LEDs were incomprehensible and the stats made no sense. I was still excited to wear them to school until they started beeping and lighting up during class with no way to turn them off. My teacher soon banned the shoes and I had to return them. 😢

In the 1990s, technologists saw that smart devices would become ubiquitous, potentially becoming an overwhelming source of distraction and information overload. This led to the concept of ambient computing

In stark contrast to my Reebok Traxtars, ambient computing is about moving computing into the background by empowering our devices with accurate context, so they can help us, without getting in the way. The goal is to create calm technology.

This article introduces a new powerful sensing technology—mmWave radar—and how it can make our environments and personal devices better aware and adaptive to our needs.

mmWave background

On July 13, 2021, the FCC announced a proposal to open up the 60 GHz band for commercial mmWave applications. This opened the floodgates for companies to commercialize products using mmWave by removing the time and cost required to obtain a waiver from the FCC.^[1] In the FCC’s proposal, they outline three general applications for mmWave:

• Indoor communication
• Outdoor communication
• Radar

Our proposal recognizes the increasing practicality of using mobile radar devices in the 60 GHz band to perform innovative and life-saving functions, including gesture control, detection of unattended children in vehicles, and monitoring of vulnerable medical patients, and it is designed to stimulate the development of new products and services in a wide variety of areas to include, for example, personal safety, autonomous vehicles, home automation, environmental control, and healthcare monitoring.

mmWave radar’s "increasing practicality" comes from recent advances in CMOS manufacturing that allow very high-frequency circuits and antennas to be placed directly onto the radar chip. And because mmWave antennas are small, they can fit into a very small package.

The size, cost, power consumption, reliability, and performance have all improved significantly in recent years. Combined with increases in computing power, these sensors can interpret complex signals using machine learning algorithms, putting mmWave radar at a sweet spot for adoption.

How mmWave radar works

Radar works by transmitting electromagnetic waveforms and processing the returning waveforms as they are reflected back from the environment. Each reflected waveform, with its comparison to the original waveform, contains information that can be used to determine the size, location, direction, and speed of objects in the radar's field of view.

Radar technology covers a wide range of frequencies and processing techniques making it extremely versatile. It has been used for a plethora of applications, from tracking insects to reporting the weather, to sensing phenomena in outer space.

The wavelength or frequency of the radar signal dictates the distance at which it can travel and the size of objects it can detect. As defined by the International Communications Union (ITU), the mmWave spectrum falls into the second-highest or “extremely high frequency” radar band, just below the terahertz or “tremendously high frequency” band. (If they could squeeze in one more band without already entering the infrared spectrum they’d have to call it the “ludicrously high frequency” band, or maybe just the “plaid band” 😄). A 60 GHz signal has a wavelength of 5 mm, about the size of a pencil eraser. Relative to human proportions this is quite small, which allows mmWave radar to capture detailed information about us and our physical environment.

For a more in-depth introduction on the fundamentals of mmWave radar (specifically frequency modulated continuous wave FMCW), see this 5 part video series from Texas Instruments.

mmWave radar advantages

What makes new mmWave radar sensors so useful is that they contain multiple antennas, allowing them to sense in four dimensions. In addition to sensing objects in three-dimensional space, they can also understand the velocity of each reflected point.^[2] Imagine having another sense where each point in space you saw had a different hue based on its instantaneous velocity.

The diagram below compares different sensor examples and how many dimensions they output. As you would expect, as the dimensions increase, the richness and usefulness of the information increases.

One of mmWave's major advantages is its privacy-preserving nature. Because mmWave radar doesn’t process visual light, what it sees is not personally identifiable. This makes it an ideal choice for monitoring where a camera would not be appropriate.

Radar sensors happen to be usually rich in detail, but highly anonymizing, unlike cameras. If your doppler radar data leaked online, it’d be hard to be embarrassed about it. No one would recognize you. If cameras from inside your house leaked online, well…
– Chris Harrison, Future Interfaces Group

Another benefit of radar is that it can see through plastic and other materials. This makes it a great choice for consumer electronics because it can be placed underneath the enclosure.

Radar is also immune to environmental factors such as light, temperature, and dust which allows it to see in environments where a camera would be blind.

It's extremely reliable. There's nothing to break. There's no moving parts. There's no lenses. There's nothing, just a piece of sand on your board.
– Ivan Poupyrev

The chart below shows how radar compares with the most common ADAS sensors: cameras, ultrasonic sensors, and lidar. Radar performs very well across almost all criteria with the exception of detection resolution. Lidar and cameras process infrared and visible light, respectively, which gives them higher resolution.

The real power of mmWave sensor data comes when it is processed using machine learning. The rich sensing data can be used to create accurate machine learning models capable of classifying various activities. For example, researchers at Carnegie Mellon University were able to accurately classify different activities such as waving, squats, cycling, clapping, lunges, and jumping jacks using doppler data.^[3] Other researchers have been able to classify a variety of complex materials using mmWave and machine learning.^[4]

mmWave radar applications

Google Soli

The first mainstream mmWave radar project was developed by Google's hardware invention studio, ATAP (Advanced Technology & Projects). Project Soli was first presented at Google IO in 2015 by Ivan Poupyrev where he showed how mmWave radar could detect subtle hand gestures. These touchless gestures could then be used to extend the user interface of something like a smartwatch.

It offers a third dimension of interaction, which compliments and enhances other interaction modalities, such as touch screen and voice input.
– Ivan Poupyrev

In October 2019, Soli shipped in the Pixel 4, marketed to consumers as "Motion Sense." It provided users with a faster face unlock along with some basic touchless gestures like controlling music. Google must have decided it was still too early as they haven’t included Soli in any of the Pixel phones since.

However, Google is still very committed to using its Soli technology. The most recent application is in the second-gen Nest Hub which launched in 2021 with "Sleep Sensing."

Sleep Sensing uses Motion Sense (powered by Soli low-energy radar technology) to analyze how the person closest to the display is sleeping, based on their movement and breathing — all without a camera or wearable.

Over the holidays, the Nest Hub was selling for $60 which shows the technology has become cheap enough for mass-market consumer devices. A teardown of the device shows the radar chip from Infineon (who helped co-develop Soli) costing $3.65. And through ongoing research, Google is continuing to improve its sensing algorithms.

Amazon 3D Motion Detection

Amazon is next in the tech giants to introduce mmWave radar into their products. At the beginning of 2021, they launched the Ring Video Doorbell Pro 2 with mmWave radar-powered capabilities marketed as "3D Motion Detection" and "Bird's Eye View."

In the announcement, Ring's founder, Jamie Siminoff explains why consumers will love it:

3D Motion Detection and Bird’s Eye View are about giving you more control and context at home. With 3D Motion Detection, you’ll receive smarter, more relevant alerts thanks to the radar sensor, which pinpoints precise information related to your motion alerts, including exactly when and where visitors travel on your property through Bird’s Eye View.

Amazon’s new doorbell shows that mmWave radar is more capable than a passive infrared (PIR) sensor and is also a powerful companion to a camera. The internet is full of articles and forums discussing false alarm problems caused by the PIR sensors. As mmWave radar becomes cheaper and more ubiquitous, Ring will be able to eliminate all of these false alarms. This is because radar is not affected by environmental factors like light and temperature and can distinguish between objects, instead of simply detecting motion. And as a companion to a simple camera, radar can see through things like trees and bushes as well as provide a 3D mapping to the camera’s 2D video stream.

Amazon is also following Google’s lead in integrating radar for sleep sensing.

Apple mmWave-powered HomePod

Apple is also working on adding mmWave radar to their products, but as you might expect, they are working behind the scenes as they wait for the technology to further mature and can release a polished implementation.

In early 2021 they were granted a patent for an Electronic device with circular radar-antenna array which shows what would likely be a HomePod using a 360-degree mmWave sensor array to understand how nearby users are interacting with it. This would give it a better understanding of context and user intent which might include detecting gestures, allowing the user to control the device without having to yell "Hey Siri!"; detecting the user's identity based on their gait, height, and physical profile; and determining a user’s medical condition by measuring their vital signs.

Office space monitoring

Since the pandemic, businesses have had to rethink what the future of the office will look like. One startup working on solving this problem, Density, provides real-time occupancy monitoring using privacy-preserving radar technology. Their product, Open Area, uses ceiling-mounted mmWave radar sensors that claim to accurately monitor human presence and cover up to 1,325 square feet per sensor.

Car safety

Car manufacturers are adding mmWave radar for advanced driver-assistance systems (ADAS) as well as occupancy detection. 4D mmWave radar is very valuable for ADAS because, in addition to sensing in 3D, it can also measure the real-time velocity of each person, car, or other objects it sees.

Tesla recently announced they will be removing radar and relying solely on cameras despite Musk previously commenting on how great radar is. However, the radar that Tesla removed had low resolution and we will likely see mmWave radar in future Tesla vehicles for ADAS. We will also start seeing mmWave radar used inside the cabin as a sensor to detect a child left behind in a hot car and to monitor the well-being of the driver by measuring their vital signs.

There are a handful of companies working on radar for cars including Novelic, Arbe, Vayyar, Uhnder, and Oculii (recently acquired by Ambarella for $307M).

Sleep sensing

The big tech giants are not the only ones using mmWave for sleep sensing. Miku is a startup company focused on sleep sensing for infants. Their product combines a traditional baby monitor camera with mmWave radar so that parents can not only see their baby in its crib but also non-invasively monitor its respiratory rate and sleep patterns.

Imaging

Given mmWave radar’s capability to understand very subtle characteristics in the reflected radar signal, it can distinguish between different materials. This enables imaging applications like package inspection and airport security.

mmWave wireless communication

mmWave is also being used for both indoor and outdoor wireless communication. The well-known outdoor application, mmWave 5G, has been advertised for a few years now. All the major cellular providers like Verizon and T-Mobile are working on mmWave deployments as are other projects like Facebook Terragraph and startup Aervivo. The challenge with deploying mmWave internet service is that given the small wavelength of the signals they do not travel very far in comparison to lower frequency radio waves, making deployments very expensive.^[5]

Indoor mmWave communication is a bit more practical because of the shorter distances required. The 60 GHz WiFi standard WiGig has been around since 2009 and has slowly been gaining more traction. Intel, for example, has built a wireless adapter for the Vive VR headset using WiGig that has near-zero latency and can transfer data at an astonishing 4.6 Gbps.

The most recent WiFi standard in development combines techniques in both radar and communications to provide wireless sensing using WiFi routers. A survey on the research already conducted shows that WiFi is capable of becoming not only a communications standard but also a fully-fledged sensing paradigm.

Standardization

mmWave communication has obviously required standardization so that devices can talk to each other, but what about standardization for radar applications?

Earlier this month at CES, the Consumer Technology Association (CTA) announced Ripple, an open API for developing general-purpose consumer radar systems. Participants include Google, Ford, Blumio, Infineon, NXP, and Texas Instruments. With recent advances making radar smaller, lower power, and cheaper, it has become practical to use across a wide range of applications. The lingering problem is the development required to get each unique application up and running from scratch.

Radar sensing systems have historically been designed for single applications, with hardware and software custom-developed for each purpose. This means that traditional product solutions — from the hardware to the final user experience — are bespoke.

Creating a standardized layer of abstraction will reduce development time by leveraging prebuilt sensing libraries. The four main applications that Ripple currently sees are:

• Non-invasive wellness monitoring
• Occupancy detection
• Human activity recognition
• Touchless gesture controls

As radar systems become more standardized across hardware and software, engineers and designers will be able to more easily ‘drag and drop’ these capabilities into new products and we will start to see mmWave radar widely used—both as an alternative to less capable sensors and to unlock completely new experiences and value for users.

Human-computer interaction challenges

Although this all sounds super exciting, the technology is still very early, especially in how designers approach these new user interfaces. One thing we surely want to avoid is this comical scene from The Hitchhiker's Guide to the Galaxy:

A loud clatter of gunk music flooded through the Heart of Gold cabin as Zaphod searched the sub-etha radio wave bands for news of himself. The machine was rather difficult to operate. For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive - you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of muscular expenditure of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same programme.

Technology shouldn’t be added just for the sake of it. Even with all the new technologies available to us, some products still call for a good old-fashioned button with a familiar tactile click.

One of the biggest challenges for mmWave adoption is rethinking how we interact with our devices. Adding a new dimension of interaction presents both a challenge and an opportunity for designers to create intuitive and accessible interfaces. It will also require marketers to educate consumers who may be hesitant to welcome unfamiliar sensors into their lives.

mmWave and the future of ambient computing

The best computer is a quiet invisible servant.
– Mark Weiser

The promise of ambient computing is that technology gets out of the way while becoming more helpful at accomplishing what we want. It anticipates our needs by understanding the context of our environment and situation. It serves us, rather than us serving it. This is what the next evolution in computing is all about. As smartphones untethered us from our desks, ambient computing aims to untether us from our smartphones.

One of the key principles of ambient intelligence is that value = functionality / attention required.^[6] mmWave radar, for example, can track my sleep without me having to wear anything and works silently in the background. It both helps me and demands none of my attention. Undoubtedly, a much better experience than my Reebok Traxtars!

Notes

1. Google seems to be one of the first to have requested a waiver in 2018. Vayyar and Amazon have also requested waivers, citing Google's original waiver.

2. The new dimension in 4D radar is not actually time, as previous radars have been able to measure instantaneous velocity using the doppler effect. The new capability comes in being able to see in 3D space by understanding the elevation and azimuth angle of each object. This is possible because antennas arrays can measure the angle of arrival of the received waveform.

3. CMU researchers show potential of privacy-preserving activity tracking using radar

4. RadarCat: Radar Categorization for Input & Interaction

5. Google did a preliminary study for the US Defense Innovation Board indicating it would cost $400B to provide mmWave coverage to 72% of the US population. Other challenges facing mmWave 5G can be found in this article, The age of mmWave 5G sputters to a dusty death.

6. 1998 Digital Living Room Conference keynote presentation prepared by Palo Alto Ventures.