It is a nice piece of technology. It is a 60Ghz millimeter-wave radar. It is a privacy nightmare. It is already shipped.
Radar uses electromagnetic waves (like Wifi but higher frequency) so it can go through walls, and even typical range for gesture recognition is less than meter, It probably can go at least 10 times as far by boosting the gain of the amplifier, it is not constrained like a theremin would be because it is already working in the far field region of the antenna.
Because it work at such high frequency (but not so high that it can still go through walls), it has many very small antennas arrays, and sense sub millimeter movements even from far away. It also has beam-forming capabilities, meaning it can focus the direction in which to sense. Because it is radar, things which moves are of interests and are filtered easily from the background.
Typically this piece of technology already can or will soon be able to : sense how many humans are around it, where they are, how fast they breath, how fast the heart is pulsating, who they are by computing some heart-base ID.
It is low-power and always-on, 360° with focus-able attention. It is cheap because it can be made on a chip. (Edit: fixing typos)
Not sure what to think about it. Seems awesome at first glance but then the examples are skipping songs and hand waving pokemons. Feels a lot like a solution looking for a problem.
I'd love to see PC devkits for this device. Something as simple as a USB 3 device you plug in and place under your monitor. Perhaps it will make its first "PC" appearance in a Google Chromebook?
I can see a lot of eccentric users figuring out interesting ways of integrate many of these gestures into their workflow. Perhaps for navigating in 3d space or switching between workspaces?
[edit] There used to be a developers page which showcased that devkits exists. http://web.archive.org/web/20181110202503/http://atap.google... Showcase video https://www.youtube.com/watch?v=H41A_IWZwZI
As is often the case, this is some very interesting technology, but for now, we’ll only see it used in some novelty applications.
An increased level of spatial awareness for phones will be huge in the coming decade. However, it will almost certainly be a result of sensor fusion between a Soli-like radar sensor, a FaceID-like ToF sensor, enhanced positioning and pose detection, RGB cameras, microphones, and a lot of ML to assemble a comprehensive picture of environmental context and user intent.
Radar is one more piece of the puzzle in building products that can read the same cues we naturally use to communicate with other humans: Imagine, instead of telling a voice assistant “Alexa, turn down the volume,” where you have to use a phonetic trigger, and all the system has to go on is audio, something more natural: You look in the direction of the hardware, say “turn it down a bit,” and make a pinching gesture with your hand. The system can assemble all these pieces (you were looking at it, you spoke in its direction, you gestured) and, with a sufficiently-trained neural network, make a more conclusive determination of your intent.
Those growing up with Remote Controls of the kind: https://i.imgur.com/AIqz63k.jpg
After a few days, the user develops a muscle memory of sorts. User doesn't even have to look at the controller and all actions (and feedback) are executed through the tactile interface. From cockpits to nuclear power plants to home tv remote control, there is absolutely nothing that replaces physical buttons, encoders, sliders and toggles.
I haven't formally studied UI/UX, but these are important:
- Feedback for an action
- Predictable steps to take an action (Muscle memory)
- Fast response
- Expose current state (sliders, toggles do this)
There should be 0% ambiguity or the user gets frustrated. Any piece of technology that puts impedance in this process is no fucking good. User shouldn't have to "guess and wait" whether the device recognized their gesture to swipe. A physical button guarantees that the action was performed by the means of feedback. Nope, sound feedback or taptic stuff still isn't as good as the click of a button. It can be but no one engineers it well. For example MacBook Trackpad that "clicks" without moving is excellent. Seeing touch screens (one exception, phone), cap buttons, gesture controls, etc everywhere makes me sad because it has nothing to do with UX but everything to do with the bottom line (cost) and marketing, and in this case perhaps better ad tracking? I will put this in plain words - Don't trust a company that sells ads at the same time as building hardware. Either sell ads or sell hardware, not both. Google already serves software which relies on trading off privacy (even if it is anonymized). When it comes to hardware, I freak out and no way in hell this thing sits in my home.
The "dial" gesture is incredibly subtle. Kudos to them if it works reliably.
MKBHD doesn't seem convinced in the efficacy of the sensor* from his Pixel 4 video: https://youtu.be/sKJ4i7p-o-4?t=326
* from Analemma_'s comment below, the Pixel 4 doesn't seem to be running the full blown chip
I just watched MKBHD's Pixel 4 video and he said the wave gestures worked maybe 10% of the time. If that is true I hope it is a bug as shipping something like that should never happen.
Interestingly, this caused the Pixel 4 to get banned in India: https://www.androidpolice.com/2019/10/15/india-wont-get-the-...
I wonder how this will impact fingerprinting of users from a privacy and security perspective. It would be useful as a means of identity verification based off of physical properties of the user beyond just their fingerprint or iris. Yet it would also be a massive privacy concern, particularly since it advertises 360° sensing.
I'm Italian, and I can tell you us Italians have waited for this for at least two thousand years!
We can finally stop pretending our sounds are the language, and get back to using only gestures :)
Oh my god, extraordinary technical feat, but do not want. Seriously, someone go out there and charge $1000 for a dumb 50" television. Give me a phone that doesn't have an assistant or this Soli and I will give a premium for it.
Gonna require "the cloud" so they can collect free machine learning model training data and spy on us, I assume?
It would be nice to bring "hover" back to touch applications. You can have hover with a mouse but not with your finger. Good for tool-tip style help.
Relevant cross post from Bret Victor and hands: http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...
This technology seems promising in the face of his ideas.
They should use this technology to charge Youtube video ad views depending on how many people are watching the ad on a phone. Multi-user presence will be interesting to see how it gets used as an API.
Slightly off topic: Where can one find these minimalist cartoons/sketches of people for websites? Ideally free, but those options appear super cartoonish and limited.
Mods: can we change the headline to include a few words on what a soli is?
So, um, where is the technology that lets my phone float in free space as I execute these gestures, like in the video?
How does waving your hands in the air "feel more human" than using your opposed-thumb-equipped hands to interact with some material object?
> Private
> Soli is not a camera and doesn’t capture any visual images.
This is a privacy misconception that really needs to die (something also discussed in the W3C ambient light sensor thread recently on HN frontpage). Sensing involves privacy implications. By. Definition.
Remember the gorilla arm.
I think a killer app for this could be small screens. You could have a UI with pseudo-"buttons" around the edge of the screen large enough to see but too small to actually press, letting the interactions take place just off screen. This could let you turn a 2x2 inch screen like a watch into one that's effectively a couple inches larger in terms of the interactions available. Right now my watch's design is almost entirely constrained by having UI elements big enough to easily touch, taking up precious screen space.
Google finally ships a research project in a real product and people don’t seem to understand the usefulness.
Human nature for some reason often devolves to “what’s the point?”
Here’s the 2015 announcement. There’s a lot of potential. Maybe we’d all benefit from “imagine what’s now possible “
why no one talks about health and safety of this technology?Not that it will cause any issues, but interested and curious about any peer reviewed studies on how prolonged exposure of 60GHz with close proximity to the body will affect? If there is none, I would love to jump on to the ship. Because, it is so cool! if there is something or have no evidence, then the community or the experts should talk about it and push for proper studies before 10 years from now folks exposed with Soli got cancer!
I've tinkered around with gesture recognition with the Magic Leap quite a bit for art projects. I hope this gets released as a standalone product with an SDK!
I think this is better for something like a Nintendo. To put it in a phone is kinda invasive. I would hope that at minimum the chip can only be specifically activated by the user in specific apps. To just have it on all the time is kinda crazy. Are they going to disclose when they can identify WHO is using it rather than just multiple bodies, etc...
I think this could be synergetic with voice commands. It wont be useful on its own but sometimes you dont want to say every command out loud, so doing a little annoyed wave feels more natural. Soli on its own is not going to sell phones in my opinion but maybe Google is building up to a larger more aware device that can ultimately guess your entire mood, behavior and position in the room and do various things based on that. Like a sort of virtual person that would detect if you look worried and then it asks you how you are feeling today. That's a futuristic direction that I can see this going towards. For example, one usecase that I would probably like is a phone that tracks my posture during workouts and if it's really advanced, it could give me advice or fire me up on my last reps. Like a personal trainer. Right now it seems we aren't anywhere near that but this is my guess for their strategic direction.
So now Pixel phones have the perfect spy chip and they call it "private" because it doesn't record images. But then they don't say what data it records and what they do with it. What they do have is boatloads of stupid buzzwords for the dumbest phone user imaginable. That doesn't bode well at all.
I might start sweating if it takes that much energy just to move to the next track.
Edit: Not sure why this got voted down? Hold your arm in an upright position, swat to the side a few hundred times, you would feel at the very least, muscle fatigue.
For repetitive tasks, you want efficiency of movement, not dramatic movement.
Note at the bottom:
> 1) Pixel 4’s Soli implementation does not include all capabilities and interactions described here
This seems an awful lot like the eye/head tracking in the Amazon Fire phone and that went over super well.
This type of input makes sense in certain applications/platforms but a phone isn't one of them - at least not with current phone usage patterns.
Why is this needed? I can see there are some niche needs like when you are cooking and cannot touch the screen. But that is what voice is for. What exactly is soli solving? Looks like over engineering to me, simply because they can.
The screen is right there but you want me to not touch the screen to interact.
I'm curious is there are any health studies done around this, what about the dangers of EMF radiation when it's on all the time so close to your body, not to mention in your pocket all the time.
This doesnt make sense for phones. But when it becomes reliable, it will be a good usecase for smartwatches. It already has less screen area and limited option for button input. So gestures makes sense.
Lots of negativity in the comments here, but I personally will love to have a way of interacting with the phone even when my fingers are wet or dirty.
> Soli is aware of your presence.
I can't wait until I get my official Google-Amazon Big Brother living room portrait with built-in Soli and Alexa
>Motion Sense will be available in select countries.
Just...why? What is the point of region locking this? It's only hardware, isn't it?
In an earlier iteration of the web page they had really cool examples of controlling sliders, radial dials, panning surfaces, highly specific pinch gestures etc.
All of those have been removed and now it's just waving. It's cool tech and requires solving lots of hard problems, but it's a shame it falls so short of the initial vision.
I was impressed when I saw it back in 2012. It took them a while to ship it into an actual product.
The technology is amazing, as seen in previous demos, but how much of it is actually possible in the phone? The video is very underwhelming, showing simple wave motion that was possible using IR, Motorola had it even before smartphones over a decade ago.
All I want is a small Bluetooth device (maybe wrist-mounted) with a few buttons, a slider, some sort of rotary sensor like a mouse wheel, and accelerometer. Then give me complete control in my phone over how the software interprets the actions.
It will be so fun to watch my 2 year old play with this . If it’s good we’ll be happy if it’s bad it’ll be like the iOS 7 upgrade from 6 when my other then 2 year was so frustrated she threw the ipad down the street shattering into many parts
I could see this being extremely useful in the health space, since you could interact without touching anything and transmitting bad bugs.
Meanwhile, deep in the bowels of Google...
if (handwave.slows > 10%){
do: flag: arthritis
do: alert: advertisers }This is a beautiful piece of tech for any ad company looking to collect reaction information to ads (like google):
* It works at low power. Turning on the front camera and processing the visual data generated needs large amounts of power.
* It can't be blocked as easily as a front facing camera.
* People are less aware that they are being surveilled, which makes them less motivated to block it.
* Yes, it's not capturing any facial expressions, but the information generated probably still gives valuable cues to advertisers.
Also, get ready for unskippable ads pausing when you aren't looking into the direction of the smartphone.
> Motion Sense will evolve as well. Motion Sense will be available in select countries².
What this has to do with the country?!
I wish Google would figure out how to do speech-to-text without mistakes before they start dabbling in radar.
I wish they could use this to turn off touch input on the screen when I'm not looking at the phone.
This reminds of the cellphone surveillance system from that one Batman movie. Scary stuff.
I wonder if that's proximity sensors or always-on gesture recognition doing image processing off the front camera.
Either way, gesturing left/right/up/down doesn't seem to add more than touching the device. My Moto G6 has that now, using the fingerprint sensor as a directional swipe button, love it.
The sensor sucks because it's small. Plus as a radar it's subject to regulatory requirements. Google should have gone with a lidar-based solution, but the concept of putting lidar in a cellphone is probably heavily protected Apple IP.
Wonder if this was created exclusively in house or did Google ATAP do this again....
Just touch the screen. It's already on your hands!
I can't take Google seriously. Every time they add a new feature, my mind goes "and then they will sell you ads on this geospatial information".
Good lord... this is a really cool piece of technology and half the comments here are just complaints that it will enable better ad tracking or erode privacy or that touch screens are good enough.
This is HN. This is cool technology. Can we just stop to appreciate what cool new interfaces or game concepts we might be able to build with this, rather than jumping on the knee-jerk Google hate train?
I honestly don't get why you'd need a motion sensor on a handheld device. Let alone the limited application of such an input, you'd literally need to keep the device stationary for the motion sensing to be effective. It's a usability nightmare.
> https://www.youtube.com/watch?v=KnRbXWojW7c
This ad not only fails to sell the feature but also makes using the touchscreen look easier.
This is just a solution looking for problems.
Unless it can help making my dream come true - of being able to open up my workspace (basically all apps in my laptop/desktop - IDE/browser/VLC/File Explorer.. basically my entire laptop OS UI) in thin air anywhere (as AR with glasses/goggles or projected hologram screens a la Iron Man) and allow me to work with a small trackpad and keyboard, it's just another toy innovation.
Why can't the Iron Man type interface be a reality by now? Meta AR seemed to have a grip on it, not sure if they've made an affordable commercial version that can support OSs out of the box - Ubuntu/Windows etc so that users can continue work in their workspaces right after putting the glasses on.
Basically I want to be able to code or watch youtube anywhere, in any comfortable position - lying down or on toilet seat or in a car.
I'm sure the answer is no but can I buy just the chip?
I really like the gesture controls in my bmw. They seemed odd first but now they feel quite natural.
I guess the best intuition comes from the fact that one gesture is mapped to one action unlike in touch screens where e.g. swiping has a platitude of semantics in different contexts.
Anyways I hope the sensor is released to the DIY crowd. I guess there is a good chance since all the tech comes from infineon and they already built small dev boards and google normally is rather open minded towards DIY.
I go to the digital world for abstract.
Why the fuck do I want real world interactions within the digital one? To slow me down?
If I want to move my hands I'll go play a game of sports in the fresh air.
"Private because we don't capture images" So they think their consumers are morons
It feels like a marketing gimmick vs a useful application. Probably will be cancelled in a year or two.
Consumers need to boycott this stuff. This chip has one purpose: surveillance. All the "features" on this press release are contrived and useless, but they are hoping it's shiny enough that we like it and want it.
I recently showed some videos of Soli in the HCI class I teach. Students immediately hit upon the two major issues I wanted to discuss (I was pretty proud!).
The first is learnability. A big problem with gestures is that there is no clear affordance as to what kinds of gestures you can do, or any clear feedback. For feedback, one could couple Soli's input with a visual display, but at that point, it's not clear if there is a big advantage over a touchscreen, unless the display is really small.
The second is what's known as the Midas touch problem. How can the system differentiate if you are intentionally gesturing as input vs incidentally gesturing? The example I used was the new Mercedes cars that have gesture recognition. While I was doing a test drive, the salesperson started waving his hands as part of his normal speech, and that accidentally raised the volume. Odds are very high Soli will have the same problem. One possibility is to activate Soli via a button, but that would defeat a lot of the purpose of gestures. Another is to use speech to activate, which might work out. Yet another possibility is that you have to do a special gesture "hotword", sort of like how Alexa is activated by saying it's name.
At any rate, these problems are not insurmountable, but it definitely adds to the learning curve, reliability, and overall utility of these gesture based interfaces.