What exactly is the use-case for a laptop or a tablet with ML acceleration? I can see the need in embedded devices; a self-driving car is a good example. Large-scale AI services are going to run in a datacenter. Who exactly is the target consumer for a laptop or tablet with an NPU?
Look at Apple for some on device examples of this.They’ve been using their ML silicon for photo editing, accessibility stuff, image classification, language, etc.
You’re basically completely wrong about how AI is going to scale. We’re not going to be stuck on, say tinkertoy models on our phones and gigantic mega-models exclusively in the cloud. That’s insane. We have good language models that will run on an ordinary laptop already. You can scale models to more or less any size and there is amazing research coming out constantly regarding how to do AI more efficiently. People are running tons of ML code on their PC’s already and the demand will only go up as companies – like Microsoft – bundle more and more features that rely on AI code into their software, more SDK’s and libraries come out that support it, etc.
Also, the feasibility of deploying AI code to more users goes up the more users have them in their devices.
Also, the main trick of NPU’s is efficient matrix math, especially the use-case of applying a single operation to entire matrices at once, which AIUI is foundational to tensor math. Plain old CPU’s are trash at this because they have to iterate over each individual entity in the matrix and apply the operation separately. NPU’s, as I guess they’re coming to be called, are designed to do those operations massively in parallel. There are likely tons of applications for this beyond just ML code that we haven’t even imagined yet.
It’s a bit like asking in 1995 what the use case for a graphics card is when you can go to an arcade and gameboys exist. At that exact moment in time, based on the exact cards that were available in literally 1995, it might have been hard to imagine that by 2024 we’d all have dedicated graphics chips of some kind in our computers – in fact, we’d be hard-pressed to imagine devices without them – and that some of the biggest computing companies in the world would be graphics card manufacturers. Yet here we are.
You have to pay attention to the research as it develops, and you have to realize that they don’t just show up to markets to satisfy pre-existing demands, they create markets and create new demand where none existed before. That’s how the tech industry works.
What exactly is the use-case for a laptop or a tablet with ML acceleration? I can see the need in embedded devices; a self-driving car is a good example. Large-scale AI services are going to run in a datacenter. Who exactly is the target consumer for a laptop or tablet with an NPU?
A person who wants to filter out noise from calls but doesn’t have an Nvidia GPU
A person who wants to translate things, but doesn’t want to send work documents to the internet, etc.
I’m sure Microsoft will respect their privacy…
Who said anything about using Microsoft?
The NPU is inside the machine whether you use windows or Linux
It’s the Microsoft surface. And those NPU optimised networks from Microsoft aren’t available for Linux (yet)
I have no interest in the networks themselves, I want the NPU to use with my programs
With what drivers
There’s a push on mobile phones, too, which are even more battery-constrained.
I don’t know if I’d call it “AI” so much as just “parallel computation”.
For phones, maybe better local speech recognition.
Look at Apple for some on device examples of this.They’ve been using their ML silicon for photo editing, accessibility stuff, image classification, language, etc.
You’re basically completely wrong about how AI is going to scale. We’re not going to be stuck on, say tinkertoy models on our phones and gigantic mega-models exclusively in the cloud. That’s insane. We have good language models that will run on an ordinary laptop already. You can scale models to more or less any size and there is amazing research coming out constantly regarding how to do AI more efficiently. People are running tons of ML code on their PC’s already and the demand will only go up as companies – like Microsoft – bundle more and more features that rely on AI code into their software, more SDK’s and libraries come out that support it, etc.
Also, the feasibility of deploying AI code to more users goes up the more users have them in their devices.
Also, the main trick of NPU’s is efficient matrix math, especially the use-case of applying a single operation to entire matrices at once, which AIUI is foundational to tensor math. Plain old CPU’s are trash at this because they have to iterate over each individual entity in the matrix and apply the operation separately. NPU’s, as I guess they’re coming to be called, are designed to do those operations massively in parallel. There are likely tons of applications for this beyond just ML code that we haven’t even imagined yet.
It’s a bit like asking in 1995 what the use case for a graphics card is when you can go to an arcade and gameboys exist. At that exact moment in time, based on the exact cards that were available in literally 1995, it might have been hard to imagine that by 2024 we’d all have dedicated graphics chips of some kind in our computers – in fact, we’d be hard-pressed to imagine devices without them – and that some of the biggest computing companies in the world would be graphics card manufacturers. Yet here we are.
You have to pay attention to the research as it develops, and you have to realize that they don’t just show up to markets to satisfy pre-existing demands, they create markets and create new demand where none existed before. That’s how the tech industry works.