AI top dog Jensen Huang is confident that engineers can and will tame AI’s risks. That confidence is misguided.
Nvidia boss Jensen Huang has been aggressively pushing back against AI alarmism recently. Even as AI leaders rallied behind a call from Dario Amodei to pace the development of frontier AI models, Huang dismissed the concerns as unfounded scaremongering. Insisting that there are sufficient guardrails in place, he laments the “doomer narratives” that get in the way of deploying the world’s most powerful technology.
Huang’s main argument is that AI is a software product and AI safety is therefore an engineering problem. In his view, recent incidents of rogue agents hacking companies and governments are common growing pains of a technology that’s making the transition from research to application.
“In the beginning of any industry, most of the engineering goes toward making the product work. As the product becomes mature, most of the engineering is dedicated to verification: testing, evaluation, verification, benchmarking, attesting to standards, making sure that the product that they ship is reliable,” Huang said in an interview with CBS News. In other words, AI labs are bound to shift resources toward testing and verification to make sure their products are safe. After all, no company would release an unsafe product.
Personally, I find the notion that companies always put safety first to be laughable. But that’s a topic best discussed elsewhere; let’s stick with Huang’s engineering angle.
Can testing truly establish safety in an open-ended environment? An agent that can write code on the fly, access the internet and interact with people and other agents essentially presents an infinite number of scenarios to test. And can verification methods even keep up with such a fast-moving technology? What about AIs manipulating the verification process, as has been reported recently?
Huang doesn’t pretend that these issues are easy to solve, but he’s convinced engineers can and will solve them. Although I like the can-do attitude that betrays his own engineering background, I’m not convinced. Even though recursive self-improvement is in its infancy, it has already produced emergent behavior from models that we simply don’t understand. That’s not an engineering issue; that’s science. In a matter of months, vast uncharted territory has been bestowed upon computer science.
While researchers piece together what’s happening, we can’t do root cause analysis. And if we can’t do root cause analysis, we can’t make AI truly safe. As a former AI safety officer at OpenAI said in the Ezra Klein podcast: “It’s not that we haven’t got the resources or we’re not trying hard enough – we don’t know how.”
Given the mind-boggling capabilities of AI, that just won’t do. I’m no AI doomer, but good engineering starts with understanding the system you’re trying to control. Until we get there, Huang’s engineering bravado looks more like wishful thinking.
Image credit: Nvidia
