AI expert Gary Marcus thinks OpenAI will be the ‘most Orwellian company of all time’

• AI expert Gary Marcus says OpenAI may be forced to become a surveillance company to make money.
• AI could be leveraged to supercharge surveillance efforts. It’s a scenario he calls “Orwellian.”
• He said the company’s employees should voice their concerns.

In modern literature, George Orwell’s 1984 is perhaps best known for alerting its readers to the dangers of mass surveillance. The dystopian novel is set in an imagined world where an omniscient “Party” constantly monitors its residents.

AI expert Gary Marcus says the company shaping the global AI arms race is on the cusp of turning what Orwell imagined into reality.

“My guess is that OpenAI is going to become the most Orwellian company of all time,” he said in a discussion with Google veteran Peter Norvig at Stanford’s Center for Human-Centered Artificial Intelligence last week. “What they’re going to be pressed to do is become a surveillance company.”

Marcus is an academic and the author of several books, including “Rebooting AI: Building Artificial Intelligence We Can Trust,” and, most recently, “Taming Silicon Valley.”

OpenAI has grown in prominence, in part, by offering businesses access to its AI models and resources. But Marcus thinks the company won’t be able to earn enough money to support its valuation that way because the technology isn’t advanced enough.

“OpenAI sold the dream of universal AI for all purposes, and in 2023 practically every big company ran pilot studies on that premise,” he told Business Insider in a direct message on X. “But in 2024, a lot of reports from the field are about disappointment: The technology isn’t reliable enough yet for production, because it is plagued with problems like hallucinations and boneheaded errors. So many businesses are starting to be cautious.”

However, there is another way to make money: surveillance. AI can synthesize vast amounts of data quickly, helping whoever can pay for it comb through data — whether that’s a government agency investigating its citizens or a political campaign targeting specific voters.

Marcus suspects that OpenAI will eventually tap into this potential income stream and become a powerful surveillance company.

Concerns that OpenAI’s technologies could be used to supercharge surveillance efforts have been raised before. In June, OpenAI announced that it had appointed former NSA director Paul Nakasone to its board of directors, which drew a deluge of criticism. US whistleblower Edward Snowden called the appointment a “calculated betrayal to the rights of every person on Earth.”

Snowden added at the time in a post on X that the “intersection of AI with the ocean of mass surveillance data that’s been building up over the past two decades is going to put truly terrible powers in the hands of an unaccountable few.”

Marcus urged the company’s employees to voice their concerns and say, “I don’t want to be part of this.”

This isn’t the first time Marcus has criticized OpenAI. After meeting CEO Sam Altman on Capitol Hill last year, he was simultaneously impressed and suspicious, noting that Altman didn’t seem “sincere in his concerns” about the potential for the technology to cause harm.

Earlier this year, he said he still hoped Altman could get the company back on track as a “force for good.” Now, he’s not so sure.

“In the last few months, the mask has really come off, and I seriously doubt he will ever return to the mission,” he told BI, referencing the company’s plan to restructure its business. “The promise to investors to turn into a for-profit basically means that ship has sailed.”

OpenAI did not immediately respond to a request for comment from Business Insider.