2024: The Dawn of the AI Arms Race [HP #1]

In January, prospective voters in New Hampshire answered the ringing of their phones to hear the distinct voice of the President:

If you’re knowledgeable on the mechanisms of American politics, you’d know that this claim makes no sense. But when heard through the same voice that gives primetime addresses from the Oval Office, someone could easily be confused. That confusion was exactly the intention of whichever person used ElevenLabs to create an AI-generated spoof of President Biden’s voice.

At essentially the same time, the Twitter/X-sphere was ablaze with the controversy of explicit Taylor Swift photos being disseminated. Naturally, these photos had no basis in reality and were AI-generated fakes, presumably from diffusion models (such as those used by OpenAI) with their safeguards removed.

Deepfakes present incredible risk to our social and informational fabric, which is why even back in mid-2021, I invested into Reality Defender and the incredible Ben Colman. Reality Defender provides enterprise- and government-grade deepfake detection. These two incredibly public incidents of deepfakes are just the forerunners of an upcoming war, not just the war on deepfakes, but what I term the AI Arms Race.

2024 is the Dawn of the AI Arms Race

I’ve had this thesis since 2016, ever since the deep learning craze entered tech zeitgeist. Highly visual examples like AlphaGo, AlphaZero, Boston Dynamics, and self-driving bots showed that computers’ ability to mimic humanity - not just through computational brute force, but through “human-like” intuition - was progressing alarmingly fast.

And when technology gets surprisingly good (read: better than humans), it starts to get used. AI systems (even if as “simple” as a machine-learned classifier) are 1) being increasingly involved in decision-making processes and 2) increasingly connected to critical aspects of financial, economic, and human health systems. One survey by IBM claimed “42% of enterprise-scale companies already had actively deployed AI in their business”; we expect this number to skyrocket. Correspondingly, the incentives to deceive or sabotage these systems are growing.

And for all the exuberance, influx of capital, and outpouring of effort into advancing AI capabilities, there has been a dramatic underinvestment into defending those capabilities from attacks.

Let’s review some potential vectors of attack:

• Social engineering a bank customer service agent through convincing voice synthesis (deepfakes, targeted for economic gain)

• “Poisoning” inputs into the training data of models

• Modifying the appearance of objects that confuses models without confusing humans (think self-driving cars reading a stop sign as a "speed limit 45 mph”)

• Breaking LLMs with inputs that bypass sanitation

• Introducing vulnerabilities or backdoors into open-source models or other open-source infrastructure surrounding those models

• Reconstructing sensitive or private information from models’ outputs (a technique called model inversion)

• Using forensics on models to find failure modes (LLMs have anomalous tokens that can completely break its function, while Loab is a notable example of a diffusion model having some kind of persistent, negative output)

• Doctoring inputs (e.g. to a loan application) to get a particular result from an automated decisioning model

This is just the tip of the iceberg. From here, we can see literally billions of dollars of potential damage that enterprises and organizations will need to protect against.

Defense For & From AI will be One of the Biggest Investment Booms of the Next 5 Years

I plan to maintain a living list of startups working to 1) protect against adversarial use of AI and 2) defend AI systems from adversaries. If you are working in this space or know someone working in this space, please reach out!

• Reality Defender

• DeepTrust

• Protect AI

• HiddenLayer

• Calypso AI

• Lakera

• Nightfall

• AIShield

• Robust Intelligence

Last updated 2/27/24.

Hyperparameter explores the intersection of business, society, and artificial intelligence (AI). Hyperparameter builds and consults about ML, AI, and data-driven solutions for organizations spanning from startups to Fortune 500 companies to investment firms. Kendrick Kho has over 10 years of experience creating and investing in groundbreaking AI-based products.