AI, Misinformation and the Future of Cybersecurity

Issues Cybersecurity Leaders Should Be Thinking About Now

Underwood and Underwood photography, 1912. Public

In the 2010s, a century-old photo of Teddy Roosevelt riding a moose in a lake began circulating on the Internet. It was rather quickly exposed as a fake, with tell-tale signs of fakery that are easy to spot on a close look. But what about today’s AI-generated fakes? They’re improving in quality at an astonishing rate, and there is good reason to fear AI’s potential to supercharge the disinformation and harmful misinformation that already plagues us today. There is also good reason to fear AI’s offensive use in cybersecurity. And in fact, there are important intersections between these two topics. This article explores three such topics, with a focus on the implications for the future of cybersecurity they raise. These issues are:

1. Emotional vulnerability and AI-powered phishing,
2. The inference of private data from public data, and
3. The conflict between ‘break-ins’ and ‘trickery’ as conceptual models for cybersecurity.

The discussion draws on issues raised implicitly and explicitly in the webinars “What the Law Can and Can’t do About Misinformation and A.I.” featuring University of Washington School of Law professor Ryan Calo (1), and “Get Ready for the AI Revolution – Fears, Hopes and Plans for AI in Cybersecurity” (2), featuring AI experts from Enea (Mohamed Amine LARABI), Arista  NDR (Gary Golomb) and Zscaler (Rex Shang and Hanchen Xiong).

Fish and bait in love, ID3942399, Sifis Diamantidis, Dreamstime. License-free image.

In the Get Ready for the AI Revolution webinar, Arista’s Gary Golomb paused during a discussion of malicious prompt engineering (i.e., using clever questions to trick generative AI (GenAI), like ChatGPT, into providing info it is not supposed to) to take the conversation to a “little more scary place.” That ‘scary place’ is how we as human beings respond to GenAI.

As AI is Anthropomorphic Tech, It Can Make Us “Unguarded and Exploitable”

As Gary aptly observes, “people interact with it [GenAI] like it is general intelligence,” by which he means ‘artificial general intelligence (AGI)’ which is a hypothetical form of AI that can think and learn like a human, and is often defined as having self-awareness and the ability to act independently using its own judgment. “We become really childlike with it,” he continues, “which is another way of describing that we become really unguarded with it…that we become exploitable.”

In the Misinformation and A.I. webinar, Professor Calo expresses the same concern. He states people are hardwired to react to anthropomorphic technology like AI as though it were human, even though they know it’s not (consider the way people engage with Siri or Alexa, for example). To illustrate his point, he recaps a case study by Claire Boine about Replika (3), a subscription service that provides users with an avatar of an AI-powered digital companion. With the free Replika service, the companion is a friend. With a paid subscription, the friend can become a romantic partner.

How Deep Can Relationships with AI Partners Go?

As Professor Calo recounts, when regulators in Italy effectively shut the Replika service down over concerns about data privacy and access by minors, paid subscribers became so deeply distraught by the abrupt loss of their beloved ‘partners’ that Replika subreddit moderators provided support resources including links to suicide prevention hotlines. (Subscribers of other social chatbot sites like Forever Voices and Soulmate were likewise devastated by the sudden loss of their romantic partners when those services were halted.)

Implications for Social Engineering Attacks like Phishing

What does this mean for phishing, which is already the type of AI-boosted cyberattack of most immediate concern to cybersecurity professionals (4)? Well, for one, it suggests we need to expand the typical framing of the phishing challenge.
Concerns that phishing emails and SMS texts are becoming better targeted, more credible, and more voluminous though AI enhancements are accurate. However, we should reflect more deeply on how AI’s capacity for emotional connection and manipulation – even deep relationship building – might be weaponized to exploit cybersecurity’s weakest link: human beings.

This might include:

1. Harnessing the emotional connection and conversational capabilities of GenAI to evolve one-off phishing attempts into far more sophisticated, multi-exchange conversations that build connection and trust before making plays to capture credentials or sensitive data.
2. Hacking open source and commercial social chatbot platforms to use for AI catphishing, or hacking the more mundane but rapidly proliferating chatbot assistant systems to exploit a human tendency to overshare with friendly digital assistants.
   Being pulled into private conversations with social media bots that with AI become far more adept at cyber espionage and phishing in addition to disinformation.
3. Being deceived by deep fake voice and video calls which exploit existing trust relationships and are rapidly increasing in number and realism. (5)

Secrets. Creator: Wodzinowski, Wincenty (1866-1940). Public Domain Dedication (CC0 1.0)

Another thorny issue at the intersection of AI, disinformation and cybersecurity is the use of private data inferred from public data for malicious ends (for example, making disinformation more credible by adding tidbits of ill-gotten truths, or using inferred private data to gain unauthorized access to IT resources).

Specifically in this context, ‘inferred private data’ refers to the use of AI to “derive the intimate from the available,” as Professor Calo succinctly puts it, and it is made possible because we are used to moving about the Internet thinking we are leaving a trail of data that “doesn’t say anything too exciting about us,” whereas in reality, he observes, in the era of AI, “we’re telegraphing intimate secrets.”

A recent study by researchers at ETH Zürich confirms this AI capability,6 and explores two forms of such AI-based inference. First, it shows pre-trained large language models (LLMs) can infer personal attributes like gender, race, marital status, location, income, age, occupation, etc., from massive sets of text gathered from the Internet – with high accuracy and stunning efficiency. Second, beyond free text inference, the researchers demonstrate malicious LLM-powered chatbots can be used to extract personal information from users by posing seemingly benign questions in chat sessions.

Concerns about such private data inference tactics are not yet on many cybersecurity radars, even though they pose huge privacy and security threats – and there are no working defenses for them (yet?). Concerns are instead focused on the extraction of explicit private or protected data from memorized LLM training datasets (consider, for instance, recent headlines about how researchers got ChatGPT to expose such data simply by asking it to forever repeat a single word – “poem” – until it essentially went haywire and began haphazardly spitting out private and protected data) (7).

Strategic Implications of Private Data Interference

However, it would be prudent for cybersecurity leaders to begin thinking now about the potential impacts of private data inference (PDI), even if much remains unknown regarding the impact PDI will ultimately have on cybersecurity. PDI could, for example:

1. Supercharge all types of social engineering attacks, including phishing. Whether acquired from existing public data or gleaned from individual chatbot interactions, the personal information that AI can infer could no doubt provide potent fuel to AI-powered social engineering attacks.
2. Make multi-factor authentication more complicated. The system of personal security questions widely used in authentication could falter if inferred data enables a (real or AI) hacker to respond correctly to a victim’s ‘personal’ security questions, or to gather enough personal data to impersonate someone else well enough to secure new credentials for access to new systems.
3. Complicate liability issues for data breaches. Organizations are widely required to provide sufficient safeguards for protecting any ‘personal identifiable information (PII)’ they store, defined as any data that permits the identity of an individual to be reasonably inferred by either direct or indirect means. With AI inference, that definition now seems to apply to ‘all data, everywhere.’ And if a data breach occurs tomorrow, and data that is stolen would have been useless for personal identification in pre-AI days, but is used to infer private data using AI, will the organization be liable for a private data breach?