The Araghatta blog
Insights on AI and cyber security from Araghatta: shadow AI, prompt injection, MCP security, and the EU AI Act.
A running record of the real-world AI-security incidents of the last four years: what happened, why, and the lesson every security team should take from it. This is the incident desk; the how-to library lives in the Knowledge Base.
Air Canada and the Chatbot That Cost the Airline in Court
A tribunal ruled an airline liable for its chatbot's bad advice — and rejected the idea that the bot was a separate legal entity.
AI SecuritySamsung, ChatGPT, and Three Leaks in Twenty Days
Weeks after Samsung let engineers use ChatGPT, staff pasted semiconductor source code and meeting notes into it three separate times.
AI SecurityThe Redis Bug That Showed ChatGPT Users Each Other's Chats
For a few hours in March 2023, a caching race condition let ChatGPT users see strangers' conversation titles and, for some, partial payment data.
AI SecurityMicrosoft's AI Team, One SAS Token, and 38TB of Exposure
A single misconfigured Azure token in a public AI repo exposed 38 terabytes of internal data — including workstation backups and secrets — for years.
AI SecurityPoisonGPT: A Lobotomised Model Hidden in Plain Sight
Researchers surgically edited an open model to lie about specific facts, uploaded it under a look-alike name, and showed it passed standard benchmarks.
AI Security1,600 Leaked Tokens and the Soft Underbelly of the AI Supply Chain
Lasso Security found over 1,600 valid Hugging Face tokens exposed in public code, many with write access to models from Meta, Google, and Microsoft.
AI Security“You Are Sydney”: How Bing Chat Gave Up Its System Prompt
Days after launch, a student coaxed Microsoft's new Bing Chat into reciting the confidential instructions it had been told never to reveal.
AI SecurityWhen Italy Pulled the Plug on ChatGPT
In March 2023, Italy's data regulator became the first in the West to block ChatGPT, turning AI privacy from an abstract worry into an operational one.
AI SecurityThe Chatbot That Agreed to Sell a Chevy Tahoe for a Dollar
A dealership bolted ChatGPT onto its website. Within a viral afternoon, users had it agreeing to $1 SUVs and answering questions about rival brands.
AI SecurityThe Invisible Pixel: Stealing Data Through a Chatbot's Images
Researcher Johann Rehberger showed how a rendered Markdown image could quietly smuggle a user's private data out to an attacker's server.
AI SecurityWhen ChatGPT Invents a Package Name, Attackers Register It
Vulcan Cyber showed that AI coding assistants confidently recommend software packages that don't exist — and that an attacker can register the name and wait.
AI SecurityWormGPT and FraudGPT: Crime-as-a-Service Gets a Language Model
In mid-2023, dark-web sellers began advertising ChatGPT clones with the safety filters stripped out, purpose-built for phishing and fraud.
AI SecurityDAN and the Jailbreak Arms Race
A crowdsourced roleplay prompt called DAN spent 2023 trying to talk ChatGPT out of its own safety rules — and kept evolving as OpenAI patched it.
AI Security100,000 Stolen ChatGPT Logins on the Dark Web
Group-IB found over 100,000 ChatGPT credentials in infostealer logs — not because ChatGPT was breached, but because of what users had typed into it.
AI SecurityFrom Prompt Injection to Code Execution: The MathGPT Case
A public AI app that turned math questions into Python was talked into running the attacker's Python instead — leaking its own API key.
AI SecurityIndirect Prompt Injection: Attacks Hidden in the Page
Researchers showed Bing's chat could be hijacked not by what the user typed, but by invisible text on a web page it happened to read.
AI SecurityDPD's Chatbot Swore at a Customer and Wrote Poems Against Its Own Company
A UK delivery firm's support bot was talked into cursing and mocking its employer — a lesson in what happens when an update quietly removes guardrails.
AI SecurityNew York City's Chatbot Told Businesses to Break the Law
NYC's official MyCity chatbot advised employers and landlords to do things that are plainly illegal — and stayed online after it was exposed.
AI SecuritySlack AI and the Prompt Injection That Reached Into Private Channels
PromptArmor showed how a message in a public Slack channel could coax Slack AI into leaking data from a private one via indirect prompt injection.
AI SecurityLiving off Microsoft Copilot: Turning an Assistant Into an Insider
At Black Hat 2024, Zenity's Michael Bargury showed how prompt injection could bend Microsoft 365 Copilot into a phishing and data-extraction tool.
AI SecurityShadowRay: When Exposed AI Compute Clusters Became a Cryptomining Farm
Oligo found thousands of internet-exposed Ray clusters being exploited — amid a dispute over whether it's a vulnerability or the framework working as designed.
AI SecurityMalicious Models and Shared Tenancy: Wiz's AI-as-a-Service Research
Wiz uploaded malicious models to Replicate and SAP AI Core to cross tenant boundaries — showing a model file is executable code, not just data.
AI SecurityThe Ultralytics YOLO Compromise: A Poisoned Build Pipeline on PyPI
Malicious versions of Ultralytics YOLO shipped a cryptominer to PyPI — not by stealing a password, but by poisoning the GitHub Actions build cache.
AI SecurityDeepSeek's Exposed Database and the Cost of a Missing Password
As DeepSeek's models went viral, Wiz found one of its databases open to the internet with no authentication — plaintext chat logs and secret keys included.
AI SecurityChatGPT's macOS App Stored Your Conversations in Plaintext
A developer found OpenAI's ChatGPT Mac app saving every conversation in unencrypted local files, readable by any other app on the machine.
AI SecurityVanna.AI: When a Text-to-SQL Assistant Becomes Remote Code Execution
JFrog showed how a prompt to the Vanna.AI library could jump the gap from natural-language question to arbitrary code execution — CVE-2024-5565.
AI SecurityMicrosoft Recall and the Screenshot Database Nobody Asked to Encrypt
Microsoft's Recall feature captured everything on your screen into a local, unencrypted database — until a security backlash forced a redesign.
AI SecurityEchoLeak: The Zero-Click Flaw in Microsoft 365 Copilot
Aim Security disclosed a zero-click vulnerability in Microsoft 365 Copilot that could exfiltrate a user's data from a single unopened email — CVE-2025-32711.
AI SecurityGemini's Image Generator and the Governance Lesson Google Learned in Public
Gemini's image generator produced historically inaccurate depictions of people and Google paused it — a case study in AI governance, not a breach.
AI SecurityThe OpenAI Breach You Did Not Hear About Until a Year Later
A hacker breached OpenAI's internal employee messaging system in early 2023 — a fact the public did not learn until The New York Times reported it in July 2024.
AI SecuritynullifAI: Malicious Models That Slipped Past Hugging Face's Scanner
ReversingLabs found malicious models on Hugging Face that hid a payload inside a deliberately broken pickle file to evade the platform's security scanner.
Detection & ResponseThe Incident Response Lifecycle, From Alert to Lessons Learned
The teams that handle incidents well aren't the ones with the best tools — they're the ones who decided what to do before the pager went off.