Generative AI in the Real World: Chloé Messdaghi on AI Security, Policy, and Regulation

Chloé Messdaghi and Ben Lorica discuss AI security—a subject of increasing importance as AI-driven applications roll out into the real world. There’s a knowledge gap: Security workers don’t understand AI, and AI developers don’t understand security. It’s important to be aware of all the resources that are available. Make sure to bring everyone together to develop AI security policies and playbooks, including AI developers and experts. Be aware of all the resources that are available; we expect to see AI security certifications and training becoming available in the coming year.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Points of Interest

• 0:00: Introduction
• 0:24: How does AI security differ from traditional cybersecurity?
• 0:44: AI is a black box: We don’t have transparency to show how AI works or explainability to show how it makes decisions. Black boxes are hard to secure.
• 2:12: There’s a huge knowledge gap. Companies aren’t doing what is needed.
• 2:24: When you talk to executives, do you distinguish between traditional AI and ML and the new generative AI models?
• 2:43: We talk about older models as well. But security is as much about, What am I supposed to do? We’ve had AI for a while, but for some time, security has not been part of that conversation.
• 3:26: Where do security folks go to learn how to secure AI? There are no certifications. We’re playing a massive catchup game.
• 3:53: What’s the state of awareness about incident response strategies for AI?
• 4:15: Even in traditional cybersecurity, we’ve always had an issue of making sure incident response plans aren’t ad hoc or expired. A lot of it is being aware of all the technologies and products that the company has been using. It’s hard to protect if you don’t know everything in your environment.
• 5:19: The AI Threat Landscape report found that 77% of the companies reported breaches in their AI systems.
• 5:40: Last year, a statistic came out about the adoption of AI-related cybersecurity measures. For North America, 70% of the organizations said they did one or two out of five security measures. 24% adopted two to four measures.
• 6:35: What are some of the first things I should be thinking about to update my incident response playbook?
• 6:51: Make sure you have all the right people in the room. We still have issues with department silos. CISOs can be dismissed or not even in the room when it comes to decisions. There are concerns about restricting innovation or product launch dates. You have to have CTOs, data scientists, ML developers, and all the right people to ensure that there is safety and that everyone has taken precautions.
• 7:48: For companies with a mature cybersecurity incident playbook that they want to update for AI, what AI brings is that you have to include more people.
• 8:17: You have to realize that there’s an AI knowledge gap, and that there’s insufficient security training for data scientists. Security folks don’t know where to turn for education. There aren’t a lot of courses or programs out there. We’ll see a lot of that develop this year.
• 10:13: You’d think we’d have addressed communications silos by now, but AI has ripped the bandaids off. There are resources out there. I recommend Databricks’ AI Security Framework (DASF); it’s mapped to the MITRE ATLAS. Also be familiar with the NIST Risk Framework and the OWASP AI Exchange.
• 11:40: This knowledge gap is on both sides. What are some of the best practices for addressing this two-sided knowledge gap?
• 12:20: Be honest about where your company stands. Where are we right now? Are we doing a good job of governance? Am I doing a good enough job as a leader? Is there something I don’t know about the environment? Be the leader who’s a bridge, breaks down silos, knows who owns what, and who’s responsible for what.
• 13:24: One issue is the notion of shadow AI. Knowledge workers go home and use things that aren’t sanctioned by companies. Are there specific things that companies should be doing about shadow AI?
• 14:24: Someone will be using those resources regardless. If we push an agenda that isn’t working, employees will bypass it, causing data leaks and breaches. The question is how do you improve things? If AI helps them to do a better job, support them but have governance policies. They should know what they can and can’t share. Make sure to ensure accountability.
• 16:00: Can you give us an update on regulations? Give us a sweeping tour.
• 16:41: The EU AI Act is risk based; it classifies AI based on risk levels. There are rules on safety, transparency, and human oversight. There are also rules against doing things that are biased or harmful. There’s required disclosure for interacting with chatbots.
• 17:47: The AI Act seeks to boost AI adoption. There are critics who say that it may limit innovation. There is also an argument that the US has promoted the anti-AI Act narrative to devalue European startups.
• 18:35: When we’re talking about the US, the Biden administration has focused on investing in AI. The US wants to become the AI superpower. There is concern about China’s growing use of AI. China has more proprietary knowledge (patents).
• 18:58: The Biden executive order about AI is voluntary and subject to discontinuation by the incoming administration. What is locked in place?
• 20:16: Because the US has no federal legislation, states have been taking it on themselves. We have the Colorado AI Act, which is risk based, similar to the EU AI Act. This is starting in other states. California’s AI legislation didn’t go forward, but a New York legislator is proposing taking the “good parts” of this legislation to New York.
• 21:15: There are rumors about what will happen with the Trump administration. Maybe they’ll ensure that there’s a federal law—but the Trump administration isn’t likely to restrict companies. They may remove the Biden executive order.
• 21:55: Historically, when it comes to regulation, companies end up planning for the strictest rule. Is that the expectation here?
• 22:27: It’s possible. It will be interesting to see how things develop.
• 22:58: Are executives now serious about AI risks and compliance? Is this a board-level issue?
• 23:15: This has been an issue since last year: We need to be concerned about regulations. Many CISOs are very concerned, especially if every state has its own legislation.
• 24:06: If we follow the money, what’s the budget for AI security and compliance?
• 24:21: That’s tricky; we haven’t had a massive breach that has taken over the news. Do we need it? CISOs want to be proactive. There’s concern that we aren’t investing enough in security. But a breach will happen. We have to be proactive rather than reactive.
• 25:25: It’s become easy to build a deceptively appealing prototype. There’s an imbalance between the ease of building something and being told to take time.
• 26:09: Legal signoff will become more of a thing for AI applications moving forward.
• 26:37: That will have everything to do with cybersecurity insurance. It will be up to insurers.
• 27:00: What are some things on your radar that the general public is not paying attention to?
• 27:14: It’s important to be aware of everything related to AI. We should have more conversations about AI ethics. It’s important to take a look at the bipartisan US House AI Task Force report.
• 28:10: Globally, I recommend checking out the OECD AI policy hub. There’s also the World Economic Forum Presidio AI Framework. And check out OWASP, MITRE ATLAS, DASF, and the NIST AI Framework.
• 29:33: AI is inherently an interdisciplinary team sport. This is a technology trend where you need people from different backgrounds and skill sets.