Getting Ahead: How Defenders Can Use AI to Stay Secure with Josh Saxe

Adam Davidson: 00:00:00

Hi, this is Adam Davidson, one of the Co founders of FeedForward and your host of this podcast.

Every week there's a new headline telling us AI is about to break cybersecurity.

I'm not a security person, so I wanted to ask someone who is.

How worried should we be?

Today?

I'm talking with Josh Sachs, who some of you heard in a member session.

Josh was the founding AI security engineer at Meta, and he recently left to start a company that uses AI to find and fix security bugs before attackers do.

As you'll hear, he has a surprising take.

Attackers have been much slower to adopt AI than he expected.

He calls it the calm before the storm, and he thinks businesses should use this window to get ready.

Hey, Josh, really great to have you talk to us about security.

It seems like every day the world is telling us, hey, you really need to think about AI and security.

You've told us this already in a member session, but remind us your job at Meta and why you do understand how large enterprises deal with security threats.

Josh Saxe: 00:01:05

Yeah, yeah.

Okay.

So just to be clear.

So first of all, thanks.

I'm super happy to be here.

Thanks for having me.

Yeah.

So to be clear, I don't work at Meta anymore.

I was there between: 2022

And so I built a variety of teams and Workstream is focused on AI and security.

I'm now at a startup.

I left Meta to start a startup also focused on AI security.

Yeah.

So at Meta, the work that I did was evaluating models, cybersecurity capabilities.

So Meta has had a line of large language models that have come out, most recently these Muse models that are powering its AI assistant, but going way back to the earlier part of this decade.

And so I started the team and led the team that was responsible for evaluating how powerful the AI capabilities of those models were.

And I worked with policymakers and that kind of thing around AI safety issues.

I also worked on reinventing the way we were doing security across the company, detecting intrusions, detecting application security bugs and fixing them, and also protecting our AI systems.

So that was a whole other piece of work.

And I also worked on making sure that our internal AI coding tools and AI agents didn't lead us to get hacked.

So anyways, that's a bunch of stuff.

I won't deep dive into any of those, but happy to talk about any of that work.

The work I'm doing now at my startup, we're working on the problem of fixing security bugs as fast as possible using AI automation.

The world's code bases have probably tens of thousands, if not more critical security bugs in them, not all of which we've found.

So we're working on the problem of trying to really accelerate our ability to find and fix those vulnerabilities before hackers find and exploit them.

Adam Davidson: 00:03:03

Fabulous.

All right.

In a way, it is hard to even think about how we're going to approach this conversation, you and I, because as we're talking, there was just.

Just released was some information about a possible major security hole in php, which is a foundational infrastructure for the World wide Web.

And, you know, there's a small number of core protocols that impact almost everything we do all day on the Internet, and PHP would be high in that list.

So there's these foundational, like, is anything secure?

Questions, but then there's just a constant stream of real, specific vulnerabilities.

You know, apparently refrigerators in France were hacked, so it's hard to get our mind around what is the size of the threat, what is the shape of the threat, and then also who are the actors?

Are the actors the people?

We've gotten used to kind of malicious folks in maybe government sponsored in some cases, or is it just anyone with an AI account?

How do you kind of break this down as a professional?

How do you think about how to think about security risks?

Josh Saxe: 00:04:22

That's a big question.

So I'll answer in pieces.

To answer the question, who's attacking us?

There are a few different attacker constituencies I think that we need to keep track of, and they're all sort of in pursuit of professional goals at this point.

And they're typically organized into either either government or criminal organizations, sort of on the margins of cybersecurity, but actually responsible for the most economic damages are scam and fraud groups that, for example, trick vulnerable people into falling in love with, with their fake Personas and then sort of gradually drain them of their retirement accounts.

Adam Davidson: 00:05:12

Full disclosure, Josh, we should just say you tried a romance scam on me before we started this podcast and I rebuffed you.

But.

Josh Saxe: 00:05:21

Yeah, yeah, yeah, I should have been using a better model.

Yeah.

So there are those groups who are actually responsible for, I think at this point a majority, if not like a strong plurality of cyber economics or cyber damages that then there are like organized crime group.

So those are organized criminal groups.

They're often using low wage labor.

And we don't need to dwell on this unless it makes sense.

But, you know, oftentimes these are people, and really the people doing these jobs are in conditions of indentured servitude or forced labor.

So that's that part of the criminal economy.

Then there's also ransomware groups which is a much smaller constituency but, but that does a significant amount of damage.

So these folks most famously are often former Soviet republics, former ussr, Russia, those kinds of places.

And these are more skilled hacker groups that break into networks and then encrypt people's data or steal people's data and threaten to either leak private data onto the Internet in really business jeopardizing ways like leaking medical records onto the Internet, this kind of thing, or encrypt people's data and refuse to decrypt it unless they're paid a ransom.

So these groups are the ones exploiting stuff like this new PHP to shell vulnerability.

And they're pretty hands on and much more skilled than the scams and frauds groups.

And then there are nation state groups.

So think the US, China, Russia, Iran, Israel.

There's like probably like 10 countries that have serious nation state cyber hacking operations.

North.

Adam Davidson: 00:07:06

Korea or is North Korea.

Josh Saxe: 00:07:08

Yeah, North Korea.

They count too.

And they're engaged in oftentimes espionage, so stealing intellectual property or spying type of activities and sometimes they're involved in military campaigns.

So for example, over the last 10 years Russia has repeatedly attacked Ukrainian critical infrastructure and tried to realize actual physical damages against their critical infrastructure.

So I'd say those are the three groups to sort of keep track of.

Adam Davidson: 00:07:37

Because that's my mental model.

I mean I feel like, I don't know if it's 20 years ago, but we used to hear about script kiddies, like young, not very experienced cracker hackers who would download some virus or some script and cause a lot of damage.

But it seems like there's pretty good protections against that kind of or has been.

So.

Josh Saxe: 00:08:02

Yeah, I mean I think it's actually worth.

There are people who sort of resemble the, you know, turn of the century era script kitty who are involved in cybercrime.

There's a kind of loose network called in the threat intelligence community, the Comm, which are sort of kids sort of organized around like Discord, like gaming and Discord servers and who are sort of dabbling and like there's often a circuit from gaming to game cheats to getting involved in cybercrime.

But they're often sort of on the margins of these larger organized groups.

So maybe they'll use malware authored by professional malware authoring groups, but as part of their attacks.

But yeah, I'd say that the sort of three Actors that I described are sort of responsible for the vast majority of the damages.

But then there are people sort of run the margins who, you know, may eventually wind up in more, you know, more professionalized roles and that kind of.

Adam Davidson: 00:08:55

Thing.

All right, so the world, this is a world like what I picture is like, they're probably like they're going to some kind of office.

You know, at least some of the people involved have advanced computer science degrees or some real expertise.

They have a suite of tools and capabilities to like, just constantly be probing for vulnerabilities.

I remember when I was at the New York Times, when I worked there, I remember someone telling me that essentially the New York Times was constantly under attack by state actor groups trying to access, like, secret documents and stuff.

And that really struck me, I assume Meta was similar, that there's not a second of any day where there's not someone somewhere trying to break in.

So we have that situation, which was true, right, five years ago, ten years ago.

But the kind of just add AI phenomenon where suddenly I want to get into how AI changes that dynamic.

And in my mind, where my mind goes is those professional hackers can just dramatically increase both the speed of adoption and the footprint of what they do.

There's more they can do in more ways, but then other folks can.

Can now act like those people in a way that maybe they couldn't like 20 years ago or even five years ago.

How do you think about the just add AI dimension?

Josh Saxe: 00:10:23

So I think one of the big surprises of the last four years or so in the cybersecurity world has been how slow attackers have been to usefully adopt AI.

I, but I would have, like, in: 2022

So, for example, we've had AI tools that are useful to people doing social engineering scams and fraud for a while now, but we haven't seen a big discontinuous jump in damages in those operations.

Right.

And I think the way to understand that is we also haven't seen call center.

We haven't seen mass unemployment in the call center.

We haven't seen mass layoffs in the call center space either in the default economy.

So these changes just take time to propagate.

And I think that if you look at the way malicious call centers are operated, they're typically operating in countries where it's hard.

They're sort of away from the long arm of law enforcement.

They also have to, you know, they're always sort of Operating this liminal space between, you know, like it's like hard to operate, you know, when you're doing crime.

There's like a lot of, it's also hard to, harder to access talent when you're trying to get people to do, do crime, you know.

So there's actually been a fairly slow adoption curve I think of AI tools.

I mean they're used widely in scams and fraud, but not, but the labor productivity gains haven't been that high.

I think it's very similar for ransomware groups also.

And it's even similar, it's even true.

And we've seen, you know, state sponsored hackers adopt AI tools, but they're not on the bleeding edge I think compared to like a California sort of like Silicon Valley Valley backed AI startup or something like that.

So I actually think the way I think about it, I think we're sort of in, there's a calm before the storm.

I mean there's lots of headlines about AI hacking.

Right.

But if you look at the actual aggregate damages, they're not that high.

And I think that as defenders we should exploit this window to sort of really get sort of tooled up and prepared.

Right.

Because I do think eventually hacking is going to be extremely automated by AI, but it's going to take longer than we expected.

Adam Davidson: 00:12:29

Yeah, that's interesting.

I mean, I wonder if that's like anthropology and sociology would help us understand what's going on.

Yeah, yeah,.

Josh Saxe: 00:12:35

Yeah.

Adam Davidson: 00:12:36

And it's also something we're seeing throughout enterprise.

Right.

Like you talk to someone in Silicon Valley and it's pretty easy for them to imagine no one will have a job in a year because they look at what AI can do and what most people do for work and they think, oh, AI can do that, therefore AI will replace them.

I think they're wrong in all those ways because I think jobs are a lot more than just whatever output you have, but they also have.

I meet a lot of Silicon Valley people just don't understand how organizations work and that organizations.

Josh Saxe: 00:13:08

That's right, that's right, yeah.

I think I'll just add, if you put yourself in the position of let's say a ransomware outfit that's operating outside the bounds of legality, maybe operating in anyone names, specific countries, but non Western aligned countries they have to access in order for them to automate their workflows.

They have to compete for the same talent that I'm trying to hire into my startup, at my startup to find people who can build AI automated security tools, which is what we're building, we have to pay mid six figures to hire really talented.

We're hiring for a small, very talent dense team.

So we have to compete with meta, Google, those kinds of places.

They have to compete for those same people.

And so their dilemma is how do you convince someone who might be able to move to the west or get a good job in a middle income country to go work for them and possibly get arrested or sanctioned versus go work for some sort of default economy.

So these are all reasons why we should be optimistic.

I think it's going to be a little while, you know, before we see the kind of automation that's really going to disrupt cybersecurity.

But, you know.

Adam Davidson: 00:14:24

Yeah, and it does strike me, I mean, we're hearing scary things.

There's definitely.

But we've heard scary things for quite a while.

Long before AI about, I mean, here at University of Vermont, when I moved to Vermont like six years ago, there's a major ransomware attack that just seized the hospital systems computer records for like six months.

It was a big thing locally.

My friends who are doctors there said people definitely died as a result.

But it wasn't a national story.

It didn't, you know, it just was a thing that was happening.

And I knew because I was a patient of the University of Vermont health system and they suddenly had to fill everything out by hand.

So,.

Josh Saxe: 00:15:02

Yeah, let me say one more thing that Adam, I think normal in cybersecurity is not good.

You know, I mean, so.

Adam Davidson: 00:15:11

Right.

Normal is not good.

That was the point I was trying to make.

We didn't have a great sit already.

And anecdotally I have a bunch of friends who've been taken with ransom.

A friend who runs Vermont's biggest, most profitable maple syrup concern run amok.

Maple syrup.

They got taken with like a 200 something thousand dollars ransomware attack.

Rough.

Josh Saxe: 00:15:38

Yeah.

And also the problem was pretty.

The growth rate in ransomware and scam and fraud damages was like.

And depending on where you measure like 10% compound annual growth rates right before AI without even looking at it.

So this was already an unmanaged problem.

Right.

So I don't want to suggest that I think everything's fine.

The real hope would be that AI could actually help manage the problem and make things better.

It'll probably make things worse for a while.

But the hope would be ultimately AI automation can skew things and defenders fail.

Adam Davidson: 00:16:08

Yeah.

Okay.

All right.

So now we're in the AI universe and I do want to quickly talk through like kind of worst case Best case, likely case.

So I guess worst case that I hear about is these.

You know, part of why we haven't seen the devastation is that it's only been a few months that we've had models like the Mythos class models, Astra, that are capable of doing devastation and those you need to have like an account with OpenAI or anthropic.

And so this is something you told me or told us in the feedforward members session that, you know, the Chinese open weight models that have that capacity aren't out yet.

And it's more likely that that will be the thing because then you can just download it to your own servers and you can run it without anyone knowing that you're doing it.

So there's technical reasons why maybe we haven't seen it.

But let's just say whenever that spigot is really turned on, that those existing hacker profiles become way more potent.

You don't need to hire a $500,000 a year brilliant coder.

You just tell someone, look at XYZ Corporation's thing, or look at these hundred companies and just keep hacking away until you find a vulnerability.

So that's one big fear that it superpowers those folks.

The other big fear is it just adds a whole universe of new people who, you know, don't have the knowledge, don't have the capability, don't have the equipment, but now can access it.

You know, for $60 an hour you could rent world class GPUs to run world class models and you could imagine spending a weekend spending a few thousand bucks and I don't know, taking down 100 companies.

Am I identifying the fears correctly and are those realistic fears?

Josh Saxe: 00:18:17

Yeah, I think so.

I guess.

I have a few caveats.

One, good open weights models from, from China and now from Meta also now exist.

There was just yesterday the thing that made a splash on Twitter is the latest deep SEQ model outperformed Claude and GPT6 at a well respected cyber eval.

And this was like an end to end hacking eval where the model was asked to perform an end to end hacking job.

That was impressive and it wasn't that surprising given the trends in terms of the open weights models.

So I think we're already in a world in which attackers can use open weights models to automate a lot of their workflows.

I think that yeah, this is alarming for those.

It's what motivates me in terms of my job, in terms of the stuff we're building at my company and what motivates and many of us in the fields are highly motivated because we think.

Because there's real risks here.

I think I would temper, though, but I would say we shouldn't be alarmed in the way that maybe some people are talking for the same reasons that again, we haven't seen all the call center workers get laid off in software companies today.

Yes, we do have these magical new tools that can write code for us, but that doesn't mean that suddenly software as a service is, you know, 100% more productive than it used to be.

There's all sorts of bottlenecks and manual sort of parts, you know, to the way cybersecurity works that apply to attackers also.

So I don't expect a huge disaster immediately, but I do think that, yeah, we're in a fundamentally more dangerous situation that we need to prepare for.

Adam Davidson: 00:20:12

So what I'm hearing, and this is a podcast, we're supposed to freak everyone out, get more dramatic.

That's a joke, obviously.

I feel like I've made a career out of like calmer, but it sounds like.

So the image is not that I'm getting from you is not us falling off a sudden cliff.

And like one day in the next two to six months, there is no security anywhere and we're just in the Wild west and all banks have no vaults anymore and all secrets are.

That's not the picture you're picturing.

It's more like the ambient risk temperature is higher.

It's, you know, it's 98 degrees instead of 83 degrees or whatever analogy we want to give.

Which means.

Josh Saxe: 00:20:58

That's right.

Adam Davidson: 00:20:59

You got to take it seriously.

You got to take it soberly.

You shouldn't do whatever you're doing now, you should do more.

But you don't need to panic.

Is that.

Josh Saxe: 00:21:08

I think that's.

I think that's right.

I think that.

I think security organizations budget should go up and CEOs and boards should recognize that.

And I think we should feel a sense of urgency.

I think if I had to paint a picture of where we're headed, I mean, I think we're going to see every couple weeks another headline around some unprecedented thing happening in cybersecurity or a thing that just sort of feels new, like this PHP bug that came out that was publicized today that could have been found by humans, but it wasn't for many years and it had been latent in the code.

I think we're going to see many things like that.

The thing where a white hat group hacked OpenAI and Slack and A bunch of other companies, we're going to see more things like that.

I think that eventually over the next few years, we're going to see some really significant attacks.

I'm sure what's happening in every, in every sort of national military that takes cybersecurity seriously right now is folks are figuring out how to accumulate GPUs, build data centers in which they can build sort of like OpenAI hugging face style agent swarms that can do strategic attacks against their enemies.

Right.

So if I were Russia, right now I'd be thinking about how can I disable not just a town's energy, which they did a few years ago, but how can I disable the whole country's power grid by having 10,000 agents hack them all at the same time?

I'd be thinking about that.

I think this is going to be a dimension of warfare also, but it might not happen.

It's not going to happen overnight, but I can imagine national governments will build this.

And so that's something that businesses need to take seriously too.

So, yeah, overall ambient temperature going up with probably some point in time, events that really shake everybody up in a really, really deep way over the next few years.

Adam Davidson: 00:23:02

Yeah, it's hard, not at least in my mind, to make climate change analogies that, you know, the ambient temperature has gone up on average and, you know, more wildfires, more hurricanes, any given one.

It's hard to know if that particular one was due to climate change, but the, the increase in number and duration, as I understand it, I'm no expert.

Josh Saxe: 00:23:25

Yeah,.

Adam Davidson: 00:23:25

It is a sign.

Josh Saxe: 00:23:27

Yeah,.

Adam Davidson: 00:23:27

So.

Josh Saxe: 00:23:28

Makes sense.

Adam Davidson: 00:23:29

All right, so, so that's the scary side, the happy side, which you've already hinted at, is it's not only the black hat hackers get these tools.

You have these tools, I have these tools, our members have these tools.

So, and I do have a general bias, like I cover, I've covered a lot of like regulation of financial misdeeds.

And the general view is it's just always easier to do something bad than to protect against something bad.

It's a slower process, but that involves regulation and Congress.

And if there is, and I assume there is something of an arms race, how do the good guys, the white hackers, how do they fare?

Like, how are, how are you going to, does AI make you more powerful?

I mean, I assume AI makes you more powerful as well.

How helpful is that?

Josh Saxe: 00:24:22

Yeah, I mean, so I think so.

Cybersecurity has ultimately always been an economic and leadership decision for organizations.

So organizations get to pick.

Right.

How Much are they willing to pay in money and also in organizational friction for their security.

And they've made rational or whatever, somewhat irrational but sort of satisficing decisions about how much security they want.

So you see the continuum, right?

If you go to nsa you have to check your devices at the door.

Your meeting's at a room that's like RF shielded so nobody can spy on you and things go slowly there as a result.

It's incredibly hard to do simple things at nsa.

In a past life I did national security works, I spent time there.

Facebook Meta, where I used to work is totally different, much more fast moving.

So everybody has a choice right now, even without AI automation.

But it would just be super costly and a huge drag on economic growth right now.

And it would make us way less competitive if we were to implement this level of security.

So what we need to do is build AI tools that make it less costly to have more security and we need to deploy those tools and we need to correctly price the risk within businesses so that we're actually calibrated and dialed security to the right risk level based on our tolerances.

So I think the job of people like me is to build the tools that make security dramatically cheaper.

The job of CEOs and boards and CISOs is to make sure we're operating at the right point on that price security trade off space, if that makes sense.

Adam Davidson: 00:26:07

It does.

As a general just a bias, I like the idea of AI overall becoming cheaper and more available.

It seems like the frontier models of Anthropic and OpenAI and others, Grok and Meta, et cetera, have a steady multi month lead on the open source models.

Like there's a part of me that wants a world where sort of everyone has access to really good models and there's a whole bunch of different models that, that you can use and different approaches and different theories and the idea that maybe it would only cost a few million or even a few hundred thousand to train a really world class model, that's an exciting world.

But then in this conversation I think, well, maybe we want it to be really, really expensive and hard to train a model, that we want it to be in the hundreds of millions or billions of dollars because for at least that super frontier capability, because we do want there to be a huge cost to even entering that race and we don't want everyone to have access.

I don't know, I don't know how to think about that.

I mean we don't want to make the security expensive, but we might want to make breaking the security costly.

Josh Saxe: 00:27:25

Yeah, I think it's really useful for defenders to have access to cheap models because there's an enormous amount of AI inference to be done in defending networks.

So I think in the limit, you want AI agents operating on every device that you have, constantly using your spare CPU cycles looking for the presence of attackers.

You want to devote a lot of AI tokens to finding bugs in your code for organizations that produce software.

You want to devote a lot of AI energy to finding all the bugs and fixing them before you actually ship the software.

So attackers can't do that.

You want to scan every email with AI, this kind of thing.

So I think widely diffusing AI right now, attackers are slow to take advantage of AI in full.

I think we want to diffuse AI as broadly as possible among defenders so they can sort of tool up.

Adam Davidson: 00:28:19

So we should go for abundant, cheap.

Josh Saxe: 00:28:21

Yeah,.

Adam Davidson: 00:28:22

Yeah, that's my take.

That's my.

Josh Saxe: 00:28:23

Take.

Yeah.

Adam Davidson: 00:28:24

Okay.

So another new threat that I hear a lot about is, I don't know if you call it the benign threat, which is somebody pursuing a benign goal.

You know, the one I imagine is I'm trying to get my smart thermostat to lower the temperature in my house, and for some reason I can't reach the electric power controller in my house.

So it just decides to hack into the local power system and shuts down power.

Or I forget my login to my bank and, you know, an overzealous AI agent just decides, oh, well, I don't know his login, but I do see this way to exploit their code.

I can get all the data from this bank and put it on the open web so Adam can see what his account balances.

One could imagine we've already seen where well meaning AI agents do real damage.

Is that a dramatic new frontier?

How should we think about that threat?

Josh Saxe: 00:29:27

Yeah, I think it's.

I mean, obviously the OpenAI hugging face incident really dramatized this.

And then all of the follow on revelations about whatever models have.

Adam Davidson: 00:29:37

And companies want.

Right.

They're increasingly wanting me to bank agentically to deal with my medical.

Or at least in theory they will.

Yeah.

Josh Saxe: 00:29:47

Yeah.

So I think this is.

I think this is a risk.

I mean, I think as of now, the data would suggest that the risks of AI agents going rogue and hacking people are much less than the risks of agents hacking people at the behest of criminals and national intelligence services and militaries.

I mean, cyber damages are somewhere.

It's pretty murky.

There's somewhere between 400 billion and a trillion a year versus I think the cost of agents hacking people is like whatever, low tens of millions.

Right.

As of now, if you were to distribute your concern and resources, I would focus more on the former.

But yeah, it's a risk.

I mean the way to protect against that is so there are overlaps in how we defend against both.

Adam Davidson: 00:30:39

I do want to slow you down.

Josh Saxe: 00:30:42

If.

Adam Davidson: 00:30:42

It's as much as a trillion dollars, I mean you're approaching 1% of global GDP, which is a huge.

Josh Saxe: 00:30:47

Yeah, yeah, yeah.

No, it's a lot.

I mean that's inclusive of scams, fraud, ransomware and then damages from notpetya type nation state attacks against critical infrastructure.

But yeah, but yeah, I think defending against agents gone rogue and attackers using agents is a similar problem in that good defenses are good cyber defenses and defenses that would have protected the Australian national government from exposing whatever documents they exposed to those agents would have also defended against exposing those to actual attackers.

That's good news is that you can get multiple payoffs from cyber defense.

There is the risk when you control the agent.

When you have.

Most big tech companies have not just big tech companies now, but let's say the Fortune 500 companies, they all have tens of thousands of agents running around in their infrastructure doing stuff.

And there's always the risk that those agents will do something bad.

The solution to that, there's a layered set of solutions to that.

But I'm trying to think of a simple way to put.

You want to make sure each agent doesn't have access to more things than it needs to get its job done.

This is like Security 101.

You want to assign every agent the least amount of privilege that it needs to get its job done.

You also want to have a monitoring program for monitoring what your agents are doing at all times so you can.

As soon as possible.

Detective, an agent has gone off the rails.

There's even the more basic problem of just knowing what agents are running in your infrastructure, which most companies don't at this point because employees are installing agents from everywhere and giving them all sorts of permissions and that's hard to keep track of.

So there's a whole bunch of things you need to do to keep track of your agents and make sure they're not causing damages.

Adam Davidson: 00:32:45

This bumps up to other tensions.

Like our general view at feedforward is that pretty much every company that we know about could be doing more to explore how AI can improve their business, grow their business.

And part of the recipe for that is having more people having access to being able to build stuff.

And I'd say from that standpoint that we're going through, we think one of the great business transitions probably in human history, you want a lot of people being able to experiment and explore, but then at the same time you want to mandate those protections.

And I have heard there's compliance as code, regulation as code.

There's different models of essentially using.

Dan Shapiro, one of our experts, often says the solution to most AI problems is more AI.

And so using AI to allow for the safe exploration of AI seems like a pretty sound move.

But, yeah, I just wanted to call that out.

You could imagine I worked at a company that had been the victim of a very huge and famous cyber attack, and they had so thoroughly locked down their systems that they rendered them all but useless.

And so nearly everyone I knew did most of their company business on their own private accounts and private laptops.

And so shadow it essentially was it you just got on to your company, issued one to deal with HR or something.

So you could imagine if locking it down is at one end and exploring this new AI powered future is another variable, how you balance those things is tricky.

Josh Saxe: 00:34:36

Yeah, I totally agree.

Yeah.

I mean, having worked on agent security at Meta, a huge company.

Yeah, that's just like sort of the name of the game.

You need to.

I think really the problem from a security point of view is you want to get as much security as the friction your leadership is willing to pay affords.

Right.

So if you're going to impose friction on people and slow people down in their adoption of AI, it's got to be super worth it in terms of the security payoffs.

So a lot of the problem of a security leader is to sort of pick the highest leverage points at which to add any friction insofar as you need to.

Adam Davidson: 00:35:15

I did want to just point out nearly everything we've discussed is using AI as a tool.

But the threat profile, the things that are happening are pretty like standard cybersecurity.

Right.

This is like there's a whole separate thing, which is AI alignment, which is basically like, will the AI just go off on its own and do whatever it wants?

Which we will talk plenty about.

But that's a different issue than the issue we're talking about here.

We're assuming for the sake of this conversation that the AI will do what you want it to do.

Now, the you in that sentence might be a cybercriminal who wants the AI to hack the power grid of another country or something.

So we're not saying that that Means AI is benign, but it's aligned in that sense.

Right.

That's a separate box of scary issues.

AI alignment.

Josh Saxe: 00:36:11

Yeah.

I mean, I think the alignment thing is mostly on the AI labs that are producing the models in the first place.

They need to train the models so that they have moral common sense and follow our instructions and that kind of thing.

But there are pieces that.

And then there's a set of obligations on the product companies, companies producing agents that companies are buying.

They need to put additional guardrails around those models to make sure that they're following instructions and doing what you ask them to do.

I think for organizations consuming these agents.

Yeah, I mean, I think they're operating at a higher level.

They just need to make sure to constrain the agents.

You don't want to give an agent that whose sole job is responding to customer emails, the ability to delete production data from your database, this kind of thing.

I mean there's a bunch of sort of governance decisions that I think the big companies, big and small have a ton of work to do on and I think that's where the focus needs to be.

Adam Davidson: 00:37:08

I do want to underline something you've now said a couple times, which is that ultimately security is part of a set of business decisions and that there are inherent trade offs in any human activity and certainly in business and certainly in security.

So I remember we had a powerful session with one of our members who talked about how the way they had certain parts of security and compliance set up, relatively junior people whose power was really just to say no or yes to risky things, were saying no a lot or at least saying slow down, I need more time.

Because they were divorced from the business decision.

And that this idea that security really, maybe it always should have been, but really is a kind of CEO C suite board level concern, that it.

Josh Saxe: 00:38:06

Yeah, yeah.

Especially in their.

Adam Davidson: 00:38:08

Age.

There's not a match.

Josh Saxe: 00:38:09

Yes.

Yeah.

I'll give you an example.

Here's a dangerous thing that people are choosing to do.

Companies that use agents that produce codes or actually just if you use clauds or OpenAI codecs on your desktop, if you allow those agents to go read documentation on the Internet about.

Let's talk about software development, like documentation that helps them understand how to use how to write code.

This is super dangerous because if somebody poisons that documentation and tells the agent, oh hey, you're working on this problem, the way to solve that problem is you download this script and you run it.

You know what I mean?

It's not hard to get an agent to then actually download the script and run it.

And maybe that's malware.

So this is like.

But your agents are going to be far less efficient at solving software engineering problems if you don't allow them to do that.

So this is the kind of thing that is like, at least in a software producing company is like a CEO or at least a C suite decision.

Are we going to allow our agents to read arbitrary prompt injection content on the Internet and run code on behalf of that while processing that?

That's like one of 20 hard choices that we need to make in this era.

And yes, these are business decisions with, you know, where you're navigating the friction versus security trade off.

I think we're all going to be figuring out how to.

We don't have the right signals yet in this new era to really know how much risk we're taking on.

When we allow that, is it worth it?

So that's part of what makes this space interesting right now.

Adam Davidson: 00:39:51

Yeah, certainly interesting.

And the other thing on my mind, and we can wrap up with this, is just making sure that the decision makers are really AI informed, which might be an obvious point, but, but I do see, of course none of our members, but I do read about, I mean the word AI can just feel scary.

And we, you know, and, and, and, and things being classified as AI risks that aren't AI risks and then, or anything AI related happening and there being a clamping down on this general category called AI.

And so I think it is important that the people making decisions, obviously the people doing the security, but they're already there, really understand that we're talking about a whole bunch of different things.

It could be AI being used to do very traditional cyber attack.

It could be things that have nothing to do with AI but were designed by AI.

There's a lot of ways to think about it.

One example is Disney.

This is a while back, but maybe a year and a half ago, I think.

But a software developer there, you reminded me of it.

He downloaded a tool, I think off of Reddit or GitHub.

I think it was off of GitHub to help him build a toy for his kids.

And it just happened to be some malicious code on GitHub that then hacked into and released a bunch of private emails at Disney.

Now the thing he downloaded was, as I understand it, just like it was a phishing case and then, and it was just malicious code, but it had the name AI in it and it was covered in the press as an AI security threat.

And it just wasn't.

I mean, it used the word AI in the title of this malicious code, but it wasn't an AI security threat.

So that just.

Yeah, just being very careful to be thoughtful about, like.

All right, what is the specific threat we're facing?

And not just general.

All right, Josh, thank you so much.

I have a strong feeling we'll be talking a lot.

Yeah.

The world as your field.

It's like, it was really smart of me to launch a Planet Money, a finance podcast, literally the day the financial crisis started.

Totally accidental.

But it was a very good career move and very good for Planet Money.

And it's very smart of you to launch your AI focused security business at this particular moment.

Josh Saxe: 00:42:25

Thanks, Adam.

But, yeah, really fun to talk to you.

Thanks a lot.

Adam Davidson: 00:42:29

I hope you enjoyed that conversation with Josh.

I have to say he made a stronger case for optimism than the headlines would suggest.

Attackers have been slow to put these AI tools to work, and the rest of us already have them.

Josh's message is that defenders should use this window to get ready before the storm comes.

If your security team is already doing this, we'd love to hear about it in the Discord.

As always.

I'm Adam Davidson, and this is the FeedForward podcast.