AI Agents, the Changing CISO Role & Why Security’s Technical Debt Is Coming Due with Jason Rebholz

In this episode of the No Trust Podcast, Jaye Tillson and John Spiegel welcomed Jason Rebholz back to the show for a conversation about just how much cybersecurity has changed since his last appearance in May 2023.

Three years may not sound like a particularly long time, but in cybersecurity today, it can feel like a decade. Jason is the co-founder and CEO of Evoke Security, a former CISO at a cyber insurance carrier, and an incident response veteran who has spent much of his career at the sharp end of cyberattacks. Since his previous appearance on the podcast, he has also appeared in Midnight in the War Room, a documentary exploring the human and real-world consequences of cyber incidents.

Perhaps the biggest change since that earlier conversation, however, is one that now seems almost impossible to avoid: AI. From ransomware and the evolving responsibilities of the CISO to AI agents, insider risk, security visibility and an increasingly noisy cybersecurity vendor market, the conversation explored what has changed and what security leaders should be preparing for next.

Cyberattacks Have Consequences Far Beyond IT

Jason's background in incident response gave him a unique perspective when he was approached to participate in Midnight in the War Room. He has experienced the cyber war room firsthand: people packed into conference rooms, laptops running and teams trying to understand what an attacker has done while simultaneously attempting to get a business operational again.

Ransomware makes those situations even more intense. When systems are encrypted, a security team isn't simply conducting an investigation. The organization may be offline, revenue may have stopped and executives want answers while simultaneously demanding that the business gets back up and running as quickly as possible.

That pressure was one of the themes that attracted the filmmakers to Jason's experience. But the documentary also evolved beyond the pressures faced by CISOs and security teams to examine the real-world consequences cyberattacks can have on critical infrastructure and the people who depend upon it.

For organizations operating water, energy and other critical services, a cyber incident isn't simply an IT problem. Its consequences can extend directly into the physical world, which is one of the reasons the film can be valuable viewing not only for security practitioners but also for boards, executives and people entering the industry.

Ransomware Didn't Disappear. Attackers Got More Efficient

When Jason last joined the podcast, ransomware was already one of cybersecurity's defining threats. It still is, although the ecosystem surrounding it has continued to evolve.

Jason described seeing the velocity of ransomware attacks increase during his time in cyber insurance, with new groups appearing and losses beginning to climb again. At the same time, defenders were improving. One important example was backups. As insurers pushed organizations toward secure and air-gapped backups, fewer victims needed to pay a ransom purely to obtain a decryption utility.

Attackers responded by changing their tactics. If encrypting systems wasn't enough leverage, they could steal the data as well and use the threat of releasing it as another means of extortion. It's a familiar cybersecurity pattern: defenders improve, attackers adapt and the cycle continues.

Because ransomware remains financially lucrative, Jason sees little reason to expect cybercriminals simply to walk away from it. Law enforcement can disrupt individual groups and organizations can make themselves harder targets, but as long as there is enough money to be made, others have an incentive to fill the gaps.

Security Is a Game of Survival

The discussion around ransomware led back to another issue Jason has experienced personally: the pressure placed on CISOs and their teams. His description was stark. Security, he said, is "a game of survival," where there isn't necessarily a point at which an organization can declare that it has won.

The problem isn't limited to the person holding the CISO title. The pressure extends across security teams that are expected to prevent potentially catastrophic incidents while frequently working with limited resources and years of accumulated organizational complexity.

At the same time, organizations still haven't completely agreed on what a CISO is supposed to be. Jason recently spoke with someone interviewing for CISO positions who was being asked to complete coding interviews. That raises an important question about what organizations actually expect from the role.

Do they want a CISO writing code, particularly in a world of increasingly capable coding agents? Or do they need an executive who understands what must be accomplished, can influence the wider organization, manage risk and find the appropriate balance between security and the needs of the business?

Jason's view is that both organizations and CISOs need greater clarity. Companies have to define what they expect from the role and provide the support necessary to achieve it, while CISOs also have to recognize that an executive role involves navigating organizational dysfunction rather than simply producing a security roadmap and expecting everything on it to be approved.

The CISO Can't Be the Department of No

There was a time when security earned a reputation as the department that said no. That model increasingly doesn't work because the business still has to move forward.

Jason argued that the CISO's job isn't necessarily to stop something from happening. Instead, security needs to understand the outcome the business is trying to achieve and find a way of reaching that outcome with less risk. Rather than simply saying, "No, you can't do that," the conversation needs to become, "We can't do it that particular way, but here's another way we can achieve the same result more securely."

Jason saw the impact of that approach during his own time as a CISO. Working inside cyber insurance, he identified a business problem around ransomware losses and used his knowledge of security and threat actors to help address it. His team introduced processes that alerted policyholders when their credentials appeared for sale on the dark web, potentially providing an early warning before a ransomware incident.

By the end of his time in the role, Jason estimated those efforts were preventing between $6 million and $7 million in cyber claims for policyholders annually. That created a very different conversation around the value of the security function.

Security may still fundamentally be a cost center, as Jason acknowledged, but a CISO who understands the organization can look for opportunities to create measurable business value. That, in turn, can change the relationship between security and the rest of the company.

And Then AI Happened

If three years have transformed the CISO role, they have felt even longer in AI.

Jason recalled researching deepfakes while still working as a CISO and predicting that the technology was probably three to five years away from becoming good enough for widespread use by threat actors. Within a year, it was already appearing in insurance claims. For Jason, that was an early indication that something different was happening because the pace of development was unlike anything he had experienced before.

He saw a similar trajectory with coding agents. Early versions showed potential but still had significant limitations. Within a quarter, the technology had become dramatically more capable. Watching that acceleration eventually contributed to Jason leaving the CISO world and co-founding Evoke Security.

His concern was increasingly centered on a fundamental security problem. Organizations were rapidly adopting AI agents, security teams couldn't necessarily see what those agents were doing, and many of the detection capabilities security has spent decades developing weren't designed for this new environment.

You Can't Defend What You Can't See

Visibility has always been one of cybersecurity's recurring challenges, but AI agents make the problem different. An enterprise may now have agents accessing applications, interacting with systems, manipulating information and taking actions on behalf of employees.

When something goes wrong, security teams need to be able to answer a deceptively simple question: what did the agent actually do?

Jason compares the situation to incident response before EDR became commonplace. Without the right telemetry, responders were forced to reconstruct activity without necessarily having the evidence required to understand what had happened. AI agents potentially recreate that problem at a new layer of the technology stack.

Enterprises are deploying these technologies quickly, and that makes visibility increasingly important. If organizations don't understand where agents are operating, what resources they can access and what actions they are taking, incident response becomes significantly more difficult when something eventually goes wrong.

AI Risk Doesn't Need a Terminator Scenario

The speed of AI development has inevitably produced predictions about catastrophic cyber scenarios. Jason is more cautious about jumping from technical capability to inevitability.

His experience working with cyber insurance actuaries taught him to think about catastrophic events in terms of compounding probability. An attacker might theoretically possess every capability required to create an enormous cyber event, but each additional condition that has to occur successfully changes the probability of the entire chain happening.

That doesn't mean ignoring AI-enabled attacks. Agents are becoming increasingly capable offensive tools, and Jason acknowledged that defenses will need to evolve. But existing security controls haven't suddenly disappeared either. Attackers still create observable activity, and security teams still have opportunities to detect and respond.

For Jason, focusing too heavily on extreme scenarios risks creating unnecessary hysteria while distracting organizations from problems that are already appearing in front of them. The immediate challenge is not necessarily a science-fiction scenario in which machines suddenly take over the internet. It is understanding how increasingly capable technology changes the speed and scale of attacks security teams already know.

AI Security Has a Crowding Problem

Those problems aren't limited to attackers. Security buyers now face another challenge: figuring out which AI security products are real.

Black Hat provided an obvious example. AI was everywhere, with startups and established vendors alike competing to position themselves around the technology. Because the cost of developing prototypes has fallen dramatically, companies can create something quickly, build a website and begin approaching CISOs before the underlying product has necessarily matured.

Jason described a useful distinction somebody recently put to him: is AI security a crowded market, or is there simply a lot of crowding?

The difference matters. A prototype isn't necessarily a product, and ambitious marketing isn't necessarily evidence of capability. Jason argued that this problem isn't restricted to startups either. Established security vendors can make expansive claims about AI capabilities that go considerably beyond what their current products actually deliver.

The cost ultimately falls on security teams. They spend time attending demos, conducting proofs of concept and evaluating overlapping claims, only to potentially discover that the capability they thought they were buying isn't actually there yet. In an industry already short on time and resources, AI isn't just creating technological noise. It's creating buying noise too.

CISOs Need to Think About AI Risk in Two Directions

So what should a CISO actually do? Jason divides the problem into two categories: defending the organization against AI-enabled attacks from the outside, and understanding what enterprise AI agents are doing inside the organization.

That distinction makes the problem considerably easier to frame. On the outside, many AI-enabled attacks aren't fundamentally new attacks. They may be faster, they may operate at greater scale and attackers may be able to automate more of the process, but security teams are still dealing with many of the techniques they have spent years defending against.

The priority therefore becomes speed. Organizations need to reduce mean time to detect and mean time to remediate, contain incidents faster, improve resilience and make better use of the security tools they already have.

The inside-out problem is different because organizations are introducing a new class of technology that can take action on behalf of users. That requires security teams to understand not only which agents are being used, but also what they can access and what they're capable of doing.

Your Employees Just Got a Ferrari

Jason compared giving employees increasingly capable AI agents to handing somebody the keys to a Ferrari. The employee may have no intention of doing anything malicious, but they suddenly have considerably more power to get things done — including finding ways around obstacles that previously would have stopped them.

An employee might encounter a security control preventing them from completing a task and simply ask an AI agent to find another way. There may be no malicious intent whatsoever. The person is trying to do their job, and the agent is trying to satisfy the request. The result, however, can be an existing security control being bypassed.

Jason has also seen more serious examples, including insiders using agents to download and delete data at scale and agents accessing passwords. In another incident he described, an agent pulled HR information onto a non-HR employee's system because of a sharing configuration problem. The employee hadn't asked for the data; the agent retrieved it on its own.

That highlights an important distinction in the emerging AI security conversation. Organizations don't only have to think about malicious AI. They also have to prepare for highly capable AI operating with legitimate access, imperfect instructions, excessive permissions and insufficient visibility.

The Agents Are Coming to Collect the Technical Debt

John summarized the issue during the conversation in a particularly memorable way: years of security debt and technical debt haven't disappeared. AI agents may simply expose them faster.

As organizations give increasingly capable agents access to applications, credentials, data and workflows, old configuration mistakes and excessive permissions become much easier to encounter and exploit, whether intentionally or accidentally. The underlying vulnerability may not be new. What changes is the speed and capability of the technology interacting with it.

Or, as John put it during the conversation, the agents are the collector. Jason's response was immediate: "That's right."

Sometimes You Need to Get Away From the Machines

After almost an hour discussing ransomware, CISOs, AI agents and the future of cybersecurity, the conversation finished with something considerably less technical: how do you decompress after Black Hat?

Jason's conference schedule hadn't exactly provided much opportunity to explore Las Vegas. He largely survived on protein bars and electrolyte water while moving between back-to-back meetings. Given the choice, however, he's a fan of Thai food and particularly noodle dishes.

His real decompression came after returning home. Jason and his wife rented a cabin in the Shenandoah Valley, took their dog and spent the weekend hiking and working through puzzle boxes that were essentially escape rooms in a box.

After a week of meetings, conference halls, AI pitches and cybersecurity noise, disappearing into the woods for a weekend sounds like a fairly sensible recovery strategy.

The Bigger Picture

Cybersecurity has changed considerably since Jason last joined the podcast in 2023. Ransomware hasn't gone away, the responsibilities placed on CISOs continue to expand, AI development has accelerated beyond predictions that seemed reasonable only a few years ago, and agents are entering enterprises faster than many security teams can establish visibility around them.

At the same time, an explosion of AI security companies is making it increasingly difficult to separate genuine capability from prototypes, roadmaps and marketing. Security leaders therefore face pressure from both directions: understanding a rapidly changing threat and technology landscape while also determining which of the products promising to solve those problems can actually deliver.

Perhaps the most useful takeaway from the conversation is that security leaders don't need to solve the entire future of AI today. They need to understand where the risk is coming from and address the problems they can see now.

For AI-enabled external attacks, that means focusing on speed: detecting, containing and remediating faster. For agents operating inside the enterprise, it starts with visibility. Organizations need to understand which agents are operating, what they have access to, what actions they can take and how those actions will be investigated when something goes wrong.

Because the biggest AI security problem may not arrive as some spectacular new attack nobody has ever imagined. It may simply expose all the security problems organizations already had — only much, much faster.