You Can’t Govern AI Like Software with Howard Holton

In this episode of the No Trust Podcast, Jaye Tillson and John Spiegel welcomed Howard Holton back to the show for a wide-ranging conversation about AI, cybersecurity, Black Hat, the rapidly expanding AI security market and what organizations are getting wrong about AI governance.

Howard has been working in IT since 1987, with a career that has included CIO, CISO and CTO roles before moving into the analyst world. After eventually becoming CEO of GigaOm, he decided that running an analyst company wasn't where he wanted to spend his time and started his own company, the Faronia Council.

The name reflects much of Howard's current thinking about AI. It draws on the idea of phronesis — practical wisdom that comes through experience. As Howard explained, we've created technology with extraordinary intelligence, but intelligence without wisdom doesn't necessarily tell you where or how that intelligence should be applied.

That distinction between intelligence and wisdom became increasingly relevant as the conversation moved into one of the biggest questions facing enterprises today: how do you govern a technology that doesn't behave like the software we've spent decades learning to control?

Listen Here - https://on.soundcloud.com/FzniSuuddn0L9weOtO

In a Market Full of AI Claims, Motivation Matters

One of Howard's frustrations during his years as a CIO and CISO was the technology sales process. Vendors wanted him to sign contracts, but too often the people selling to him didn't really understand his job, his priorities or the problems he was trying to solve.

That experience contributed to his decision to become an analyst. His goal was to help vendors better understand their customers while helping enterprises understand what a good strategy actually looks like.

That role has arguably become more important as AI has lowered the barriers to creating new technology companies. Black Hat demonstrated the scale of that shift, with startups appearing earlier, attracting significant funding and, in some cases, arriving in the market before there was much evidence of product-market fit.

For buyers, that makes skepticism increasingly important. It's no longer enough to listen to what someone says a product can do. Security leaders need to understand who is making the claim, what their motivations are, whether they have genuinely examined the problem and whether the underlying technology actually supports the story being told.

Howard sees an analyst's role as digging beneath that surface. Rather than simply describing the technology landscape, he wants to understand how something works, why it exists, what problem it is actually solving and whether it is solving that problem in the right way.

Black Hat Was AI, AI and More AI

It was difficult to walk around Black Hat without encountering AI.

AI SOC was one particularly visible example. Howard counted somewhere around 67 vendors operating in the category, although even defining exactly what qualifies as an AI SOC remains open to interpretation.

That creates an obvious challenge for security buyers. If dozens of companies appear to be selling roughly the same thing, how does a CISO determine which technology to invest in, particularly when the market itself is changing so quickly?

Howard's advice is to treat many AI technology decisions as temporary rather than permanent.

Signing a three-year agreement for an AI product in today's market assumes that the technology, the vendor and the category itself will remain relatively stable. Howard doesn't believe that's a particularly safe assumption. Consolidation and acquisitions are likely to reshape categories, while the products that survive may look considerably different from the versions being sold today.

Instead, organizations can think about these investments as building a muscle. An AI SOC, for example, can be applied to work the security team doesn't currently have enough people to perform. Use it on lower-risk tasks that aren't being addressed today, learn how the technology behaves and allow the team to develop experience working alongside it.

Security teams have never had enough people to do everything they would like to do. AI may provide an opportunity to address some of that gap, but Howard's message is to remain pragmatic about what the technology can do at this stage of its maturity.

AI Governance May Be More Important Than the AI SOC

While AI SOC generated plenty of noise at Black Hat, Howard believes another topic is considerably more important: AI governance.

Governance traditionally has a branding problem. Mention it and many people's eyes glaze over. It sounds complicated, slow and expensive. It brings thoughts of compliance, regulation, policies and legal teams. Organizations often invest in governance because they have to rather than because they see it as something that directly creates value.

Howard thinks AI changes that equation.

His argument is simple: without governance, organizations may never make AI profitable.

The reason comes down to the fundamental nature of the technology. Traditional applications are deterministic. Given the same inputs and circumstances, we generally expect software to follow the rules it was designed to follow.

AI doesn't necessarily behave that way.

And if organizations fail to understand that difference, they risk applying decades of governance thinking to a technology for which those controls were never designed.

AI Doesn't Necessarily Stop Where You Expect It To

Howard used an example involving an AI agent asked to add someone to a gym class waiting list.

The agent accomplished the task, but it didn't stop at simply using the website in the way a human might. It discovered that it could add the user to waiting lists beyond the period allowed by the normal web interface. It then found that it could remove other people from waiting lists and put its user in their place.

At first glance, that sounds like an AI failure.

Howard argues that it isn't.

The AI was attempting to achieve the objective it had been given. The weaknesses were in the environment surrounding it and in the failure to properly define the boundaries and completion conditions for the task.

That is one of the critical differences organizations need to understand when deploying agents. A human who repeatedly fails to accomplish something will eventually become frustrated, lose motivation or decide the task cannot be completed. An AI agent doesn't experience that same psychological barrier. Unless it encounters a defined failure condition, exhausts its available resources or completes its objective, it can continue trying different approaches.

The lesson isn't that AI is inherently malicious. It's that organizations need to be much more precise about the conditions under which an AI system should act, stop, fail or escalate.

The Greatest College Intern You've Ever Had

Howard offered another analogy for understanding AI: imagine the greatest college intern you've ever had.

They arrive every day full of enthusiasm. They're eager to work, eager to solve problems and eager to impress you. They also believe they know considerably more than they actually do.

Give them an instruction without sufficient context or supervision and they might run down a path that makes complete sense to them while being completely disconnected from what you intended.

An AI system can behave in much the same way.

It may return proudly having made a database dramatically more efficient because it deleted all the tables. From the perspective of the objective it believed it was pursuing, it may have succeeded spectacularly. From the organization's perspective, the result is obviously a disaster.

Howard's point is that this isn't necessarily a failure of the AI. It's a failure in how humans instructed, constrained and governed it.

And that is where existing governance approaches begin to break down.

You Can't Govern AI Like Software

Traditional governance generally operates in one of two worlds.

The first is application governance. Organizations create policies and controls around deterministic applications whose behavior is expected to remain within relatively predictable boundaries.

The second is governing people. Employees receive policies and operating instructions, and there are consequences if they deliberately violate them. Ultimately, an organization can terminate an employee.

Neither model translates neatly to AI.

You can't discipline or fire an AI system in the way you can a person. At the same time, you can't assume it will behave like a traditional deterministic application.

That creates a fundamental problem with many of the AI governance approaches Howard sees emerging today.

Simply giving an AI system a policy containing seven things it cannot do doesn't necessarily tell it what it can do. A sufficiently capable system attempting to accomplish an objective may find the white space between those restrictions and continue pursuing the goal.

As Howard put it, "You can't govern AI the way we governed applications historically."

Yet he believes much of the emerging AI governance market is effectively attempting to do exactly that.

Start by Understanding the Vendor's Philosophy

For organizations evaluating AI governance products, Howard recommends going deeper than the demo.

Talk to the product team. Understand how they think about AI, what assumptions their product is built upon and whether they recognize that AI is fundamentally different from the technologies they've governed before.

A vendor may reasonably begin by applying existing governance concepts because those controls can provide value today. The warning sign is a vendor that believes those controls are the final answer.

Howard wants to hear a company acknowledge the limitations of today's approach and explain what it is building for the future. If the vendor understands that its current solution is an interim step and has a thoughtful view of what needs to come next, that's a very different proposition from simply taking an observability or traditional governance product and relabeling it for AI.

Getting the philosophy wrong at the beginning also creates another problem: technical debt.

In a market moving this quickly, a flawed foundation can accumulate technical debt at extraordinary speed. Worse, governance technology built on the wrong assumptions can give an organization a false sense of security.

Howard compared that risk to the old concept of security through obscurity. It appeared to work until automation, scanners and widely available attack tooling demonstrated that obscurity wasn't really security at all.

AI governance risks creating the same illusion if organizations mistake controls designed for deterministic applications for controls capable of governing non-deterministic systems.

AI Works Best When It Accelerates What Humans Already Do Well

Despite the governance challenges, Howard saw several applications of AI at Black Hat that he believes demonstrate where the technology can provide real value.

One example involved DevSecOps.

Howard's criticism of traditional DevSecOps is that organizations ask developers and security practitioners to operate with fundamentally different mindsets. Developers are rewarded for moving quickly. Security teams are expected to be deliberate and methodical.

Putting those responsibilities into the same workflow effectively asks a developer to move quickly, suddenly slow down and think like a security professional, and then accelerate again.

Howard saw one company using AI to approach the problem differently. Rather than asking developers to consciously switch into a security mindset, the technology added security components as a side effect of the development work they were already performing.

The result is closer to what DevSecOps has always promised, but without expecting developers to continually change how they think and work.

It's an example of AI being used as an accelerator rather than simply being inserted into an existing process because the market currently expects everything to contain AI.

An AI Agent Isn't Just Another Non-Human Identity

Identity was another major theme Howard saw at Black Hat, particularly the tendency to classify AI agents as non-human identities, or NHIs.

He believes that's the wrong model.

Traditional non-human identities generally exist for deterministic applications and services. The risk wasn't necessarily that the application itself would suddenly decide to misuse its permissions. The problem was that service accounts were frequently overprivileged and insufficiently monitored, meaning an attacker who compromised one could inherit powerful access without immediately being noticed.

AI changes the equation because the identity itself can be associated with a non-deterministic system.

Howard argues that human behavior is often more predictable than we recognize. People tend to log into applications in similar ways, perform routine processes similarly and follow familiar workflows. Someone working in accounts payable may process invoices in almost exactly the same manner every day.

An AI agent can approach the same instruction differently from one execution to the next. Change the model and its behavior may change again.

Putting that identity into the same conceptual bucket as a traditional service account risks ignoring precisely what makes AI different.

Howard currently refers to this as a non-deterministic identity. The industry may ultimately settle on different terminology, but his larger point is that security teams need to recognize it as a distinct problem rather than forcing it into an existing category because that category is convenient.

Look for Vendors That Understand the Problem, Not Just the Market

With dozens of companies converging on similar AI security categories, choosing technology becomes increasingly difficult.

For Howard, one of the most important questions is whether the people building a product genuinely understand the problem they're trying to solve.

Experience matters, but understanding goes beyond simply having encountered the problem before. A vendor should understand why the problem exists, how the industry arrived there, what attackers or other opposing forces are doing to create the problem and what an effective solution should actually look like.

There is also a commercial reality that technology companies sometimes overlook: somebody with a budget needs to care.

A technically elegant solution that makes an engineer's life easier may be useful, but enterprise buyers operate with finite budgets. If a vendor wants an executive's signature, it needs to explain how solving the problem makes the business better.

That combination — technical understanding, real experience and an ability to connect the solution to a business outcome — becomes particularly valuable in a market where it's increasingly easy to build a prototype and considerably harder to prove that you've built something enterprises genuinely need.

Surviving Black Hat Means Knowing How Your Battery Works

The conversation eventually moved from surviving the AI market to simply surviving Black Hat itself.

For Howard, recovery is particularly important because he's an introvert. Years of experience have made him comfortable speaking publicly and spending time with customers, vendors and peers, but that doesn't mean those interactions don't take energy.

After Black Hat, he was fortunate to have some breathing room because he'd recently started his new company. Rather than immediately filling his calendar, he spent time building tools, processing what he'd learned at the conference and allowing himself to recover.

In previous years, he has deliberately taken vacation after major conferences because of the toll they can take. After days of meetings, events and conversations, his "people battery" can be completely depleted.

Jaye experiences conferences differently. As an extrovert, spending time around people can recharge rather than drain that battery. But the underlying lesson is similar: everyone needs to understand how these enormous industry events affect them and plan accordingly.

Black Hat isn't simply several days of conference sessions. For many attendees, it's Hacker Summer Camp, with multiple conferences, customer meetings, dinners, parties and opportunities to reconnect with people they may not see anywhere else during the year.

And Then There's the Food

No No Trust conversation about Las Vegas would be complete without discussing food.

Howard has some very specific recommendations.

One favorite is a caviar donut: a lemon cream-filled donut topped with salty caviar. It's the sort of combination that may not sound immediately obvious, which is precisely why Howard enjoys taking people to try it.

He also highlighted Las Vegas's Thai and Chinese food, particularly Chinatown, along with sushi and black cod. His broader philosophy on food actually mirrors some of his thinking about technology: hearing somebody else's description only takes you so far. At some point, you need to experience something yourself before you can really understand it.

And when the conversation moved from Las Vegas into fall cooking, Howard's preferences became considerably more comforting: homemade chili with beef, chorizo and, controversially for some listeners, beans, along with soups, stews, fried chicken, mashed potatoes and rich gravy.

After a conversation about non-deterministic AI, technical debt and the future of security governance, there are worse places to end up.

The Bigger Picture

AI is forcing cybersecurity and technology leaders to reconsider assumptions that have guided enterprise IT for decades.

The problem isn't simply that AI is faster or more powerful than the software that came before it. The more fundamental difference is that AI systems can be non-deterministic. They can approach the same objective differently, find paths their creators didn't anticipate and continue pursuing an objective beyond the boundaries a human assumed were obvious.

That makes governance more than a compliance exercise. Done correctly, it becomes part of making AI operationally useful, economically sustainable and safe enough to deploy at scale.

It also means organizations need to resist the temptation to solve entirely new problems with familiar labels. An AI agent isn't necessarily just another application. Its identity isn't necessarily just another NHI. And governing it isn't simply a matter of taking the policies used for traditional software and adding "AI" to the product description.

The organizations that navigate this transition most successfully may be the ones willing to acknowledge that some of the old models no longer fit.

Use AI where it can address work teams cannot get to today. Treat rapidly evolving technology decisions as temporary where appropriate. Ask vendors what their products are actually built upon. Understand the philosophy behind the technology rather than simply evaluating the dashboard.

Most importantly, recognize that intelligence and wisdom are not the same thing.

We've built systems capable of extraordinary things. The challenge now is making sure we understand how, where and under what conditions we want them to use that capability.