During my nightly FaceTime with my parents — somewhere between them playing with their grandkids and the usual recap of our days — they asked me whether AI was really going to kill us all in the next ten years.
Given the past week’s headlines, I wasn’t surprised by the question. Researchers and former employees of frontier labs have publicly raised concerns that the race toward more capable systems is moving faster than our ability to understand and control them. Former Anthropic researcher Jacob Coxon warned that the industry is racing toward self-improving AI without knowing how to control it. That concern was amplified when Anthropic alignment lead Evan Hubinger said he believed there was a greater than 10% chance AI could kill all humans within the next decade. At the same time, Anthropic released a report detailing attempts to misuse current AI systems for cyber operations, surveillance, propaganda, and potentially dangerous biological research.
Those warnings deserve to be taken seriously; when the people closest to a technology are worried about its direction, we should listen. Personally, I don’t think human extinction from AI in the next ten years is a likely outcome. Current models don’t have motives of their own, a survival instinct, or a desire to harm people. They aren’t plotting to kill us all, or anyone for that matter.
But AI in the wrong hands, or AI given too much access and authority without the right guardrails, can create serious harm without any of that. It only needs an objective, access to consequential systems, and the ability to act faster and at a greater scale than the people supervising it.
That isn’t science fiction. It’s the practical security challenge we’re already facing.
The real risk is authority, not motive
I don’t think we should dismiss long-term AI safety concerns. As systems become more capable and autonomous, we should be investing in the research, testing, and governance needed to understand where the boundaries are. But I also don’t think we need to settle big questions about what a model may someday “want” before deciding how to use it responsibly today.
Current AI systems can appear purposeful because they can pursue a goal across multiple steps. But that’s not the same thing as human judgment, values, or intent. And in my view, AI plotting to kill all humans is not the risk that should concern us most right now. The more immediate risk is that people give AI systems too much authority before they’ve earned that trust.
An agent that’s asked to research a topic, draft an email, or summarize documents is one thing. An agent that has the ability to access and manipulate sensitive data, source code, financial systems, production environments, or the ability to act on someone’s behalf is another.
Here’s a simple formula I use to think about AI risk:
Risk = capability × access × autonomy × scale
A highly capable model operating with tightly scoped tools and access can have a smaller blast radius than a less capable agent with broad credentials, vague instructions, and permission to take action.
That’s true whether the harm comes from a criminal using AI to run a more convincing scam, an employee trusting an answer they should have verified, or a company deploying an agent that can make meaningful changes without enough oversight.
What we should be worried about
The risks I’m most focused on are the familiar problems of trust, fraud, access, and accountability, which are now happening at machine speed.
- AI will make deception cheaper and more convincing. We’re already seeing more realistic phishing, fake voices, fabricated images, and messages that look and sound increasingly credible. The answer can’t be expecting everyone to spot every fake. The answer is rebuilding the habit of verifying unexpected requests, especially when money, credentials, sensitive information, or urgency are involved.
- AI will also make bad human intent more scalable. The concern is less that a model independently decides to attack someone and more that a person with harmful intent can use a capable system to move faster, operate at greater scale, and lower the cost of doing damage.
- Humans will outsource judgment too quickly. A polished answer can feel more reliable than it actually is. That matters when someone uses AI to make a hiring decision, assess a security issue, interpret legal guidance, or decide what information is true. AI can be an incredibly useful assistant. But it shouldn’t become an unaccountable decision-maker.
As agents become more capable, the question shifts from “What can this model tell me?” to “What can this system do?” That’s the question we should all be asking before we hand an AI system access to the things that matter.
What shouldn’t distract us
We should take concerns about catastrophic AI risk seriously, but let’s not let the most cinematic version of the problem distract us from the harder, more immediate work in front of us.
The idea that AI will wake up and decide to harm humanity gives the story a clear villain. The actual risk is less dramatic: people and institutions deploying powerful systems with consequential incentives, weak controls, poor visibility, and no one who’s clearly accountable when things go wrong. Unfortunately, “unclear ownership and excessive permissions” isn’t nearly as marketable as a rogue superintelligence. It is, however, where a great deal of real-world harm starts.
We shouldn’t assume that every new model release means we‘ve lost control. We shouldn’t confuse human-like language with human-like judgment. And we shouldn’t accept the false choice between panic and complacency.
There’s a more useful middle ground: use the technology, understand where it creates risk, and insist that the people building and deploying it remain accountable for the outcomes.
What I hope the average person takes from this
You don’t need to be a technology expert to navigate this moment. Here are a few basic rules we can all keep in mind as we learn to manage AI in our work and personal lives:
- Verify unexpected messages, calls, or requests through a separate channel.
- Be thoughtful about what personal or work information you share with AI tools, especially tools that are connected to your files, email, or other accounts.
- Use AI to help you think, learn, and create, but prioritize your own judgment when it comes to consequential decisions.
- Ask simple questions: What information can this tool access? What can it do on my behalf? Can I see or undo its actions?
The best defense in the AI era isn’t fear; it’s healthy skepticism.
What security leaders should take from this
For leaders, the responsibility is heavier.
The central question is no longer simply, “Is this model safe?” It is: What can this system see? What can it do? Who is responsible for it? And how do we stop it when it gets something wrong?
Model guardrails are useful, but they are not, on their own, a security boundary. The systems around the model matter just as much: clear identities, limited permissions, constrained access to sensitive data, human approval for high-impact actions, reliable audit trails, and the ability to contain or reverse an action when needed.
This is why I found Dario Amodei’s recent call to “pace the frontier” worth paying attention to. The point isn’t to slow progress because a model might suddenly decide to harm humanity. It’s to make sure our ability to evaluate, monitor, and control increasingly capable systems doesn’t fall behind the capability and authority we give them.
We shouldn’t build critical assumptions around whether a model chooses to behave. We should build systems that make unsafe actions difficult, visible, and reversible.
The goal isn’t to slow down innovation, but allow innovation to be trusted at scale.
AI will create real risks, and some may be profound. But fear is not a strategy, and neither is pretending the risk only begins if a machine develops human motivations. The work in front of us is more practical: keep humans accountable, match autonomy to controls, and make sure speed never outruns judgment.
AI doesn’t need to go rogue to cause harm. But harm is not inevitable if we remain deliberate about who gives it authority and under what conditions.
