Staying ahead of AI-powered bad actors

How to understand and prepare for the threat of Adversarial Artificial Intelligence

AI is a major force for the future of business, offering incredible increases in productivity and effectiveness for organizations across every industry. It can even be leveraged to improve an organization’s security posture, anticipating threats and adapting protections. But unfortunately, these AI superpowers aren’t necessarily limited to the good guys.

What if cyber-attackers could harness these as well, planning and executing not just as a single actor, but as a team of hyper-intelligent malicious experts that are determined to take over the world (or at the very least a few compute clouds, billions of endpoints – and your most valued content). They could manipulate all the things that we interface with and rely on, directly impacting safety through control of physical interfaces, and potentially subverting the human experience.

This virtual team of miscreants would be led by a charismatic, ruthless and stealthily-distributed AI known as The Strategist. The Strategist would be the brains of the operation, devising plans to undermine foundational elements of confidence spanning from clouds to humanity (while making quite a bit of money in the process, as well). The other members of the team would be experts in diverse fields that include knowledge-based social engineering to leverage human risk, escalating propaganda to spread misinformation and erode trust, and performing persistent active exploitation to transfer value at their discretion.

This AI-empowered supervillain team could cause major damage – developing autonomous cyberweapons, destroying reputations, careers, and generally eroding defenses through:

• Cyberbullying and harassment – fatiguing and exhausting key resources
• Reputational attacks – think highly-targeted deepfakes
• Impersonation – convincingly leveraging video, audio and live interaction
• Content subversion – generative AI maliciously altering content and biasing interactions
• Eroding privacy – correlating information, de-anonymization and viciously profiling
• Impacting safety – controlling physical interfaces from self-driving cars to medical devices
• Anticipating your next move as a cyber-defender

This scenario would make for a thrilling edge-of-your-seat movie plot – but would also make for a lousy day at work for us cyber-defenders.  Let’s keep that last bit from happening, OK?

In the second half of this blog, we focus in on the aspects of Adversarial AI that impact the confidentially, integrity, availability and safety of enterprise content.

Preparing Your Content for Adversarial AI

While we can only guess at the velocity and extent of Adversarial AI, we see looming realities in the above storyline.  We also know that to counter Adversarial AI, we must beat The Strategist at his own game – by leveraging guidance from our own thoughtfully-devised content protection strategy.


To prepare our most valued content for AI-infused interactions, here are a few key tactics to employ:

• Know your content better than an attacker would
• Control full lifecycle usage of content (not just access)
• Automate desired behaviors and content protections
• Stay vigilant

Know Your Content Better than an Attacker Would

• Perform a content inventory, particularly scanning for and discovering PII and other sensitive content.  Now might also be a good time to consider whether all that distributed content really needs to be distributed.
• Assign or reassign ownership for content to an individual, role or department.  Ownership should be granted to whoever is directly responsible for content classification, retention and disposition.
• Assign classification to content in accordance with organizational policies (e.g. Internal-Only, Public, Confidential, High-Restricted).  Classification can be manual, fully-automated or recommended and is a must-have attribute for understanding and managing content sensitivity and appropriate usage.
• Complement and extend the classification labels with rich content insights expressed through metadata.  Metadata is an attached set of tags and attributes that provide further specificity to the nature of the content.  Content type (e.g. contract, NDA, resume), key dates, project codenames, intended usage (e.g. Human Resources and Legal only), can all be expressed through metadata.  You define what’s important to you for searching, organizing and controlling usage of content.
• Content has meaning, value and purpose – define these in business terms and include in metadata. Tokenize these values if they’re too sensitive to expose directly.

Control Full Lifecycle Usage of Content

• Review who and what should be able to utilize your valued content – to what extent and under what circumstances.  Define across users, APIs, service accounts and automation frameworks.
• Assign specific and granular content permissions (Co-owner, Editor, Viewer Uploader, Previewer Uploader, Viewer, Previewer, Uploader) that apply to users, groups and roles. Revoke outdated permissions that are overly-permissive.
• Require Multi-factor Authentication (MFA) to ensure user identity.
• Compartmentalize appropriate usage and define information barriers (aka ethical walls) for highly-restricted content used by security teams, human resources, legal, M&A teams and others throughout the enterprise.
• Review security policies and configurations, updating regularly to address evolving AI directives and defenses.

Automate Desired Behaviors and Content Protections

• Protect content based on its meaning, value and purpose – as defined in classifications and metadata.  Assign policies that automatically control desired outcomes, such as preventing Internal-Only content from being externally distributed or collaborated on.
• Institute and manage autonomous controls for protecting content based on Device Trust, geolocation, external collaborator status, disposition and retention, usage thresholds and workforce behavioral heuristics.
• Provide automated reporting to IT and security teams, along with rich insights to line-of-business and content owners to prove that content security objective are being met – and to investigate anomalies.
• Investigate using the Box Content Cloud as the platform supporting your SaaS and Cloud-based applications.  With 1500+ integrations, the capabilities and controls described here could now natively empower content protection for your favorite apps!

Stay Vigilant!

• Monitor, report and alert on content change velocity, engaging your content owners.
• Integrate with threat-informed defenses from endpoints to APIs and all agents that access, create and manage your content.
• Extend observability by interfacing with CASB (Cloud Access Security Broker), SIEM (Security Incident and Event Management) and SOAR (Security Orchestration, Automation and Response).
• Include elements of your content protection strategy in your organization’s security awareness, training and human risk management initiatives.

That’s a long list of considerations – but it’s only the beginning of what’s needed to protect content against evolving threats. By focusing on the appropriate lifecycle usage of content, we can collectively recognize and thwart attacks against our most critical content, and stay one step ahead in the ongoing struggle with bad actors. Contact your content protection resources at Box to help you get started or to accelerate your security journey.