Enterprise-grade AI security: What it takes to trust AI with your data

January 29th 2025 | Ben Kus, Chief Technology Officer at Box

How to evaluate which AI models and vendors offer enterprise-grade capabilities

With all the astounding (and evolving) benefits of generative AI, it's no wonder companies are in a mad dash toward integration.

However, a powerful countervailing force has hindered AI adoption, as enterprises evaluate its data security, privacy, and compliance complexities. Companies want to know whether they can use AI models with confidence and that their data will stay secure. The implications of the decision to trust AI are profound and urgent, determining whether an organization can leverage generative AI innovations (including the latest models from OpenAI and Google).

This article highlights how enterprises can trust AI models and outline essential criteria for ensuring that the models they select offer enterprise-grade capabilities.

What’s the worst that can happen? The AI breach scenario:

Imagine you’re working with a highly confidential set of data, such as files related to an acquisition. One of the evaluators of the deal is tasked with due diligence on sensitive financial details, employee salaries, and other critical information. In this context, AI could be extremely beneficial, providing insights, identifying strategies, assessing risks, and summarizing complex documents. But remember: this means your AI model naturally has access to confidential data.

Now, say your evaluator is so impressed with the AI results that they click the “thumbs up” icon to indicate their approval of the generated content. The AI vendor then logs its work as “helpful,” so the response is saved into a set of data to be used for training future models — along with all of the confidential data the AI model used to generate the response.

Next thing you know, your sensitive data is getting trained into future models accessible to every user. In the case of the large, publicly used models, that could mean everyone on the internet. From a data security perspective, it’s really no different than your garden-variety data leak.

This scenario underscores the critical importance of ensuring that AI models and vendors adhere to strict data security and privacy standards, preventing potential breaches and maintaining the integrity and confidentiality of enterprise data. Case in point: You need to be concerned with controls around AI data usage.

The role of SaaS vendors and hosting platforms in AI data security

You need to be able to trust every system and organization that touches your data, because every layer of your tech stack has the potential for data security issues. AI is no exception, and AI trustworthiness involves more than just evaluating an AI model itself. It requires a thorough assessment of the vendors that control access to your data.

Let’s consider some of the major players in the AI ecosystem:

Employees (with data permissions)
SaaS companies (like Box, Salesforce, etc.)
AI model (OpenAI’s GPT series, Anthropic’s Claude, Gemini, etc.)
Hosting layers (like Azure, GCP, IBM, AWS)

At first, evaluating security and compliance across multiple layers might sound daunting. The good news is that the vast majority of enterprises already trust some of their most sensitive data and operations to these companies and have the appropriate data protection frameworks, legal agreements, compliance certifications, and technical frameworks to keep their data secure.

Also, most leading AI models are hosted and used by the same vendors — AWS hosts Anthropic models, GCP hosts Gemini models, Microsoft Azure hosts OpenAI models. Companies like Box, Salesforce, Zoom, Slack, and others use these models to apply AI to customer data in a secure way.

Thus, from a data security perspective, using existing trusted enterprise-grade vendors for AI will significantly minimize your risk, since they already meet your data security trust standards. None of this means that enterprises can’t use a new vendor for AI security — including startups, AI models not hosted on a hyperscaler, or new SaaS companies with AI offerings. However, any new company must first align with existing standards of security, compliance, and privacy, in addition to assessing the operational maturity and track record to be able to protect your most sensitive data.

With AI, trust in data security is a prerequisite for deployment, but by itself it’s not sufficient. AI brings new challenges and concerns that must be specifically addressed before you can safely use AI models on your sensitive data.

4 questions to ask when determining if you can trust an AI model

You can place your confidence in an AI model if you answer “yes” across the board.

1. Do you trust the vendor (and all their hosting layers) with your data?

As illustrated above, the prerequisite for trusting AI is knowing an AI company can handle your most sensitive data. Very few enterprises would trust their data to companies without established enterprise agreements such as consumer-grade free products that lack enterprise-friendly data-protection terms of service. Trusting a vendor with confidential data involves a multitude of considerations, including compliance certifications, contractual terms, reputation, company size, and the use of sub-processors. A straightforward approach to addressing this question is to apply the same data protection standards you currently use. If you already trust a vendor with your data, you might consider using that same vendor to manage AI operations on your data. Ensuring that your AI vendor meets these established standards helps to maintain the security and integrity of your enterprise data.

2. Does the vendor provide guarantees on data usage for AI training?

After establishing that you can trust a vendor with your data, the next question is AI-specific and involves the controls over data usage for AI training. As the hypothetical scenario above illustrates, one of the most alarming risks with AI is the potential misuse of confidential data in training.

Before deploying any AI solution, enterprises need explicit guarantees regarding AI training on their data. This could be as straightforward as a policy of “no training on enterprise data.” However, given the potential benefits of enterprise-trained models, a more nuanced approach might be “no training on enterprise data without explicit authorization.” It is also crucial to understand who owns and controls the model after it is trained, ensuring that proprietary information remains secure.

Logging is another critical consideration. AI logs of input and output often contain sensitive information. In all cases, access to the data should be logged (i.e., details around how and why the AI model accessed the data). However, to mitigate risks of inadvertently exposing sensitive data (e.g., the salary information in the M&A example above), logging should either: 1) include the option to disable data from being stored or, 2) adhere to stringent data security standards if logs are maintained. Many vendors offer the option to disable logging for generative AI to alleviate concerns about data being inadvertently used in future training runs. Ensuring that these controls are in place, is vital for maintaining the confidentiality and integrity of enterprise data.

3. Is the vendor transparent about their AI models and operations?

Generative AI is rapidly evolving, with new models being released constantly. Alongside this technological advancement, various legal cases concerning different AI models are progressing through courts in multiple countries, and new laws and regulatory concerns are being considered. Different AI models come with varying levels of protection, reputations for safety, and details about their training sets.

Significant information is often published about each new model, including model details, architecture, training data and methodology, potential biases and limitations, and approved use cases. This transparency allows enterprises to make informed decisions about the AI models they adopt, ensuring they understand the potential risks and benefits. Clear communication about model updates, changes, and ongoing evaluations is also essential for maintaining trust. By having access to comprehensive information about the AI models and their operations, enterprises can better assess the suitability and security of these technologies for their specific needs.

4. Does the vendor provide the right level of control over your data and AI models?

Many enterprises require centralized control of AI implementations to ensure security and proper governance. One key aspect of this control is the ability to turn the AI system on and off, potentially allowing a phased rollout that starts with specific groups within the company. This flexibility helps control the rollout of AI technologies into existing workflows without overwhelming the organization.

Control over AI logging configurations is another critical factor. Enterprises need to ensure that AI logs, which may contain sensitive information, are managed securely. This includes deciding whether to enable or disable logging based on security needs and ensuring that any logs maintained meet stringent data protection standards.

Additionally, having the capability to run administrative reports on AI usage and data governance is essential. These reports provide insights into how AI is being used across the organization, helping to enforce data governance policies and to identify any potential issues early. Robust control mechanisms ensure that enterprises can maintain oversight and compliance, adapting quickly to evolving AI landscapes and regulatory requirements.

Box AI: Intelligence you can trust

You’re rightfully cautious to carefully consider the AI models you use on enterprise data. And we get it: Data security and safe AI use are critical components to all our offerings.

With Box AI, you get peace of mind knowing we’re committed to helping protect your sensitive data, as you empower teams with the latest AI solutions (like secure RAG). We stay transparent about our AI practices, technology, vendors, and data usage.

To reinforce this commitment, Box has published and adheres to a set of AI principles. These principles guide our development and deployment of AI technologies, ensuring that they align with our values and the stringent security requirements of our customers. With Box AI, enterprises can confidently use advanced AI tools, knowing that their data is handled with the utmost care and integrity. Visit the Box AI Trust page to learn more.