In this series we’ve covered AI-first transformation, from AI principles and strategy to agent ideation and pilots through rollout and scaled adoption.
We’ll conclude with a look at the crucial role of the AI manager in this new AI-first organization. To do so we’ll focus on the emerging org structure at Box as we continue on our own company’s (far from complete) AI transformation.
The AI org chart: top-down leadership meets bottom-up creativity
Organizationally speaking, an AI-first company flourishes through the creative tension between top-down structure and bottom-up creativity.
As we’ve discussed, a key element of Box’s AI-first philosophy is to give as many employees as possible the tools, training, and incentives they need to explore new ways that AI agents could help them and their teams.
But as recent studies suggest, simply investing company resources in AI doesn’t guarantee positive results from those investments.
“We’re seeing returns on our AI investments,” says Kelly Millsaps, Box’s VP for Digital Customer Success, “but it's been harder and taken more time than we expected. Many companies approach this as a hobby. If somebody is working a 40-hour work week, 38 hours of which they’re spending on something else, and they're supposed to create a magical AI outcome in two hours a week — from what we’ve seen, that's just not realistic.”
The Box system of AI management studies and selects potential AI agents as they emerge from each function’s creative pool, then carefully manages and optimizes the chosen agents as they move through production and into adoption into our long-term workflows.
At the top is the Operating Committee (OpCo) — the executive team which makes decisions and sets direction around Box’s overall AI strategy and priorities, best practices, governance, and usage guidelines.
OpCo sessions are also attended by Functional Leaders representing broad company areas like sales, marketing, engineering, customer support, etc, who use OpCo guidelines to set their department’s strategic direction for AI: which processes need AI agents, which processes need to be reinvented.
Robert Ferguson, Chief of Staff - CEO at Box, says “functional leaders do the strategic change management which will define how their teams will actually use AI in their ongoing work.”
Functional leaders, in turn, work closely with teams of AI managers who oversee individual AI agents for specific roles and skills (within Sales, for instance, there might be an AI manager for BDRs, AEs, prospect research, contracts, etc).
The evolving role of AI managers at Box spans four broad categories, from short-term tactical thinking to medium-term research and optimization and finally long-term strategic planning.
I: Tactics – Ensuring AI agents are designed and built correctly
Once a given AI agent graduates from its Ideation and Pilot phases, a Box AI manager monitors that agent’s development, training, and enablement, collecting feedback and offering guidance to ensure that it’s built properly and functions as intended.
But AI agents aren’t set-and-forget.
“Okay, we have a functioning agent,” says Ferguson. “‘Now how do we supervise it? How do we run redesign processes?”
Once an agent has moved through rollout and scaled adoption into the wild, the AI manager bears day-to-day responsibility for ensuring the agent is performing as expected, gathering feedback on how it’s being used, how it’s performing, and how the team can update and improve its functionality.
Part of this task is making sure the agent’s training stays up to date.
“When we began the informational solution journey,” Millsaps says of the agents whose knowledge bases supply answers and information to Box customer support teams, “we wildly misinterpreted the velocity with which our AI toolset would plateau in efficacy without material improvements in that knowledge base. Within 45 days, we went from fine-tuning the prompts to, ‘We have to update this knowledge base, and we have to do it now.’”
Only experience can provide that kind of learning, and only that kind of learning can give AI managers the perspective to move into the longer-term strategizing that will ensure their company’s ongoing successful AI transformation.
Ready to identify your AI transformation priorities?
II: Function – Applying AI agents to team and workflow updates
Okay, so you’re an AI manager. You’ve got your teams working with AI agents, you’ve seen what those agents can and can’t do, where their work can and can’t be trusted, how their output might be improved.
Now let’s widen the aperture to focus on the bigger picture.
“After your agent is rolled out and scaled and you've got adoption,” Ferguson says, “you need to start evaluating the longer term implications in order to really make the most of them.” For AI managers, this means answering a series of difficult questions about how AI agents and their human teammates function:
- How should the makeup of your team itself change now that it includes AI agents that are performing what used to be human tasks? Does it need to be bigger, smaller, or just different?
- Should some roles be redefined as human activities are assumed by AI?
- Do you need to upscale or reskill team members, not just in how to use agents, but how to supervise agents, identify when they’re working well and when you need humans in the loop?
- Do you need to change not just team members’ skills, but the makeup of teams themselves? Do you need more or less people in specific roles?
- Are entire human roles lost, and new roles created?
- The answers to these questions will be informed, at least in part, by yet another aspect of the AI manager’s change management role: working out the most effective new ways to calculate the organization’s AI investments’ ROI.
III: ROI – In the end, which AI agents are actually worth it?
“There's a lot to unpack around the cost of using an agent versus a human,” notes Ferguson. Some aspects of this calculus — the baseline cost of running a given agent, the stakes of whether this is a customer-facing process, the basic quality of the agent’s output — have already been considered in the Pilot phase.
But these judgments, like so much about AI adoption, should be considered perpetual recalculations that remain an AI manager’s ongoing responsibility.
ROI 1: does your agent deliver high-quality results?
Last fall the Box Content Team built and began using AI agents for several core aspects of producing written content: an outline agent, a drafting agent, and an editing agent.
Over the first few years of the GenAI era, the most popular LLMs acquired a great reputation for writing extremely fast first drafts. But when our Content Team put in-house agents to work writing for our specific voice, recalls Head of Content Marketing Nick Johnson, “we found that the outline agent and the editing agent are very powerful, but the drafting agent just produced a bunch of AI slop.” So, the team decided to adopt the outline and editing agents. “But the reality for our team is that we need to do the actual drafting ourselves.”
ROI 2: How much measurable savings does your AI agent produce?
A slightly more complex ROI question involves measuring a given agent’s cost savings. Millsaps’ customer support function uses Box’s broadest range of AI agents and thus might be the most advanced at this analysis. “In a broad sense,” Millsaps explains, “we're looking at each of the AI projects that we run and trying to baseline something” against which the agent’s genuine ROI can be measured.
Box’s customer support chatbot offers one clear, simple example.
“We historically have had six people running human-driven chat,” Millsaps notes. “So by looking at the number of support tickets we’re able to deflect from any human interaction and the number of humans it would have taken to respond to all those tickets, we can estimate the value that this AI solution has had.”
Other cases, Millsaps says, aren’t that straightforward, “because you have to measure something that didn’t happen.”
To arrive at a useful ROI metric for the AI search function at support.box.com, for instance, Millsaps’ team goes through this exercise:
- Study each user query the AI search agent successfully handles
- Estimate how often the lack of that capability would have led to a call to a live human
- Calculate the resulting theoretical cost savings.
The bottom line for this “reverse approach” to ROI is less precise, but that doesn’t make it any less important for Millsaps team — or yours — to measure.
ROI 3: When does it make sense to use an AI agent in the first place?
In the sales and broader go-to-market world, for instance, functional leaders and AI managers are learning to study in advance which smaller accounts and lower-buying-propensity prospects you might be able to service digitally, “with just a human in the loop to do some checking,” Ferguson notes. “That’s how you scale with technology and hit a huge number of prospects.”
But with huge prospects and important existing customers, you flip that model on its head and let humans lead those crucial interactions, with AI playing an augmentation and support role. “That's where a lot of the ROI calculation will come in,” Ferguson says; “you want to be thoughtful about where to place your agents based on the stakes — as long as you’re never compromising any customer experience.”
Track ROI, time saved, and revenue alignment in one place
IV: OKRs – How should AI impact human performance reviews?
And what about us humans? How, if at all, should we change how we evaluate human performance in workflows to which AI agents have begun to contribute?
“I think if you're a better performer because you’re using an AI agent, you should get rewarded for that,” says Ferguson. “We should be rewarding and encouraging it.” He offers two principles for thinking about performance reviews in this dawning era of human/AI collaboration:
- Employee performance shouldn’t be discounted because AI made it better
- We should seek out new ways to reward and encourage AI-first behavior.
“We're still at an early stage — Box and our customers — of realizing AI’s full potential,” Ferguson concludes. “We need to be rewarding adoption, which includes figuring out and implementing new and better ways of taking advantage of our AI tools. We need to think about how we put that into folks’ OKRs.”
Over at Box customer support, Millsaps’ team is already doing so.
“We’ve made changes to our roles and responsibilities for FY 27,” he says; “not specifically measuring AI per se, but shifting from traditionally customer-facing metrics like, ‘How many tickets did I close in a period of time?’ and “What was my CSAT for those tickets?’ to components like ‘How many knowledge base articles did I create, how frequently were those articles consumed, and how frequently was consumption of those articles a resolution to a problem?’ In a lot of ways, we're focusing now on key dependencies to make AI effective.”
Onward: Beyond the end of the beginning
As we begin 2026, that’s the state of the art at Box for AI transformation.
We’re further along this journey than many companies. But we’re very far from where we know we need to be. We think the ideas and practices we’ve outlined in this series will be helpful guides to setting up your own planning and processes, but we’re also pretty sure we’ll wind up revising most of these processes in the months and years to come, and that you’ll find your own ways to design an effective AI-first enterprise.
When you embark on your own AI journey, make those discoveries, and reach your own conclusions, we hope you’ll be sure to let us know.
