Evaluating Meta's Llama 4 Models for Enterprise Content with Box AI

Meta recently introduced Llama 4 Scout and Maverick, its first models featuring a Mixture of Experts (MoE) architecture. At Box, we wanted to see how this MoE strategy, alongside the models' open-weight nature, performs against real-world enterprise content demands.

Llama’s performance on enterprise tasks

We tested Llama 4 Scout and Maverick on tasks relevant to enterprise workflows, using information extraction from complex contracts via our Box AI Enterprise Eval process to gauge performance. Here’s a breakdown of our findings:

• Information complexity: Maverick and Scout showed similar near-perfect accuracy (~99%) in extracting straightforward information fields (e.g., identifying named parties or specific dates in documents). However, when analyzing document sections with more complex logic, nuance, or conditional statements (e.g., clauses defining specific rights, restrictions, or obligations), Maverick, with 128 MoE experts, significantly outperformed Scout (16 experts), achieving ~85-92% accuracy compared to Scout's ~45-70%. This indicates Maverick's superior ability to grasp complex requirements and edge cases common in enterprise documents.
• Reasoning for complex requirements: While Scout performs well generally, Maverick's advantage in deep reasoning and handling intricate requirements seems rooted in its architecture. Its larger total parameter count (400B vs 109B) suggests a richer knowledge base, and its far greater number of experts (128 vs 16) allows for finer-grained specialization relevant to diverse business concepts. This combination enables Maverick to process complex information more effectively, making it better suited for sophisticated enterprise tasks demanding high accuracy and nuanced understanding.
• Growth from Llama 3: In looking at Llama 4 vs its predecessor Llama 3 Nemotron, Llama 4 Maverick shows a 33% gain in accuracy over Llama 3 Nemotron. Llama 4 Maverick consistently achieves higher accuracy than Llama 3 across nearly all fields tested, however, Llama 3 does still outperform Llama 4 Scout at some specific extraction fields, such as Audit Rights and Effective Dates.

Our analysis shows that Llama 4 Scout achieves performance comparable to leading models in its class, such as Claude Haiku, Gemini Flash, and GPT-4 Turbo, specifically for tasks like multi-document processing and general document Q&A. This establishes Scout as a capable foundation for broad enterprise information retrieval needs.

The role of open-weight models in the enterprise

The Llama 4 family continues the trend of capable open-weight models. For enterprises, this ecosystem offers several potential advantages:

• Cost Efficiency: Open-weight models, particularly those using efficient architectures like MoE, can offer attractive performance relative to cost.
• Control & Customization: Access to model weights allows fine-tuning for specific business needs and deployment on preferred infrastructure, reducing vendor lock-in.
• Transparency: Open-weight models can provide greater visibility into model architecture and operation.

Llama 4 and Box AI

To try Llama 4 in Box AI Studio and Box AI APIs send us an email at [email protected] to request to be part of our early access program.