Testing Gemma 3 on the Box AI Enterprise Eval

Choosing the right AI model is a critical decision for organizations, requiring a careful balance of performance, cost, and infrastructure control. Box recently evaluated Google’s new open-source model, Gemma 3, focusing on its ability to extract metadata from unstructured data within documents—a crucial task for many businesses. Our findings reveal a compelling narrative: open-source models rival proprietary ones in performance, while offering many advantages.

Gauging Gemma 3's capabilities

Our evaluation centered around two key metrics for metadata extraction: accuracy (how correct the extracted information is) and extraction volume (how much information is successfully extracted). We compared Gemma 3 to Google’s Gemini 1.5 Flash, Gemini 2.0 Flash, and its predecessor, Gemma 2.

When it comes to accuracy, the playing field is surprisingly level. Gemma 3 achieved accuracy on par with Gemini 1.5 Flash and closely followed Gemini 2.0 Flash. These numbers reflect how many fields the models identified and had correct values for. Notably, Gemma 3 showed significant improvement over Gemma 2, with substantial gains in accuracy, often in the double-digit percentage point range, making it a much more reliable model for extracting essential information, especially in areas critical for contract analysis.

In terms of extraction volume, Gemma 3 surpassed Gemini 1.5 Flash, demonstrating a strong capability, though Gemini 2.0 Flash remained the leader among the Google models tested.

These results demonstrate that Gemma 3 isn’t just "good for an open-source model"; it’s a high-performing model overall. This level of performance, combined with the cost advantages of open-source, makes Gemma 3 a strong contender. Our analysis indicates that Gemma 3’s operational cost is roughly half that of Gemini 1.5 Flash, presenting a compelling cost profile.

Reasons to consider an open-source model like Gemma 3

Beyond performance and cost, open-source models like Gemma 3 offer multiple other strategic advantages that are worth considering:

• Customization and flexibility: Think of Gemma 3 as a powerful engine with available blueprints. It allows for extensive fine-tuning for specific tasks, such as training on legal documents for a law firm, significantly improving accuracy for contract analysis. You can also adapt it to your unique data, integrating proprietary datasets.
• Transparency: Open-source code allows for auditing, bias examination, and direct debugging, fostering trust, ensuring responsible AI use, and addressing ethical considerations. This openness is crucial for building confidence in the model’s outputs.
• Control and reduced vendor lock-in: Choosing open-source options like Gemma 3 provides flexibility and control, allowing customization and avoiding dependence on a single vendor. Self-hosting Gemma 3 offers data privacy, infrastructure optimization, and potential long-term cost savings, although it requires technical expertise and ongoing maintenance.

Embrace model flexibility for enterprise innovation

The rise of strong, affordable, open-source AI models like Gemma 3 is a turning point for businesses using AI. Open-source models offer a good option for companies that want more control, customization, transparency, and lower costs. This is especially true for businesses that want to host AI themselves.

Box's analysis of Gemma 3 highlights a key strategy for businesses using AI: While new, proprietary models provide the latest features, open-source alternatives often catch up quickly in performance, offering significant cost savings. Businesses can leverage this by initially adopting the newest models for innovation, then transitioning to comparable open-source options a few months later to optimize costs--In this comparison, 6 months elapsed between the Gemini 1.5 Flash release and Gemma 3 release. By doing so, companies can stay at the forefront of AI advancements while maintaining a practical approach to expenses.