📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large-scale language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, a result that questions the effectiveness of current sovereign-LLM scaling strategies.

The Minerva project, led by Sapienza University of Rome and funded through Italy’s national AI strategy, built the model using 128 GPUs on the CINECA Leonardo supercomputer. Despite the extensive investment and open data approach, Minerva-3B’s performance on the Italian academic test was near chance, contrasting sharply with its impressive technical architecture.

Minerva’s training involved 2.5 trillion tokens, with about half being Italian, making it one of the largest native-language models publicly available. However, the evaluation by researchers revealed that the model’s ability to handle complex language tasks, such as academic exams, remains limited, suggesting that scale alone may not suffice without further methodological innovations.

This empirical result complicates the narrative that larger, native-language models inherently produce deeper country-specific knowledge. It underscores the importance of not just data quantity but also the quality and training strategies, especially for complex language tasks.

Implications for European Sovereign-Language Model Strategies

The underperformance of Minerva-3B on academic benchmarks highlights a critical challenge for European countries investing in sovereign AI. It suggests that substantial scale and resources may be necessary to develop models capable of understanding and performing complex tasks in national languages, which has broad implications for policy and funding decisions across Europe.

This finding questions the assumption that training models with a large share of native language data will automatically lead to high-quality, country-specific AI applications. It emphasizes the need for strategic investment in both data scale and training methodology to achieve meaningful AI capabilities at the national level.

European Sovereign LLM Development and the Scale Debate

Italy’s Minerva project represents a deliberate departure from approaches like Portugal’s AMÁLIA, which layered language-specific data onto a multilingual foundation. Minerva trained from scratch on a massive dataset, aiming to produce a highly capable Italian language model. Despite this, the recent evaluation reveals that even large-scale native-language models may not meet expectations for complex academic tasks.

Earlier European efforts have debated whether continuation training or training from scratch is more effective, with the broader consensus increasingly recognizing the importance of scale. Minerva’s results add a new dimension to this debate, highlighting that scale alone may not guarantee high performance, especially in specialized tasks.

“The overall size of the dataset and the number of parameters are more crucial for handling complex language tasks than the pre-training dataset composition alone.”

— Research team evaluating Minerva

Unresolved Questions on Model Scaling and Performance

It remains unclear whether further scaling, different training strategies, or enhanced data quality could improve Minerva’s performance on complex academic tasks. The ongoing research aims to determine if the current limitations are fundamental or surmountable with additional investment and methodological refinement.

Next Steps in European Sovereign LLM Development

The Minerva team continues to iterate on training methodologies, with upcoming experiments focusing on larger models and refined data curation. Policymakers and researchers will closely monitor whether increased scale or new approaches can bridge the performance gap for complex language tasks in national languages.

Further evaluations are expected as models grow in size and sophistication, providing clearer guidance on the investment levels required for truly effective country-specific AI systems.

Key Questions

Why did Minerva-3B perform so poorly on the Italian academic benchmark?

Despite extensive training on a large dataset, the model’s limited performance suggests that scale alone may not be sufficient to handle complex academic language tasks. Methodological factors, data quality, and training strategies also play critical roles.

Does this mean European sovereign LLM projects are not worth pursuing?

Not necessarily. The results highlight challenges but also provide valuable insights into what is needed for effective development. Larger scale and improved methodologies may still achieve desired outcomes.

What implications does this have for other countries developing their own models?

It suggests that countries should carefully consider their investment scale and training approaches, as simply increasing data volume may not guarantee success without strategic methodological enhancements.

Will further scaling improve Minerva’s performance?

Future experiments aim to test this hypothesis. Whether larger models trained with refined strategies can better handle complex tasks remains to be seen.

How does this affect the European AI strategy overall?

It underscores the importance of balancing scale with methodological innovation and resource allocation, informing future policy and research directions across Europe.

Source: ThorstenMeyerAI.com