ELOQUENCE project coordinator dr Jordi Luque participated in the first European LLM Multilingual Evaluation Expert Workshop, organised by the European Commission and the European Language Data Space (LDS), as an expert in multilingualism.
The Workshop was held online on July 2nd 2024, and structured in four parallel breakout sessions with participants split into discussion groups. After discussions, participants could exchange their group findings with presentations and conclusions.
Participants agreed on the following conclusions:
- The design of an LLM multilingual evaluation framework needs to follow a holistic approach.
- Recent work done by the LDS group shows an alarming lack of evaluation benchmarks, particularly datasets, for LLMs in multilingual scenarios.
- A general evaluation framework for LLM multilingual evaluation should encompass both an automated part and a human evaluation part.
- There is a need for significant investment in the adaptation of existing evaluation datasets and the creation of new ones.
- The role of verification datasets in a multilingual setting needs to be further explored.
- Data contamination has to be addressed.
- Metrics are necessary even if they are hard to understand by the public.
- Open need to standardize the description of the evaluation dimensions for LLM multilingual evaluation.
More about the organizer: The European Language Data Space aims at building a trustworthy and effective data market for the exchange of language resources in the public and – even more importantly – in the private sector, in line with the EU Data Strategy. The LDS user group consists of European stakeholders from mainly industry, representing all market segments, but also public administration and academia. LDS user group members shall actively contribute to and take advantage of the LDS, bringing in their own requirements and validating the emerging LDS infrastructure.
Useful links:
