Help choose the best Estonian-speaking AI

Estonian researchers have created the AI Barometer where users can compare the responses of language models and assess their proficiency in Estonian. Everyone is welcome to try out the barometer at baromeeter.ai. The researchers aim to collect at least 50,000 comparisons by the end of June.

Current AI text generators work best in English. Assessing language models’ Estonian language skills and knowledge of Estonian culture is difficult. One way to do this is to let users compare the responses of two models. In the AI Barometer, users can compare versions of GPT, Gemini, Claude, LLAmA, and Mistral, among others. Everyone interested in the development of AI and the state of the Estonian language is welcome to use the barometer.

According to Kairit Sirts, Associate Professor in Natural Language Processing at the University of Tartu Institute of Computer Science, the barometer helps raise awareness of the varying quality of language models and encourages people to use AI text generators in Estonian. “On the website, you can enter a prompt to which two anonymous language models respond. The user can choose the better answer, after which the names of the models are revealed. Based on the ratings, a continuously updated leaderboard is created,” said Sirts. She added that the selection of language models in the comparison is constantly updated. This allows researchers to track how new models perform in Estonian compared to older ones.

“For the best results, researchers need as much and as diverse input as possible,” said Krister Kruusmaa, Visiting Lecturer of AI and Digital Humanities at Tallinn University. According to him, users of the AI Barometer have complete freedom in choosing what to ask. “You can be creative when asking questions. If you don’t have good ideas right away, you can start by asking for help with something practical or inquiring about your local area,” Kruusmaa suggested. Researchers can later use the questions people ask the AI to create datasets to improve the Estonian language skills of open-source language models. The fun part of using the barometer is that the responses from less advanced models are often unintentionally comical. “We encourage everyone to share funny and interesting responses with others!” said Kruusmaa.

The AI Barometer was developed by Associate Professor Kairit Sirts and Junior Researcher Fellows Hele-Andra Kuulmets and Aleksei Dorkin from the University of Tartu, and Visiting Lecturer Krister Kruusmaa from Tallinn University. The barometer was created as part of the project “Estonian language support in open-source large generative language models”, which involves the University of Tartu, Tallinn University of Technology, Tallinn University, and the Institute of the Estonian Language. The project is funded by the national programme “Estonian Language Technology 2018–2027”.

Pictured are members of the working group: Associate Professor Kairit Sirts and Junior Research Fellow Hele-Andra Kuulmets from the University of Tartu and Visiting Lecturer Krister Kruusmaa from Tallinn University.