From pretraining towards finetuning

-    As part of a strategic move to make our research more industrially relevant and increase the innovation activities of NorwAI, we are slowly moving away from pretraining generic language models to fine-tuning domain-specific models, says Professor Jon Atle Gulla. 

The Director of the Norwegian Research Center for AI Innovation has for some time now directed the teams working on our LLMs named NorLLM, to engage in domains where language models are modeled towards specific purposes.

There are several projects where this policy is now being implemented.

-    During the summer we have fine-tuned multiple language models for the project management/construction domain.  With the help from the «Bedre megaprosjekter» team, we are collecting supervised datasets that help us adjust the behavior of our model to solve identified project management issues in these large construction projects.

The work is done in collaboration with Professor Eilif Hjelset at the Department of Civil and Environmental Engineering at NTNU. «Bedre Megaprosjekter» is a national development program that aims to contribute to the success of the largest and most complex projects in the country. 

Health

Also, there is now a preliminary project with Hemit and St. Olavs hospital to assess interesting hospital use cases that can be supported by specialized language models.  

-    As the idea here is to take advantage of sensitive health data, we need to proceed cautiously and make sure that we meet all necessary regulations and certification requirements, says Jon Atle Gulla.   

Hemit HF is the technology company of the Ministry of Health of Central Norway. Hemit develop, manage and operate common ICT systems for all hospitals in Trøndelag and Møre og Romsdal. This means that they serve over 20,000 of Norway's most important ICT users 24 hours a day, seven days a week.

Finetuning for finance

-    And soon we will have a new post doc candidate in place that will start fine-tuning language models for the financial sector in collaboration with DNB, he continues. 

Early experiments with Schibsted suggest that these fine-tuned models are competitive with the much larger international language models, even though they are less expensive to train, are based on smaller datasets, and are small enough to the hosted locally if needed.

A new LLM

NorwAI has since June this year been preparing a new Norwegian language model that is bigger than our existing models, is free of copyrighted material and has some more advanced reasoning capabilities.  

-    We will use the 24B parameter Magistral model from Mistral as the starting point, says Jon Atle Gulla, adding: 
-    We continue pre-training the model with improved Norwegian training data on our Idun supercomputer cluster.

An API

NorwAIs portofolio of models have since last year been available ob the Hugging Face platform. Many has asked the Research Center if an API service will be available. Jon Atle Gulla is happy to announce that this is close to be realised in the near future

-    To make our latest large model more accessible to both partners and other interested parties, we are together with Telenor’s AI factory in Oslo in the process of setting up a hosted service that will allow our models to run on Telenor’s infrastructure.  

He says An API interface to the first version of our new Magistral model is planned for the NorwAI Innovate conference in a few weeks, though the NorwAI team intend to continue the pretraining until November with additional training data from the health sector and other domains.

-    The idea is gradually to offer a range of NorwAI language models over this Telenor API and help organizations test and use our models without having their own infrastructure for downloading, managing and running large language models.  This is the first time we set up a real hosting service for our language models, and the collaboration with Telenor’s AI factory has been very constructive and efficient, says Professor Jon Atle Gulla. 

By Rolf D. Svendsen

2025-09-05