Research & Application – 3rd wave of AI

read more

LeoLM was pre-trained with 2 billion primary English language tokens and additionally with 65 billion specifically filtered and deduplicated tokens from web texts of the OSCAR-2301 corpus. The model was fine-tuned using six German or German-English language datasets.

The quality of the model was improved by using linear RoPE scaling and Flash Attention 2 to improve training efficiency and doubling the context length to 8k tokens.

Below you will find a detailed description of the model, the associated repositories and corresponding chat demonstrators.

The project was developed in cooperation between the Hessian AI Service Centre and LAION gemeinnütziger e.V.. Many thanks for the excellent co-operation and support.

Detailed description: LeoLM: Igniting German-Language LLM Research | LAION
Demonstrator 7b: Gradio
Repository 7b: LeoLM/leo-hessianai-7b · Hugging Face
Demonstrator 13b: LeoLM 13b Chat – a Hugging Face Space by LeoLM
Repository 13b: LeoLM/leo-hessianai-13b · Hugging Face

read more

This architecture enables

• cost-effective memory decoding in Hyena blocks by representing convolutions as state space models (modal or canonical form) or as truncated filters.
• Low latency, faster decoding and higher throughput than Transformers.
• Improved training and inference-optimal scaling laws compared to optimised transformer architectures such as Llama-2.
• By training on sequences of up to 32k, longer prompts can also be processed.

This makes StripedHyena 7B the first alternative model that is competitive with the best open source transformers in short and long context evaluations.

The project was developed in cooperation between the Hessian AI Service Centre and Together Computer Inc. Many thanks for the excellent cooperation and support.

Detailed description: Paving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond Transformers
Repository 13b Foundation model: togethercomputer/StripedHyena-Hessian-7B · Hugging Face
Repository 13b Chat model: togethercomputer/StripedHyena-Nous-7B · Hugging Face

read more

The model is a general basic model that has neither been tuned for commands nor optimised for chats or other applications.

The model was trained in cooperation between the German Research Centre for Artificial Intelligence (DFKI) and the Hessian AI Service Centre.

Repository: occiglot/occiglot-7b-eu5 · Hugging Face

read more

The manual annotation of large data sets is not only error-prone, but also tedious and requires a lot of human resources. Intelligent data documentation and data cleansing are therefore key solutions to these challenges, with the aim of optimising the preparation of high-quality datasets for AI applications.

In this context, this project focuses on the potential of machines to assist in the documentation of potentially inappropriate content by utilising the knowledge stored in Transformer models. This could significantly reduce the amount of human labour involved in data preparation.

The aim is to develop intelligent data documentation and data cleansing that can be offered as services to recognise inappropriate content. Planned steps include the training of documentation models for images, the extension to text and tabular data and the automatic documentation of mixed data. Generative and axiomatic synchronisation of the documentation models and provision as a service will ensure practical suitability and marketability.

The results of this research project are used in the AI Service Centre as modules for data documentation, cleansing and quality assurance.

read more

A particular interest lies in the use of modular and parameter-efficient transfer learning methods. Such methods update only a fraction of the parameters of a model, add a small number of trainable parameters or reuse existing parameters. Other methods learn smaller models from larger models or combine several modules.

In this sense, the use of various prompting techniques, i.e. analysing and using in-context learning capabilities (where a model learns from examples in the prompt) is very promising to adapt models on-the-fly; or in the case of instructional tuning as a related learning technique to tune models to specific requirements. Retrieval augmented generation uses external knowledge to extend the capabilities of a model.

In addition, we dive deep into model architectures, i.e. we interpret and edit model-internal representations. This can be done, for example, by tracing the flow of information or analysing individual model components. Such an approach views the modules of the model as part of a stream that can be traced back to the input at any point, with the processing of the stream at certain points leading to measurable results.

We find it interesting to see which tasks benefit from different approaches. Since today’s very large models perform very well on a large number of tasks, in the case of language-only models, we are particularly interested in their reasoning capabilities on most, if not all, of the more traditional NLP tasks.

read more

GPUs are powerful computing units for many data- and processing-intensive workloads and outperform CPUs by several orders of magnitude in these tasks. For this reason, we are investigating how the speed advantages of GPUs can be utilised for data processing on modern storage and fast networks.

The results so far show that GPUs can use techniques such as heavyweight decompression and pruning to achieve a significant increase in data load and processing bandwidth that is currently unattainable for CPU systems.

read more

It has been shown that SQL queries for single-node setups can be converted into a series of operations that are comparable to tensor runtimes and that the performance of this type of execution is comparatively high. However, it is still unclear whether these advantages can also be realised in a networked environment.

Since distributed query processing requires efficient key-based network shuffling, the overlapping of network and computational operations, and the handling of skew, the transformation is not trivial. In this line of research, we investigate how to transform distributed queries so that they can be efficiently executed via Tensor frameworks with the same benefits.

read more

Current high-performance models have black box structures that learn from huge amounts of data generated by collecting code in the “wild”. Such datasets contain vulnerable code or even malware. It is also worth noting that many of these models learn from these datasets by sequentially predicting the completion of a particular code, ignoring the rich structure it contains.

The Code Transformers project aims to develop efficient code models by first investigating the depth of their understanding and their limitations. It will then design customisable and modular structures that can use different techniques such as multimodal training or neurosymbolic learning. Such models can utilise rich metadata associated with the code, they adapt to user preferences and are more trustworthy as they can provide explanations for their generations.

read more

To provide some background, let’s dissect the project title word by word. “Multimodality” means that there are non-text elements present in the document. For example, these can be images, tables or charts. The modalities video and audio are not considered because they are quite rare in business documents. Importantly, multimodality means that the modalities are interleaved and their interplay is complex, i.e. we will go beyond simple pairs of image and corresponding text caption.



The term “structure” refers to everything which goes beyond text as a plain sequence of characters. Structure can manifest itself via linebreaks in a poem, chapters in a book or columns in a newspaper. Further, “structure” also describes the relations between text and non-text elements of a document. This can be implicit by the spatial arrangement or explicit via references to tables and charts.



A “Transformer” is a special kind of Neural Network (i.e. a model that can learn from data) and can be used for analysis and prediction. In the language domain, most recent breakthroughs (i.e. ChatGPT) were mainly due to this architecture. Currently, the vanilla Transformer pushes hardware to the limit as its key component, the so-called self-attention, scales poorly with sequence length. Thus, we also want to look into recent attention-free models, e.g. state space models such as Mamba.


The goal of the project is to provide an easy-to-use Code-Framwork for understanding multimodal content input.

read more

Sum product networks (SPNs), an important member of the PC family, are characterised by their efficient inference capability, their ability to be trained with relatively little data and hyperparameters, and their ability to model the uncertainty of their predictions. However, using SPNs on computing platforms such as CPUs, GPUs and FPGAs is a challenge. In particular, these networks do not fit well with massively parallel architectures due to their sparse connections and convergence to a single output node.

This project addresses these issues by developing an MLIR-based SPNC compiler. The SPNC compiler aims to bridge the gap between advanced hardware architectures and the accessibility of these powerful models to AI practitioners without hardware knowledge.

This integration maximises the benefits of the hardware and minimises the learning curve for AI researchers and developers. By efficiently computing SPNs on specialised hardware, such as FPGAs and IPUs, our project opens up new avenues for AI applications.