Data preprocessing for AI language models

In this video, we explain how our digital ecosystem and our Retrieval Augmented Generation solution work and what benefits they can offer (German with English subtitles):

Introducing language models or other AI- based systems poses enormous challenges for companies. Most companies quickly realize that their data is not AI-compatible, i.e. it cannot be read by ChatGPT & Co.

This is because the AI is unable to interpret many connections that are easy for humans to understand – for example, turnover compared to a time period in tabular form. In contrast, the language model easily understands information in coherent sentences, such as: “Sales with customer X amounted to Y euros in May 2024.”

This is where Brain4Data comes in. We embed your data in a semantic context so that it can be understood by language models. This opens up unimagined possibilities, from internal search engines to troubleshooting and automated responses to customer inquiries.

Employees can access information from a wide variety of document types (e.g. invoices, transport registers or personnel plans) or programs (e.g. CRM and ERP systems or SAP applications) in a matter of seconds without having to deal with specific search terms or the respective software. This reduces the communication effort in the company considerably.

Depending on the system, the language models are not limited to simply outputting data. They can also interpret it and draw logical conclusions. This means that they not only answer specific questions – for example about project costs, the personnel situation, stock levels or similar – but can also abstract answers to complex questions.

This results in almost unlimited opportunities for companies to take their workflows from merchandise management to sales and HR management to the next level. AI-ready data represents an enormous competitive advantage that will only grow with the increasing capabilities of artificial intelligence.

Users decide for themselves how they want to use their processed, AI-ready data. This data can be interpreted by all common language models which means that companies can easily rely on programs such as ChatGPT or Microsoft’s Copilot if they wish to do so.

We also provide our own language model which uses Retrieval-Augmented Generation (RAG) and generative AI from the Oracle Cloud Infrastructure.

On the one hand, this ensures that the language model cannot “hallucinate”, i.e. make up information. It only uses the available data and includes detailed source information in its answers. On the other hand, the queries are processed exclusively in the Oracle data center in Frankfurt. The company data is stored there under the highest security standards in compliance with the GDPR. This guarantees that no unauthorized person can access the information.

We compile information from very different sources (e.g. CRM and ERP systems, tender portals, HR and accounting, merchandise management, Excel, SAP, etc.). This means that every employee can continue working with their familiar programs. The existing IT infrastructure within the company remains untouched. No change management or expensive additional hardware is required.

As with all Brain4Data services, our Augmented Intelligence principle also applies here: our systems do not decide over the heads of those involved; they just provide them with the data they need to take an informed decision. Ultimately, however, it is always the human being who makes final call!