The fairness dilemma in the EU AI Act:
Why AI fairness will always be a matter of conflicting objectives
1. The countdown is on
The countdown is ticking inexorably: on 2 August 2026, key requirements of the EU AI Act will come into force and become binding for numerous stakeholders. What for many companies long sounded like an abstract regulation from Brussels is now becoming an operational reality. Anyone who develops, markets or deploys AI systems in Europe will in future have to comply with extensive compliance, risk and governance requirements, the likes of which we have previously encountered mainly in heavily regulated sectors such as finance or healthcare.
However, if, like us, you have been developing and operating machine learning systems in regulated environments for years, many of the fundamental principles of the AI Act will not be new to you. Issues such as traceability, model risks, human oversight and reproducible training processes are already part of our day-to-day work.
For me personally, ethics has always been a central guiding principle in my work. Driven by this deep conviction, I discussed the implications of this development and the resulting strategic steps in depth as early as 2025, together with the Supervisory Board of KENBUN IT AG. We were already aware at that time that AI governance is not a downstream compliance process, but a central component of modern software architecture.
With a view to the deadlines in August 2026, which are now inexorably drawing nearer, I have taken another close look at the Regulation over the past few weeks. Within this framework, the EU sets out a series of non-binding ethical principles intended to serve as a foundation. These seven principles include:
- Human agency and human oversight
- Technical robustness and safety
- Privacy and data governance
- Transparency
- Diversity, non-discrimination and fairness
- Social and environmental well-being
- Accountability
During my renewed in-depth analysis, one term in particular from this list emerged as arguably the greatest regulatory and technical sticking point: the concept of ‘fairness’. Whilst values such as transparency or IT security allow for clear technical standards, fairness remains, at its core, an extremely vague concept.
For several years now, I have been supporting financial institutions and other organisations in the implementation and operation of AI-powered systems. In doing so, I have observed time and again that the greatest challenges rarely arise from the algorithms themselves. The question of how we can document decisions in a traceable manner, control risks and clearly define responsibilities is usually far more complex. It is precisely at this point that regulatory requirements come up against mathematical limits. And these limits are crucial to understanding what you can actually achieve within your organisation under the AI Act.
2. What does the law require? The challenge of fair data foundations
A closer look at the EU AI Act reveals that the regulation defines the concept of fairness primarily in terms of avoiding discriminatory effects. A key reference point for developers and operators is found in Article 10, which sets out the requirements for data and data governance.
For designated high-risk AI systems – such as those used in human resources, credit decisions or critical infrastructure – the legislator requires that the datasets used be subject to appropriate governance processes. We must examine them for potential biases that could lead to discrimination prohibited under European law. Furthermore, the AI Act requires that training, validation and test data must be relevant, representative and, as far as possible, error-free and complete.
From a regulatory perspective, this objective is entirely understandable. Nobody wants an AI system to disadvantage job applicants because historical data reflects societal imbalances. From a machine learning perspective, however, a more challenging question arises: what does fairness actually mean in concrete terms – and can it be defined unambiguously in mathematical terms? In my day-to-day practice, I see significant conflicts of interest here.
SCHUFA Case Study: When Regulation Meets Core Processes
I recently observed just how profoundly these requirements are already transforming the economy today, using SCHUFA as an example. Following landmark rulings by the European Court of Justice (ECJ) on automated decision-making under the GDPR, Germany’s best-known credit reference agency was forced to fundamentally overhaul its scoring system. This shows that regulatory pressure is not a theoretical threat for the future, but is already changing real business models today.
However, the question remains as to whether the new system will also withstand the forthcoming requirements of the EU AI Act. As creditworthiness assessment systems are classified as high-risk applications under that legislation, the issue of algorithmic ‘fairness’ takes centre stage. SCHUFA thus finds itself facing a Herculean task, one that many financial institutions and companies will be confronted with in the coming months. This is because the attempt to make a scoring model – which has evolved over time – ‘completely non-discriminatory’ by decree often fails in practice, not because of a lack of will on the part of those involved, but because of the inexorable laws of statistics.
3. The mathematical reality: Why fairness cannot be unambiguously defined
When we examine the mathematical foundations of machine learning, we quickly encounter a central problem: fairness is not a uniform mathematical concept. There are several established fairness criteria that reflect different notions of justice – and which, under certain conditions, we cannot satisfy mathematically at the same time.
Jon Kleinberg and his co-authors at Cornell University made an important contribution to this understanding in their paper “Inherent Trade-Offs in the Fair Determination of Risk Scores”. They demonstrated mathematically that it is impossible to satisfy certain fairness criteria simultaneously when baseline rates differ between population groups.
In ML practice, three established core metrics usually come into conflict with one another:
- Predictive parity (calibration): The predictive power of a risk score should be identical for all groups. For example, if a score predicts a probability of occurrence of 80 per cent, this probability should carry the same significance regardless of group membership.
- Equalised Odds (equality of error rates): The rates of false positives and false negatives must be comparable across different groups.
- Demographic Parity: Positive decisions made by the system must occur with the same frequency across different groups.
However, as soon as groups differ in relevant statistical characteristics, irreconcilable conflicts of interest arise between these requirements. Optimising in favour of one fairness metric inevitably leads, mathematically speaking, to other fairness criteria being met to a lesser extent.
To illustrate these mathematical trade-offs clearly, I have prepared an interactive simulation below. It shows a classic credit scoring model for two groups (A and B) with statistically different historical baseline rates and makes the inexorable mathematical interactions directly visible:
Credit Scoring Fairness Simulator
This model simulates two population groups with different historical baseline rates (credit scores). Adjust the threshold to see how the approval rates (demographic parity) change.
Group A (historically higher rate)
Approval rate: --%
Group B (historically lower rate)
Approval rate: --%
4. Country Focus: Different Approaches in Germany and France
This issue is a key focus for me and my team, and not just from a regulatory perspective. As a company that has been supporting financial institutions and other organisations in several countries for years in the development, implementation and operation of AI systems – including in the field of fraud detection – we regularly see how differently requirements are interpreted in various markets. Through our activities in Germany and a growing presence in France, I have observed interesting differences in approach.
Germany: Focus on governance and traceability
In Germany, structured processes, risk management and traceable documentation are particularly emphasised. Institutions such as the Federal Network Agency (BNetzA) and the Federal Office for Information Security (BSI) help to create an environment that is strongly focused on compliance and traceability. This poses a considerable challenge, particularly for small and medium-sized enterprises. In addition to legal issues, they must also document statistical and technical decisions in a traceable manner – often with limited staff resources.
France: Focus on innovation and practical testing
In the public debate, France places greater emphasis on combining regulation with the promotion of innovation. Supported by national AI initiatives and companies such as Mistral AI, there is intense discussion on how Europe can build technological sovereignty without unnecessarily restricting innovation potential. The French data protection authority, the CNIL, began early on to support companies through guidelines, recommendations and sandbox initiatives. Such regulatory test environments enable organisations to trial new AI applications under controlled conditions and address regulatory issues at an early stage.
Both approaches ultimately pursue the same goal: to promote trustworthy AI systems. However, there is a noticeable difference in the emphasis placed on the various measures.
5. Findings and actions: How companies should respond
In my projects, I regularly observe that we can rarely resolve fairness issues through technical optimisation alone. Particularly in fraud prevention or other high-risk application areas, there is no configuration that guarantees maximum accuracy, minimal error rates and absolute fairness all at the same time.
Companies should use the time remaining until the relevant AI Act requirements come into full effect to establish robust governance structures.
In my experience, four areas of action in particular have proven effective:
-
Develop realistic expectations regarding fairness: Abandon the notion of a completely neutral AI. This goal is unattainable, both technically and organisationally. Our primary aim should not be the theoretical elimination of every conceivable bias, but rather a conscious, transparent approach to the mathematical risks and conflicting objectives.
-
Establish interdisciplinary decision-making structures: Questions of fairness cannot be answered by code alone. Lawyers, data scientists, compliance officers and business departments must work together to define which fairness criteria take the highest priority in a specific application context.
-
View documentation as a strategic tool for providing evidence: This transforms documentation from a tedious chore into a protective shield. If you can provide a comprehensive account of which data was used, which risks were identified and why certain model decisions were made, this will significantly strengthen your position vis-à-vis regulatory authorities.
-
Use human-in-the-loop approaches where risks are high: Particularly in high-risk applications, AI should not make decisions in isolation. Human review and final decision-making remain an indispensable part of responsible systems.
6. The Often Overlooked Advantage: Regulation as a Competitive Asset
In many discussions, the EU AI Act is primarily viewed as a regulatory burden. From our practical experience, however, this perspective only tells part of the story.
One of our teams is currently supporting a company in Thailand with the secure introduction of agentic AI solutions. What has surprised us positively is the extent to which existing and emerging regulatory frameworks in Thailand are aligned with European principles. This applies both to data protection requirements under Thailand’s Personal Data Protection Act (PDPA) and to the country’s evolving approach to AI governance, which is being shaped by organisations such as the Electronic Transactions Development Agency (ETDA).
Naturally, there are important legal and regulatory differences between jurisdictions. However, from a technical and organisational perspective, the similarities are often greater than many organisations expect. Our experience in this project has shown that companies which have already internalised the principles of the GDPR and the EU AI Act frequently possess a significant head start when operating in other regulatory environments.
Processes for governance, risk assessment, documentation, human oversight and auditability can often be adapted with relatively limited effort. In practice, this means that investments made today to comply with European regulations may deliver benefits far beyond the European market.
For this reason, I believe it is a mistake to view the EU AI Act solely through the lens of compliance costs. Organisations that build robust and trustworthy AI governance frameworks today are not only preparing themselves for European regulation; they are also strengthening their ability to deploy AI solutions internationally in a responsible and scalable manner.
7. Outlook: What comes after 2026?
August 2026 marks not an end point, but an important milestone on the path towards a long-term European AI regulatory framework. Further requirements will come into force in the following years. These include additional obligations for providers of general-purpose AI models in 2027, as well as the gradual integration of the AI Act’s requirements into existing sectoral regulations by 2028.
The EU AI Act pursues the legitimate aim of making the use of AI systems more transparent, secure and trustworthy. However, implementing these requirements demands more than simply grappling with legal texts. Anyone wishing to develop and operate AI systems responsibly must understand that fairness is not a clearly defined mathematical state. Rather, it involves a series of conflicting objectives that we must consciously assess, document and manage.
This is precisely where the real challenge of modern AI governance lies: not in the pursuit of perfect fairness, but in dealing with these inevitable conflicts of interest in a transparent and accountable manner.
How is your organisation preparing for the requirements of the EU AI Act? Join the discussion with us or contact the team of experts at KENBUN IT AG for a professional exchange on AI governance, compliance and responsible AI systems.
8. Sources and further reading
EU regulation
- Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act)
https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Academic papers on fairness in machine learning
- Kleinberg, J., Mullainathan, S., Raghavan, M. (2016): Inherent Trade-Offs in the Fair Determination of Risk Scores. https://arxiv.org/abs/1609.05807
- Kearns, M., Roth, A. (2019): The Ethical Algorithm: The Science of Socially Aware Algorithm Design.
Interactive demonstrations
- Google PAIR Explorables: Attacking Discrimination with Smarter Machine Learning. https://pair.withgoogle.com/explorables/attacking-discrimination-in-ml/
