The AI Engineer's Guide to Surviving the EU AI Act

Chapter 1. Understanding the AI Regulations

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

As people, organizations, and the public sector increasingly rely on AI to drive decision-making, the technology must be trustworthy. The EU AI Act aims to provide a legal framework for developing, deploying, and using AI technologies within the EU, emphasizing safety, transparency, and ethical considerations. The EU AI Act is a regulatory framework for artificial intelligence that includes specific requirements for AI systems of different risk categories within the European Union. This book is focused on understanding and implementing the regulatory requirements set by the European Union’s legislation on artificial intelligence. Please note that this book is not a substitute for obtaining professional legal advice.

Warning

The author is not a lawyer. Nothing in this book is legal advice. The intersection of law and artificial intelligence is a complex subject that requires expertise beyond the scope of AI, data scientists, and machine learning. Legal considerations surrounding AI systems can be complex and far-reaching. If you have any legal concerns related to the machine learning systems you are working on, seek professional legal advice from qualified experts in the field.

The Motivation for the EU AI Act: Trustworthy AI

As AI becomes increasingly intertwined with our daily lives, one of the challenges we face is learning to navigate the uncertainty that comes with it. This uncertainty is inherent to AI. ML/AI models’ predictive accuracy has long been considered a single evaluation criterion when building an AI system. With the widespread use of AI in critical areas such as human resources, transportation, finance, medicine, and security, there is a growing need for these systems to be trustworthy. So, there is a need to extend the building principles of AI systems beyond predictive accuracy. Accuracy alone is not sufficient to build trustful AI applications. Trustworthy AI is a concept, and within the AI community, it is used interchangeably with responsible AI, ethical AI, reliable AI, or values-driven AI. To better understand trustworthy AI, let’s start with its definition.

Trustworthy AI is an umbrella term that refers to artificial intelligence systems that are designed and developed with principles such as fairness, privacy, non-discrimination, and robust mechanisms to ensure reliability, security, and resilience. These systems must be adaptable to diverse and changing environments and robust against various types of disruptions, including cyber threats, data variability, and operational changes. They should operate transparently and be held accountable, with continuous monitoring and evaluation to respect human rights, including privacy and freedom from discrimination, and to ensure adherence to democratic values.

The term “Trustworthy AI” is quite complex and includes a long list of concepts and principles, which you can see visualized in Figure 1-1 (https://dl.acm.org/doi/pdf/10.1145/3555803). These concepts will lay the foundation for understanding the EU AI Act. The seven trustworthy AI requirements in figure 1-1 come directly from the key requirements outlined in the Ethics Guidelines for Trustworthy AI developed by the European Commission’s High-Level Expert Group on AI (AI HLEG). This is also known as the Assessment List for Trustworthy Artificial Intelligence. In this chapter, I discuss each of these requirements.

It’s important to note the distinction between the technical and ethical requirements of trustworthy AI. The technical requirements include robustness, explainability, transparency, reproducibility, and generalization of the AI model. On the other hand, ethical requirements comprise fairness, privacy, and accountability for AI.

Trustworthiness in AI is grounded in three pillars: lawfulness, ethics, and robustness. The AI system should be lawful, meaning it should comply with all regulations, to guarantee the fair market, economic benefits, and protection of citizens’ rights. The AI system should be built on ethical principles and values, by including all stakeholders and establishing the appropriate feedback mechanism. Finally, the AI system should be robust, meaning it requires designing a system by considering risks and safety factors.

Standing on these three pillars, AI trustworthiness implies seven key requirements or concepts that AI systems should implement to be trustworthy:

1. Human agency and oversight

2. Technical robustness and safety

3. Privacy and data governance

4. Transparency

5. Diversity, non-discrimination and fairness

6. Societal and environmental well-being

7. Accountability

Figure 1-1. [The foundation and seven requirements of Trustworthy AI.]

As you can see, the term „trustworthy AI“ is a broad and multifaceted term. In the following sections, I provide a primer explanation of each requirement.

Human Agency and Oversight

Human oversight is crucial in developing and operating AI systems. Human agency refers to the ability of humans to make informed decisions and maintain control over AI systems. AI systems should support human agency by providing transparency, interpretability, control, and intervention mechanisms that allow humans to understand and influence the system’s decisions and actions.

At the same time, human oversight involves establishing governance processes and mechanisms that enable human monitoring, evaluation, and intervention in the operation of AI systems. This includes mechanisms for implementing the transparency and interpretability of the decision-making processes based on the results produced by the AI model. Additionally, the AI system should provide the human control mechanisms (e.g., ability to override, adjust, or stop the AI system). As depicted in the Figure 1-2, I distinguish between four different modes of human oversight:

1. Human-in-Command - implies an explicit authorization of any AI systems action by humans.

2. Human-in-the-loop - Humans are involved in decision making processes which are made by the AI system. Often this involvement means navigating the direction of decisions, where the AI system makes a prediction. The level of intervention and control might depend on the risk-level of the system. Medical AI applications will require intensive human intervention and control, compared to the less critical AI applications.

3. Human-on-the-Loop - the human plays an observer role while monitoring all the actions of the AI system and intervening if necessary.

4. Human-out-of-the-Loop - in this mode the AI system operates independently without human intervention. Obviously, this scenario is possible when the decisions based on the AI system predictions can be programmatically manifested. Often, this mode is applied in trading applications which require data processing at speed and human oversight is not realistic.

Figure 1-2. [The different levels of human oversight in AI.]

Each of the modes described above are suitable for different AI use cases. Humans should play a crucial role in AI systems. Proper implementation of the human agency and oversight requirement is crucial for designing and developing the ethical AI where we can benefit from the strength of AI while protecting us from the potential risks.

Technical Robustness and Safety

AI systems need to be accurate, reliable, and reproducible. They need to have a fall-back plan if something does not work properly. Generally, in computer science, we denote robustness as the ability of a computer system to cope with errors during execution and manage erroneous input. Robustness in AI refers to the ability of a machine learning (ML) model to maintain stable and reliable performance across varied and unexpected environmental conditions, data distributions, and perturbations.

In other words, a machine learning (ML) model is considered robust when it continues to make accurate predictions despite changes made to the input data. Such changes are expected to fall within a certain range. When the model’s performance degrades beyond the acceptable level due to these changes, it is no longer considered robust. Robustness is a crucial requirement for establishing trustworthiness in AI systems deployed in real-world scenarios, especially for deploying ML models in safety-critical applications, such as autonomous vehicles, cyber-physical systems, and healthcare, where performance degradation can have severe consequences.

I distinguish robustness on different levels of AI systems, namely data, algorithms, and underlying software systems. Data robustness means that the AI model training should be a robust process. The model should be trained on the various input data distributions and in this way provide a robustness against the distributional shift. The key idea behind strategies for data robustness is to expose the AI model to a diverse range of scenarios and perturbations during training, enabling it to learn robust representations and generalize better to unseen data variations encountered during deployment. A typical example is autonomous driving where the model should be trained on different weather conditions to guarantee the AI system reliability under different weather conditions. In cybersecurity, we are talking about training intrusion detection systems on a diverse set of attack vectors, network traffic patterns, and system logs to improve their ability to detect novel and evolving cyber threats.

The algorithm’s robustness denotes robustness against algorithm-level attacks, which might exist in various forms. For instance, the decision-time attack performs input sample perturbation to mislead the prediction of a given model. Training-time (poison) attacks poison with adversarial samples into the training data to change the system’s answer to patterns. A robust AI algorithm should be able to handle noise and perturbations in the input data, as well as distributional shifts or changes in the data distribution between training and testing. A robust AI algorithm should be immune against intentionally crafted inputs designed to fool the model (adversarial attack) and handle out-of-distribution examples not seen during training. For example, robust language models can handle misspellings, grammatical errors, slang, and out-of-vocabulary words without degrading performance.

Finally, AI System-level robustness considers illegal inputs on the AI products level and should consider the holistic view of the entire AI lifecycle, and the interactions between the AI model and the larger system it is embedded in. The AI lifecycle typically consists of phases like data creation/acquisition, model development, model evaluation, and model deployment. A holistic approach examines potential issues, risks, and failure modes across all these phases.

In academia, robustness has two main aspects: Non-adversarial Robustness and Adversarial Robustness. Non-adversarial robustness refers to how well the model performs when it is given corrupted or altered inputs that may not match the original data distribution. It means that the model can handle unintentional image corruptions, distributional shifts, or changes in the data generating process while maintaining its performance. On the other hand, adversarial robustness relates to the model’s ability to resist adversarial examples, which are intentionally crafted input perturbations designed to fool the model into making incorrect predictions. Adversarial robustness aims to defend against such intentional attacks.

To evaluate the robustness of AI and machine learning models, several metrics are used, the most well-known are:

1. Performance metrics under different conditions:

• Accuracy: The proportion of correct predictions over the total number of predictions, evaluated on different data subsets like training, validation, and test sets.

• Error rate: The proportion of incorrect predictions, including false positives and false negatives.

• Sensitivity (recall or true positive rate): The proportion of positive cases correctly identified by the model.

• Specificity (true negative rate): The proportion of negative cases correctly rejected by the model.

1. Robustness curves: These are plots showing how the model’s performance (e.g., accuracy or error rate) changes as a function of parameters affecting data quality, such as noise level, missing values, outliers, data size, or feature selection.

Privacy and Data Governance

AI systems must ensure full respect for privacy and data protection. Since privacy is a fundamental right within the European Union, privacy is a prerequisite to building trustworthy AI systems. Training AI models, especially the Foundation Models, requires a large amount of data.

In many cases, such as recommender systems or personalized prediction, some level of personally identifiable information (PII) might be necessary for such AI systems. Handling PII is a subject of regulations within the EU and USA under GDPR and CCPA. Therefore, the information system designers are requested to ensure that “privacy by design” is implemented within the system, as well as data governance processes are established within the organization. Generally, data governance is a data management function that guarantees the availability, usability, integrity, and security of data collected and used in an organization. Key elements of data governance are visualized in Figure 1-3.

Figure 1-3. [Key elements of data governance (adapted from https://learning.oreilly.com/library/view/data-governance-the/9781492063483/)]

Data governance is becoming more important because the size of data is growing, organizations are becoming data-driven, and increasingly more people have access to data. The ultimate goal of data governance is to enhance the trustworthiness of data, which is the foundation for trustworthy AI. To ensure trust in data, three key aspects of data governance must be addressed: discoverability, security, and accountability of data.

Discoverability refers to the availability of the dataset’s metadata, data provenance (lineage), and domain entities glossary. Furthermore, data quality is crucial in building trust in data. Data should be correct, complete, timely, and integral. Additionally, data governance determines procedures that guarantee that the right data is assessed by the appropriate people in the organization. Data security and privacy are about protecting data and ensuring adherence to regulations such as GDPR (or CCPA).

The third aspect of data governance - accountability - is treating data as a product. This means that the “product thinking” philosophy is applied to data units for each business domain. In this way, we consider data as a product, a data unit that is valid within a business domain where the domain team has a clear responsibility for this data unit. Technically, a data product is an autonomous, read-optimized, standardized data unit containing at least one domain dataset that exists because of the needs of data users.

AI development should respect the fundamental right for privacy at any point of the AI application lifecycle: data collection, data processing, storage, model design, development, and deployment. The privacy issues might become evident in many aspects of AI systems, such as social media platforms using AI to analyze user behaviors and preferences can inadvertently expose sensitive information through targeted advertising or data breaches.

Furthermore, AI can enhance the capabilities of surveillance systems, leading to potential overreach in monitoring activities. This can result in a loss of anonymity and freedom, as every action can be watched and recorded. For example, governmental structures using facial recognition technologies in public spaces can track individuals without their consent, potentially leading to misuse of power and privacy violations.

Moreover, AI systems can amplify biases present in their training data. When these biases affect how data is collected, processed, or used, they can disproportionately impact the privacy of certain groups. For instance, AI-driven credit-scoring models might use biased data that discriminate against certain racial or gender groups, affecting their privacy related to financial data.

Example metrics to define and track for privacy and data governance include:

• Data encryption levels - Degree of data protection during transmission and at rest.

• Access controls - Effectiveness of policies and tools that manage who can access or alter data.

• Data retention and deletion policies - Compliance with data minimization principles and regulations.

• Data Product Usage metric.

Transparency

Data, AI models, and software systems that include AI components should be transparent and provide traceability. This is identified as a necessary element of trustworthy AI. In almost every guide for AI (https://arxiv.org/pdf/1906.11668), transparency for AI is the most requested principle in working with AI, disregarding the industry or use case.

Transparency in AI doesn’t have a uniform technical definition. We refer to transparency in AI systems as the ability to understand how an AI model works internally and makes decisions. It provides explanations about the AI system’s components, algorithms, decision-making process, and overall functioning comprehensibly to stakeholders such as users, developers, and regulators. Key dimensions of AI transparency include explainability, data, algorithm, governance, and communication transparency:

• Explainability: Also known as “XAI,” refers to the ability to explain in human-understandable terms how and why an AI model arrived at a particular output or decision for an individual case. This is important for building trust and accountability. As depicted in Figure 1-4, explainable AI models might not be the models with the highest accuracy, so depending on the use-case requirements, we might select explainable but less accurate ML algorithm. Techniques for creating explainable AI are used at every stage of the ML lifecycle, including analyzing data for model development, incorporating interpretability into the system architecture, and producing post-hoc explanations of system behavior. These techniques are referred to as the three stages of AI explainability: pre-modeling explainability, explainable modelling, and post-modeling explainability. (https://insights.sei.cmu.edu/blog/what-is-explainable-ai/ and https://towardsdatascience.com/the-how-of-explainable-ai-explainable-modelling-55c8c43d7bed)

• Data transparency: Openness about the training data used to build the AI model, including its sources, characteristics, and potential biases or limitations.

• Algorithmic transparency: Visibility into the AI algorithms and how they process input data to generate outputs or decisions. This includes understanding the features, weights, and logic the model uses.

• Governance transparency: Documenting key decisions made during the data and AI development process, establishing clear protocols and responsibilities, and ensuring organizational oversight. This aspect is particularly relevant to compliance with the EU AI Act.

• Communication transparency: Sharing information about the AI system’s purpose, capabilities, and limitations to relevant stakeholders in a timely, clear, and accessible manner.

Figure 1-4. [Explainability and the prediction accuracy trade-off.]

AI transparency aims to open AI systems’ “black boxes.” This metaphorical “black box” represents the internal operations of AI systems, which are often complex and opaque. By opening this “black box,” humans can understand the inner workings of AI systems, audit them for errors or biases, and foster trust in their use.

Examples of metrics to define and track for transparency in AI include the following:

• Explainability index: A measure of how easily the decisions of the AI system can be explained to users. Availability of explanations across the AI system lifecycle. The types of explanations provided (e.g. feature importance, counterfactual examples, visual aids). The scope of explanations (global model behavior vs local predictions).

• Documentation completeness: Availability and clarity of documentation on the AI system’s purpose, functionality, limitations, model training details, and feature engineering process.

• Algorithmic auditability: Ease of auditing AI algorithms for compliance and performance.

Diversity, Non-Discrimination and Fairness

AI systems must mitigate unfair bias. Discussion about bias and fairness in machine learning is a recent topic with intensified research since 2016 (https://dl.acm.org/doi/pdf/10.1145/3616865). Data collected in an unequal manner and processed by non-diverse teams might potentially cause harm. Therefore, incorporating diversity and inclusion throughout an AI system’s complete lifecycle is a clear requirement for trustworthy AI. This also includes accessibility, which is the user-centric approach that guarantees that the usability of the AI system takes everyone into account, especially people with disabilities. This is why inclusive engineering plays an important role in AI system development, not just technical engineering. Furthermore, it is crucial to establish a strategy or set of procedures to mitigate bias and promote fairness in the AI system, both regarding the use of input data and algorithm design.

Addressing potential bias should happen at every stage of the AI system development. In this book, I adopt the CRISP-ML(Q) (https://arxiv.org/pdf/2003.05155) development process model to specify a fair and effective AI system development strategy. During all phases of that development model, there is a need to set up processes for testing and monitoring for potential biases and detecting non-representativeness in data. Evaluating a complete end-to-end AI development workflow for fairness is important for building successful AI products. Improving diversity and representativeness in AI products is a step towards compliance with the EU AI Act and providing value for AI system users. Establishing a robust mechanism for flagging issues related to bias, discrimination, or poor performance of AI systems allows developers and end-users to become aware of these issues, for instance, implement flagging of issues such as bias detection tools, reporting tools for bias categories, and detailed guidance on the reporting of issues by the affected person.

An underestimated and less technical aspect of an organization’s data culture is often a lack of educational and awareness initiatives for bias and fairness in AI. These initiatives are intended to help AI product managers, designers, and engineers become more aware of the possible bias they can inject while designing and developing AI systems.

The EU AI Act ensures that every entity along the AI system’s value chain, from producer to deployer, is responsible for providing users with a fair and ethical experience. There are several metrics used to evaluate fairness in machine learning (ML) systems, as illustrated in the Figure 1-5. These metrics aim to quantify potential biases or disparities in the model’s predictions across different demographic groups. Here are some commonly used fairness metrics to evaluate the fairness and non-discrimination:

• Demographic Parity: This metric measures the difference in the probability of receiving a positive outcome (e.g., getting hired or receiving a loan) between different demographic groups. It is calculated as the difference or ratio of the selection rates for each group. A value of 0 (for difference) or 1 (for ratio) indicates demographic parity.

• Equal Opportunity: This metric evaluates whether the true positive rates (TPR) are equal across different groups. In other words, it measures whether the model has an equal chance of correctly predicting positive outcomes for individuals from different groups who actually deserve a positive outcome.

• Equalized Odds: This is a stricter version of equal opportunity that requires both the true positive rates and the false positive rates (FPR) to be equal across groups. It ensures that the model not only has equal opportunity but also equal mistreatment across groups.

• Predictive Parity: This metric measures the difference in the positive predictive values (PPV) or the precision of the model across different groups. It evaluates whether the precision of the model is consistent for all groups.

• Individual Fairness: This metric ensures that similar individuals are treated similarly by the model, based on a predefined similarity metric.

• Counterfactual Fairness: This metric evaluates whether changing an individual’s sensitive attribute would change the model’s prediction for that individual.

Figure 1-5. [High-level illustration of fairness intervention types in ML. Technical approaches for fairness improvement and bias mitigation are typically applied before modeling (pre-processing), at the point of modeling (in-processing), or after modeling (post-processing). (Source: https://dl.acm.org/doi/pdf/10.1145/3616865)]

Societal and Environmental Well-being

AI systems should benefit all human beings. The EU Act is concerned specifically with the societal and environmental aspects of trustworthy AI areas, which I discuss below.

First, there are proven negative impacts of AI technology on the environment. The primary environmental concerns associated with AI include energy consumption, carbon footprint, e-waste, and indirect environmental impact, as visualized in the Figure 1-6 (https://proceedings.mlsys.org/paper_files/paper/2022/file/462211f67c7d858f663355eff93b745e-Paper.pdf). AI and machine learning models, especially large ones, require significant computational power and consume a lot of electricity. The data centers housing these processors thus require vast amounts of energy to run and to cool. Large-scale AI systems have a high energy demand, which leads to a significant carbon footprint. To mitigate this issue, efforts are being made to power data centers with renewable energy sources. The production of AI hardware requires precious materials, which can be environmentally damaging. Additionally, hardware quickly becomes obsolete, contributing to electronic waste. Based on its applications, AI can have indirect effects on the environment. For instance, AI-driven automation can lead to increased production capacities and potentially increase resource consumption. Conversely, AI can optimize systems to be more energy-efficient, reduce waste, or enhance resource management, thereby potentially having a positive impact.

Figure 1-6. [The amount of computing resources used to train deep learning models has increased 300,000x in six years from 2012 to 2018. (Source: https://openai.com/index/ai-and-compute/)]

The proposed “Green AI” approach considers efficiency as an additional evaluation criterion, together with accuracy. “Green AI” efficiency is measured as the number of floating-point operations required to generate a result (https://dl.acm.org/doi/pdf/10.1145/3381831). FinOps, which stands for Financial Operations, is suitable for managing the environmental and financial impacts of AI. FinOps is a cloud financial management practice for companies to manage their cloud spending more efficiently (https://www.finops.org/).

Second, AI is changing the way we work in three key ways: by automating tasks, changing work processes, and affecting job skills. Many jobs could be automated by AI, potentially leading to the displacement of jobs globally. Especially, jobs that involve repetitive tasks are most at risk (https://www.innopharmaeducation.com/our-blog/the-impact-of-ai-on-job-roles-workforce-and-employment-what-you-need-to-know). At the same time, AI technology has the potential to create new jobs that demand advanced technological and analytical skills, such as machine learning engineers, data scientists, and AI ethics specialists. Additionally, AI can drive the development of innovative products and services, opening up new career opportunities in emerging sectors like AI-driven digital assistants and smart devices. (https://www.forbes.com/sites/forbesbusinesscouncil/2023/07/26/how-does-artificial-intelligence-create-new-jobs/).

Additionally, the rise of AI has led to job polarization, with high-skilled jobs increasing and low-skilled jobs facing obsolescence. This worsens income inequality and creates challenges for those without access to education and training. AI in the workplace can also lead to increased stress, anxiety, and job insecurity due to the fear of job loss and uncertainty about the future (https://www.forbes.com/sites/elijahclark/2023/08/18/unveiling-the-dark-side-of-artificial-intelligence-in-the-job-market/).

Lastly, in addition to ethical concerns and job changes, AI systems can negatively impact society at large and democracy through misinformation and surveillance. AI, particularly generative AI, can produce misinformation and disinformation at scale. Tools like ChatGPT, pi.ai, or perplexity.ai can create realistic but false content, which can be used to spread misinformation. AI-generated “deepfakes” can create convincing but fake videos and images, which can be used to manipulate public opinion (https://pubmed.ncbi.nlm.nih.gov/37881016/).

Figure 1-8 shows that at least 75 out of 176 countries actively utilize AI technologies for surveillance. AI systems used in surveillance can lead to unauthorized data collection and potential misuse of personal information. This can result in privacy violations and unauthorized access to sensitive data. AI in law enforcement, such as predictive policing, can lead to biased outcomes and discrimination against certain demographic groups. (https://carnegieendowment.org/research/2019/09/the-global-expansion-of-ai-surveillance?lang=en)

Figure 1-7. [AI surveillance technology is being adopted by a larger number of countries and at a faster pace than experts typically believe. AI technologies for surveillance are actively utilized by at least 75 out of 176 countries worldwide. (Source: https://carnegieendowment.org/research/2019/09/the-global-expansion-of-ai-surveillance?lang=en)]

Furthermore, AI technology might negatively impact democracy, contributing to misinformation in elections, spreading propaganda, and influencing voter behavior. AI-driven content personalization can create “information bubbles,” where individuals are only exposed to information that reinforces their existing beliefs, leading to increased polarization and social fragmentation.

Addressing these challenges requires robust ethical guidelines, transparent regulatory frameworks, and ongoing public dialogue to ensure that AI is developed and deployed to support democratic values and societal well-being.

Accountability

AI systems must be developed and operated responsibly. AI accountability means establishing mechanisms for holding AI developers and users accountable for their systems’ impacts along the AI system’s complete development cycle. (https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf)

Accountability implies that information about the AI system’s purpose, design, data, and processes is available to internal and external auditors. As an example, we can refer to the Google Responsible Generative AI Toolkit that covers risk and mitigation techniques to address safety, privacy, fairness, and accountability (see Figure 1-8). This includes maintaining detailed documentation and records of the AI development process, decision-making, and outcomes to enable traceability. Logging and record-keeping are key for accountability. Additionally, accountability means that an AI system explains or justifies its decisions.

Finally, accountability guarantees that there are mechanisms for redress for each incorrect or unjust output. Redress refers to the mechanisms and processes put in place to minimize or correct negative impacts or unfair outcomes caused by AI systems. Designing AI systems should provide appropriate opportunities for feedback, relevant explanations, and defined procedures for escalating concerns. (https://ai.google/responsibility/principles/)

Figure 1-8. [Google Responsible Generative AI Toolkit covers risk and mitigation techniques to address safety, privacy, fairness, and accountability. (Source: https://ai.google.dev/responsible)]

In the context of trustworthy AI, several mechanisms should be implemented to create accountability. Here are the key mechanisms:

• Clear responsibility guidelines and processes: Establishing guidelines and clear responsibilities for various stakeholders involved in the AI system lifecycle, including developers, deployers, and users.

• Transparency and explainability: Maintaining detailed documentation of the AI development process, training data, algorithms used, and decision-making criteria to enable traceability. Ensuring that AI systems are transparent about their capabilities, limitations, and decision-making processes. Provide clear explanations for decisions made based on the AI predictions.

• Human oversight and intervention: Establishing human checks and oversight, especially for high-stakes AI decisions, with the ability to override the AI when needed.

• Auditing and evaluation: Conducting regular internal and third-party audits of AI systems to identify and eliminate biases, ensuring compliance with regulations and ethical standards. By systematically auditing AI systems using the CRISP-ML(Q) process model, organizations can assess their AI applications’ quality, reliability, and trustworthiness. The audit depth and focus areas can be tailored based on the risk and criticality of the use case. The audit might be in the form of internal and external evaluations.(https://www.isaca.org/resources/isaca-journal/issues/2018/volume-1/the-machine-learning-auditcrisp-dm-framework)

• Redress and complaint mechanisms: Establishing user-friendly channels for submitting complaints, feedback, or requests for explanations about AI decisions. Providing clear processes for affected individuals to challenge decisions and seek remedy.

Trustworthy AI is a motivational concept behind the legislation of the EU AI Act. In this chapter thus far, you examined in detail the seven key requirements and concepts that AI systems should implement to be trustworthy.

EU AI Act in a Nutshell

The EU AI Act is about the adoption of human-centered and trustworthy artificial intelligence. The objective is to guarantee a high level of protection for health, safety, and fundamental rights as outlined in the Charter, including democracy, the rule of law, and environmental protection, against the damaging effects of AI systems in the European Union, all while fostering innovation. To understand the AI Act, you must understand the legislation’s scope, to whom this Act applies, and the timeline of the implementation of the AI system compliance. I’ll explain the detailed requirements for different risk categories throughout the book and in this chapter, I will give a high-level overview of the EU AI Act.

The subject of the EU AI Act is the following (https://artificialintelligenceact.eu/ai-act-explorer/):

1. “Harmonized rules for the placing on the market, the putting into service, and the use of AI systems in the Union;

2. Prohibitions of certain AI practices;

3. Specific requirements for high-risk AI systems and obligations for operators of such systems;

4. Harmonized transparency rules for certain AI systems;

5. Harmonized rules for the placing on the market of general-purpose AI models;

6. Rules on market monitoring, market surveillance, governance and enforcement;

7. Measures to support innovation, with a particular focus on SMEs, including startups.”

The EU AI Act is structured into the following main sections (see also Figure 1-9):

• Titles (I, II, III, etc.) - The Act is divided into 12 main Titles, covering different aspects like general provisions, prohibited AI practices, requirements for high-risk AI systems, governance, etc.

• Chapters (under each Title) - Each Title contains several Chapters, grouping related Articles together

• Articles (under each Chapter) - The Articles lay out the specific rules and obligations. They are numbered sequentially within each Chapter. For example, Article 6 under Chapter 1 of Title III covers the “Classification rules for high-risk AI systems”

• Recitals (paragraphs explaining the rationale behind provisions) - The recitals are numbered paragraphs at the start that explain the reasons and context behind the Act’s provisions. They can help interpret the Articles but are not binding.

• Annexes (documents appended to the Act) - The Act has 13 Annexes (I-XIII) that provide supplementary information like lists, definitions, procedures, etc. You can navigate to a specific Annex by its number or title, e.g., “Annex III” or “High-Risk AI Systems Referred to in Article 6(2)”.

Figure 1-9. [AI Act Overview (Source: https://artificialintelligenceact.eu/ai-act-explorer/)]

AI Definition

The AI Act also provides a set of definitions. We are interested in the definition of the AI system for engineering purposes. According to the AI Act Chapter 1, article 3:

“‘AI system’ means a machine-based system that is

• designed to operate with varying levels of autonomy and

• that may exhibit adaptiveness after deployment, and

• that, for explicit or implicit objectives, infers, from the input it receives,

• how to generate outputs such as

  • predictions,

  • content,

  • recommendations, or

  • decisions

• that can influence physical or virtual environments;”

It’s important to emphasize that, given the rapid and unpredictable technological and AI development, the above definition is not entirely static, and a dynamic regulatory tool has been integrated into the AI Act.

The following AI techniques and approaches refer to machine-based systems mentioned in the AI Act:

• Machine Learning methods (such as supervised, unsupervised, semi-supervised machine learning)

• Deep Learning methods

• Reinforcement Learning

• Logic- and Knowledge-based Methods (such as logic programming, expert systems, inference and deductive engines, reasoning engines)

• Statistical approaches

• Bayesian Methods

• Search and Optimization Approaches

Models such as BERT, DALL-E, Claude, Mistral and GPT-2 to 4o have become increasingly popular. They are trained on extensive data using self-supervision at scale and can be adjusted to various downstream tasks. These are called models’ foundation models. In light of the recent development of the foundation models (https://arxiv.org/pdf/2108.07258), the AI Act provides a definition of the ‘general-purpose AI model’, as follows:

“AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market”

Although not explicitly mentioned, the definition of the “general-purpose AI model” implies the notion of the foundation model and generative AI as a whole.

The EU AI Act aims to ensure that the above techniques used in digital and physical products, services, or systems are safe and respect existing laws on fundamental rights and European Union values.

Key Players From Creation to Market Operation

The AI Act is a legal framework for developing, distributing, and using AI in the EU. This regulatory framework applies to companies and persons that make, bring in, or distribute AI systems or general-purpose AI models in the EU. It also applies if they are located in a country outside the EU. Figure 1-10 visualizes AI systems key players, defined by the EU AI Act.

Let’s examine the key players who are affected by the AI Act throughout the complete AI system lifecycle.

• Provider - a person or organization that develops an AI system or model, or has one developed, and puts it on the market or uses it, whether for payment or for free, under their own name or brand.

• Importer - a natural or legal person in the EU who trades an AI system bearing the name or trademark of a person in a non-EU country.

• Distributor - a person or company in the supply chain, other than the provider or the importer, that makes an AI system available on the Union market.

• Authorized Representative - a person or organization in the EU who has been given written permission by an AI system or general-purpose AI model provider to carry out the responsibilities and procedures outlined in this Regulation on their behalf.

• Deployer - refers to a person or organization using an AI system for professional activities.

• User - persons impacted by an AI system.

Figure 1-10. [AI systems key players, defined by the EU AI Act.]

Classification of the AI Systems by Risk Levels

Another important aspect of the AI Act is its risk-based approach. According to the regulation’s risk-based classification, AI systems might be categorized as one of the four possible classes :

1. Prohibited,

2. High-risk,

3. Limited-risk, and

4. Minimal risk.

Figures 1-10 picture an estimation of the number of AI systems for each risk category. Roughly 20% of all AI systems are expected to be classified as high-risk (https://www.appliedai.de/assets/files/AIAct_WhitePaper_DE-1_2024-03-04-123021_uwgf.pdf).

Figure 1-11. [AI system risk categories.]

The EU AI Act prohibits certain AI systems that are considered to pose an unacceptable risk. The bans are aimed at AI systems that could heavily influence or harm people’s decision-making or infringe upon their rights. Specifically, the AI Act prohibits AI practices such as using manipulative subliminal techniques, exploiting vulnerabilities based on age, disability, or social circumstances, and making high-stakes assessments based on profiling or predictive traits without sufficient human oversight. The legislation also prohibits the unregulated use of ‘real-time’ biometric identification in public spaces, unless under strict conditions for law enforcement purposes related to significant public safety concerns. (See more in AI Act Chapter II Article 5, https://artificialintelligenceact.eu/chapter/2/)

High-risk AI systems, as defined by the EU AI Act, are either intended to be used as safety components of products or are products themselves. These AI systems fall under specific Union legislation and require third-party conformity assessments. In addition, high-risk AI systems include those specified in a designated list, which are subject to stringent compliance requirements due to their potential impact on the health, safety, or fundamental rights of individuals, such as biometrics, critical infrastructure, education, etc. (See more in AI Act Chapter III, https://artificialintelligenceact.eu/chapter/3/). As many products traded on the extended Single Market in the European Economic Area (EEA), AI products, classified as high-risk AI systems must receive the CE Marking to be certified within the EU (please see Figure 1-12).

Figure 1-12. [CE marking. The letters ‘CE’ appear on many products traded on the extended Single Market in the European Economic Area (EEA). They indicate that products sold in the EEA have been evaluated to meet stringent safety, health, and environmental protection requirements. https://single-market-economy.ec.europa.eu/single-market/ce-marking_en]

Limited-risk AI systems pose lower risks, mostly in the form of manipulation, deception, or impersonation. Usually, these AI systems interact with individuals and are neither of unacceptable risk nor high-risk. This category includes systems like chatbots and deepfakes. For limited-risk AI systems, the main obligation is transparency - providers must disclose that the output is AI-generated and users must be made aware they are interacting with an AI system. There are also requirements to label deepfakes clearly. (See Art. 52, I)

The last category is minimal-risk AI system. According to the EU AI Act, minimal-risk AI systems are those that pose little to no risk to people’s safety, fundamental rights, or privacy. These include AI applications like video games, spam filters, or simple image editing tools that perform narrow tasks with limited decision-making capabilities.

It is important to understand that the AI Act classifies use cases and not the AI/ML technology or algorithms itself. Furthermore, the AI Act compliance obligations apply to each AI system or ML/AI model separately and not the organization as an entity. Proper classification has an impact on the estimation of the AI system requirements because different risk categories imply different governance and MLOps architectural decisions and obligations. However, determining the risk level of an AI application is challenging as it relies on various factors and involves classifying how the capabilities of a non-deterministic system will affect users and systems that may interact with it in the future. In Chapter 4 I outline a risk classification framework for AI systems.

Please note that scientific research activities and fields like military AI are outside the scope of the AI Act.

Enforcement and Implementation of the EU AI Act

The implementation of the EU AI Act has a clear timeline (https://artificialintelligenceact.eu/ai-act-implementation-next-steps/). This timeline, as depicted in Figure 1-13, implies that organizations that create high-risk AI systems have a grace period to prepare their internal processes. By the end of the grace period they are responsible for complying fully with the AI Act. Here are the key milestones and deadlines for the implementation of the AI Act in the European Union:

• EU AI Act entered into force (August 2024): The AI Act is effective 20 days after it is published in the Official Journal of the EU.

• 6 months after entry into force (Q4 2024 - Q1 2025): Prohibitions on unacceptable risk AI systems are effective.

• 9 months after entry into force (Q1 2025): Codes of practice for general-purpose AI (GPAI) models must be finalized.

• 12 months after entry into force (Q2-Q3 2025): Obligations on providers of GPAI models begin.

• 18 months after entry into force (Q4 2025 - Q1 2026): The Commission implements acts on post-market monitoring for high-risk AI systems.

• 24 months after entry into force (Q2 - Q3 2026): Obligations on high-risk AI systems listed in Annex III become applicable. Member states must have implemented rules on penalties and established AI regulatory sandboxes.

• 36 months after entry into force (Q4 2026 - Q1 2027): Obligations for high-risk AI systems not prescribed in Annex III but subject to existing EU product safety laws are effective.

• By the end of 2030: Obligations apply for specific AI systems that are components of large-scale EU IT systems in areas like security and justice.

Figure 1-13. [The timeline outlining the key milestones and deadlines for the implementation of the AI Act in the European Union.]

The Full Picture of Compliance

Table 1-1 briefly outlines the end-to-end compliance process to adhere to the EU AI Act requirement for AI systems that are either high-risk or GPAI. Practically, the AI Act aims to ensure that all digital and physical products that employ AI as a feature or as a core product are used in a safe and ethical manner, following the EU fundamental rights. Therefore, the EU establishes product regulation. Like every product that is used within the EU, the AI products should be either CE marked or prohibited.

Warning

Again, the author is not a lawyer. This book does not provide legal advice.

Table 1-1. [The end-to-end process steps towards AI Act compliance.]

AI Act Compliance Steps
Steps	Guiding Questions
1. Creating AI System Landscape and AI System Risk Classification	How many AI systems are in place and are intended to be put in production? What risk category do these AI systems belong to?
2. Compliance Requirements Structuring for AI Systems and GPAI	What requirements do we need to fulfill?
3. Compliance Operationalization	What processes, structures, engineering practices, and roles need to be established to comply with the AI Act?
4. Pre-market Compliance Verification	What has to be done before placing AI systems on the market and putting them into service? What are internal and external conformity assessments for compliance verification? How do we CE-marking our AI systems? Where should our AI system be registered (database)?
5. Post-market Continuous Compliance	What has to be done to ensure compliance after putting the AI system into service?

Many organizations using and embedding AI technology into their products are asking the same question: “What does the AI Act mean to us?” To answer this question, I outline the five most essential steps for providers and deployers of AI systems to reach compliance with the AI Act.

Before you set up all the technical and organizational processes for establishing compliance, you need to determine the scope of the AI Act compliance. This means you would need to do an inventory of all AI use cases (systems) that are deployed in production. The initial question that you will answer is, what AI system risk categories do we have in our organization? The compliance measures depend on that classification outcome. Having a prohibited category of AI use cases would mean an immediate preparation for abandoning such systems.

Knowing the risk category will shape all the requirements you have to fulfill to become compliant. It’s important to distinguish between obligations for AI systems and for usage General Purpose AI in the systems. For each of the groups, there are different articles in the AI Act, and you will have to review the respective articles for concrete requirements. I discuss risk classification for AI systems in detail in Chapter 4.

Structuring compliance requirements from the previous stage sets up the scope of the technical and organizational processes, governance structures, engineering practices, and roles that have to be established for successful compliance with the AI Act. The goal is to fully and correctly understand the obligations. The AI Act outlines specific requirements that must be met, including establishing risk management measures, providing technical documentation, ensuring human oversight, and guaranteeing AI systems’ accuracy, robustness, and security. It also involves setting up a quality management system that covers testing, incident reporting, data management, record retention, and logging. I call this phase “Compliance Operationalization,” which is the focus of this book.

After the engineering part of the compliance, providers and deployers of AI systems have to demonstrate compliance, which means establishing the internal and external conformity assessment. The conformity assessment is followed by the CE-Marking and registering the AI system in the database. We have to perform these steps before placing the AI system in the market and putting it into service. We call this phase “Pre-market Compliance Verification”

“Post-market Continuous Compliance” is the final and, at the same time, continuous phase. The goal is to demonstrate adherence to the AI requirements in the ongoing phase. This means we demonstrate that the AI system remains compliant, disregarding all changes. This phase means post-market surveillance and AI system monitoring.

Penalties for EU AI Act Violation

The AI Act proposes significant penalties and fines for companies that violate its rules and requirements. Table 1-2 provides the key details on the penalties:

Table 1-2. [Key details on the companies’ penalties for violating the EU AI Act.]

Violation Details	Penalties
Prohibited AI Practices (AI practices listed in Article 5, such as exploiting vulnerabilities, social scoring, real-time biometric identification in public spaces, etc.).	Administrative fines of up to €35 million or 7% of their total worldwide annual turnover for the preceding financial year, whichever is higher.
High-Risk AI Systems (non-compliance with requirements for high-risk AI systems under Article 10).	Administrative fines of up to €30 million or 6% of their total worldwide annual turnover, whichever is higher.
Other non-compliance with other obligations under the AI Act, apart from Articles 5 and 10.	Administrative fines of up to €20 million or 4% of their total worldwide annual turnover, whichever is higher.
Providing incorrect, incomplete, or misleading information to authorities.	Administrative fines of up to €10 million or 2% of their total worldwide annual turnover, whichever is higher.

Existing AI Regulations and Standards

The EU AI Act serves as a comprehensive model for other nations, underscoring the balance between innovation and trustworthy AI. Its strict regulations on biometric systems and high-risk AI applications already set a high standard for AI governance globally. Let’s briefly review the existing landscape of AI regulations.

UNESCO AI Ethics Recommendations

(https://unesdoc.unesco.org/ark:/48223/pf0000381137)

The UNESCO Recommendation on the Ethics of Artificial Intelligence, adopted in November 2021 by UNESCO’s 193 Member States, provides a framework for ethical AI development. It emphasizes human rights and offers guidance in 11 key policy areas. It outlines core values and principles for the ethical development and deployment of AI, including respect for human rights, inclusion and diversity, fairness and non-discrimination, transparency and explainability, accountability, safety and security, and sustainability. The recommendation aims to influence ethical AI practices globally and includes innovative tools and methodologies to translate ethical principles into practice.

U.S. Executive Order on Trustworthy AI

(https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/)

The Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence emphasizes the potential benefits and risks of AI. It highlights the importance of responsible AI governance to address societal issues and prevent negative consequences such as fraud, bias, and threats to national security. The order stresses the need for collaboration across government, industry, academia, and civil society to ensure the safe and responsible development and application of AI.

The administration aims to align executive departments and agencies with eight key principles and priorities for AI development and governance, with a focus on engaging various stakeholders including businesses, academia, civil society, labor unions, and foreign partners. This policy framework underscores the commitment to leading AI governance to promote its responsible advancement and enhance American society, economy, and security.

China Generative AI Service Law

(https://www.ashurst.com/en/insights/new-generative-ai-measures-in-china/)

The Cyberspace Administration of China (CAC) and other government agencies issued Interim Measures for the Administration of Generative Artificial Intelligence Services. These measures signal China’s proactive approach to regulating generative artificial intelligence (AI) services. The regulations take effect on August 15, 2023, and oversee companies offering generative AI services to the general Chinese population. Generative AI technology includes models that produce text, graphics, audio, and video. The Interim Measures recognize the potential for foreign investment while also promoting innovation and research. Future artificial intelligence laws are expected to expand the regulation scope beyond generative AI. Given the potential penalties or shutdowns for non-compliant services operating in China, it is essential to ensure compliance.

NIST AI Risk Management Framework

(https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

The NIST AI Risk Management Framework (AI RMF) Generative AI Profile draft released in April 2024 identifies 12 key risks that are unique to or exacerbated by generative AI (GAI) technologies. It includes risks related to CBRN information, confabulation, dangerous or violent recommendations, data privacy, environmental impact, human-AI configuration, information integrity, information security, intellectual property, obscene, degrading, and/or abusive content, toxicity, bias, and homogenization, and value chain and component integration. The draft proposes over 400 actions organizations can take to manage these risks, organized by the AI RMF core functions of govern, map, measure, and manage, aiming to help organizations identify and mitigate the unique risks posed by generative AI technologies.

IEEE P2863 - Recommended Practice for Organizational Governance of Artificial Intelligence

(https://standards.ieee.org/ieee/2863/10142/)

The IEEE’s Recommended Practice for Organizational Governance of Artificial Intelligence outlines governance criteria for AI development and use within organizations, including safety, transparency, accountability, responsibility, and bias minimization. It also provides steps for implementation, auditing, training, and compliance. The document aims to ensure responsible AI practices by integrating internal governance and engaging with external AI governance instruments.

These laws and frameworks collectively aim to regulate AI in a way that promotes innovation while ensuring safety, transparency, and respect for human rights, similar to the objectives of the EU AI Act.

Conclusion

The main motivation of the EU AI Act is trustworthy AI. In this first chapter, we examined the notion of trustworthiness in AI, which is based on three pillars: lawfulness, ethics, and robustness. I outlined the seven key requirements for trustworthy AI systems, which include human agency and oversight, technical robustness and safety, privacy and data governance, transparency, diversity, non-discrimination and fairness, societal and environmental well-being, and accountability. The EU AI Act focuses on adopting human-centered and trustworthy artificial intelligence to protect health, safety, fundamental rights, democracy, the rule of law, and environmental protection in the European Union.

Understanding the AI Act involves knowing its scope, applicable entities, and the timeline for AI system compliance. The AI Act is centered around AI use cases and establishes rules for developing, distributing, and using AI in the EU that are relevant to businesses and individuals in and outside the EU. It covers providers, importers, distributors, authorized representatives, deployers, and users of AI systems.

Despite having many already available AI regulatory frameworks, the EU AI Act is the first regulation that establishes a comprehensive risk-based legal framework for artificial intelligence within the European Union that has an impact beyond the EU. The EU AI Act classifies AI systems into four categories based on risk: Prohibited, High-risk, Limited-risk, and Minimal risk. It prohibits AI systems that pose unacceptable risks and requires compliance with specific obligations for high-risk and limited-risk AI systems. The Act also emphasizes transparency and labeling requirements for limited-risk AI systems, and it applies to individual AI systems or ML/AI models, not to organizations as entities.

Classifying AI systems and understanding the corresponding compliance requirements will shape the volume of Data and AI governance, as well as MLOps engineering practices. In the following Chapter, I’ll discuss the CRISP-ML(Q) - structured ML development process and MLOps, which provides technical and organizational best practices for ML operationalization.

Quiz

References:

Li, Bo, et al. “Trustworthy ai: From principles to practices.” ACM Computing Surveys 55.9 (2023)

Braiek, Houssem Ben, and Foutse Khomh. “Machine Learning Robustness: A Primer.” arXiv preprint arXiv:2404.00897 (2024).

https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment

https://www.researchgate.net/publication/369110605_The_assessment_list_for_trustworthy_artificial_intelligence_A_review_and_recommendations

https://certx.com/ai/fostering-robust-ai-understanding-its-importance-and-navigating-the-eu-artificial-intelligence-act/

Data Age 2025: The Evolution of Data to Life-Critical Don’t Focus on Big Data; Focus on the Data That’s Big Sponsored by Seagate

https://insights.sei.cmu.edu/blog/what-is-explainable-ai/

https://towardsdatascience.com/the-how-of-explainable-ai-explainable-modelling-55c8c43d7bed

Caton, Simon, and Christian Haas. “Fairness in machine learning: A survey.” ACM Computing Surveys 56.7 (2024)

Wu, Carole-Jean, et al. “Sustainable ai: Environmental implications, challenges and opportunities.” Proceedings of Machine Learning and Systems 4 (2022).

Novelli, Claudio, Mariarosaria Taddeo, and Luciano Floridi. “Accountability in artificial intelligence: what it is and how it works.” AI & SOCIETY (2023).

https://www.marktechpost.com/2024/05/02/top-artificial-intelligence-ai-governance-laws-and-frameworks/