I. AI in Health: Market Growth and Regulatory Landscape Challenges

Market growth

Over the past five years, the landscape of artificial intelligence (AI) in healthcare has witnessed unprecedented growth and transformation. The global AI in healthcare market was valued at approximately $11 billion in 2021, and projections estimate it could surpass $64 billion by 2027, representing a compound annual growth rate of over 40% (Grand View Research, 2023). This surge is largely driven by advancements in machine learning and big data analytics, which are transforming diagnostics, treatment personalisation, and patient monitoring.

In terms of investment in AI in healthcare, growth is also significant. In 2022, the largest investments in AI were made in the field of medicine and health, totalling $6.1 billion. In Europe, in 2019, investments were valued at $1,368.59 million and are expected to reach $36,015.25 million by 2027. Notably, the European Commission is investing more than €1 billion a year in AI, mainly through the Horizon Europe and Digital Europe programmes, with the aim of mobilizing additional investment from the private sector and Member States to reach an annual investment volume of €20 billion over this decade. (Statista, 2023; Business Market Insights, 2023).

AI use cases

AI is increasingly transforming both healthcare and research, with notable projects leading advancements in diverse areas:

• In healthcare, IDx-DR developed by Digital Diagnostics has become a landmark FDA-approved AI solution for diabetic retinopathy screening, allowing non-specialists to detect the disease early in primary care settings.
• Meanwhile, IBM Watson for Oncology has been applied as a decision-support tool, aiding oncologists in making treatment recommendations by analysing vast amounts of clinical data and relevant medical literature.
• In hospital settings, Aidoc, a radiology-focused AI platform, assists radiologists by flagging critical conditions like brain hemorrhages in real-time, significantly reducing diagnosis times.
• In research, AlphaFold by DeepMind has revolutionized drug discovery by predicting protein structures with remarkable accuracy, addressing a critical challenge in biomedical research and enabling faster drug target identification.

These projects exemplify how AI applications are extending from molecule discovery in research to actionable, life-saving support in clinical practice.

Growth in Data Acquisition

The increase in AI in healthcare goes hand in hand with a growing need for personal data to train and improve these algorithms. Therefore, two approaches are possible for obtaining data: 1) reusing existing data, such as radiographs from hospitals databases or clinical trial databases; and 2) generating data, often through medical devices, which directly train algorithms with already structured data.

However, it is still rare for companies developing AI to plan a budget to acquire data. This is due to the uncertainty, at least in Europe, of who owns the data. Is it the patient? Is it the hospital that generates it? Is it the state as a ‘public good’? All these questions are holding back the circulation of data, even though the consensus recognises that this data has a financial value.

Regulatory Inflation

The growing demand for data acquisition in healthcare AI has not only raised questions about who controls and benefits from this resource but has also spurred an increase in regulatory frameworks designed to address these challenges.

The regulatory response has been swift yet complex, reflecting a global effort to establish frameworks that accommodate rapid innovation without compromising data protection and patient safety. The European Union’s proposed AI Act aims to introduce specific guidelines and classifications for AI in high-risk sectors, including healthcare, while the U.S. has released the “Blueprint for an AI Bill of Rights,” setting preliminary principles for AI governance in sensitive applications (European Commission, 2023; U.S. White House, 2022). Looking ahead, the healthcare sector anticipates a steady increase in both AI applications and regulatory scrutiny. Experts suggest that regulatory frameworks will need to strike a delicate balance between innovation and compliance, enabling technologies to advance patient care while safeguarding data protection and ethical standards. As Dr. Eric Topol, a prominent AI healthcare advocate, stated: “AI holds transformative potential for healthcare, but its safe integration relies on trust and stringent regulatory oversight.”

Surge in Regulatory Approvals

Together with the trend toward increased regulation of AI, the surge in regulatory frameworks has been accompanied by a growing number of approvals for AI-driven tools, reflecting efforts to balance innovation with safety and ethical standards.  From 2019 to 2023, the number of new AI-driven health products approved by agencies like the U.S. Food and Drug Administration (FDA) and the European Medicines Agency (EMA) has more than doubled. For example, the FDA had approved only 14 AI-based medical devices by 2018, but this figure rose to over 300 by 2023 (FDA, 2023).

Key Goals of the Article

Building on the above discussion, this article series sets out to examine the critical data protection, ethical, and compliance challenges associated with AI in health, offering a comprehensive analysis of the evolving regulatory landscape and its implications for stakeholders.

Accordingly, Part I of the article will focus on the compliance requirements of EU regulations, with a particular emphasis on the GDPR and the recently adopted EU AI Act. Part II will analyse the legal requirements of the applicable U.S. framework and provide a global perspective, offering an overview of existing AI regulations and guidelines that may impact the healthcare and research sectors worldwide.

More specifically, the first section of Part I of the article series explores the unique characteristics of AI in health, focusing on the data protection and ethical challenges posed by the sensitive nature of health information and the immense volume of data processed by these systems. The second section examines the requirements of existing EU data protection regulations and guidelines for AI developers and deployers in the health sector, addressing how these frameworks tackle the challenges outlined in the first section. Finally, the third section introduces emerging regulatory frameworks for AI in the EU, outlining their objectives, key provisions, and the obligations they impose—highlighting a broader trend toward fostering responsible AI across sectors, with a particular emphasis on health.

II. AI in Health: Unique Features and Challenges

AI systems in health have the potential to revolutionise patient care, streamline clinical processes, and enhance medical research. However, the transformative nature of these technologies introduces unique characteristics that create complex data protection and ethical challenges. From the intricacies of the involvement of various actors in data processing and algorithmic decision-making of these systems to the opaque nature of AI models, AI systems often operate in ways that challenge traditional principles of transparency, accountability, and fairness.

In the health sector, where the stakes are high due to the sensitivity of patient data and the criticality of accurate medical decisions, these challenges are amplified. This section delves into the defining characteristics of AI systems and examines the data protection and ethical hurdles that arise as a result.

The “many hands” problem

Accountability related to the use of AI within the health sector presents unique challenges, especially given the complexity and "many hands" problem characterising AI systems, as multiple stakeholders contribute to their development and deployment. The diffusion of responsibility complicates efforts to trace harm when it occurs, as no single party is solely accountable for adverse outcomes.

In healthcare, for example, where AI-driven decisions impact patient lives, clear accountability mechanisms are vital to protect patient rights and ensure ethical care.  This situation becomes particularly complex when considering scenarios where AI systems directly influence patient treatment, such as drug and dosage recommendations. Suppose an AI system suggests a specific drug dosage for a patient, but the physician lacks an understanding of how the AI arrived at this recommendation. This creates a scenario where the physician must decide whether to rely on the AI or disregard it due to a lack of explainability. If the physician follows the AI’s advice and harm occurs, the question of who should be held responsible, whether the AI developers, the healthcare institution, or the physician, would come into play.

The "many hands" problem also raises accountability challenges from a data protection perspective. With numerous actors involved in the development and deployment of AI systems, it becomes difficult to determine who holds primary responsibility for safeguarding the rights and freedoms of affected individuals and ensuring compliance with applicable data protection regulations.

Leveraging extensive data for training

AI systems developed and used within the health sector require vast amounts of sensitive data, which are often extracted from original databases for AI purposes.

As a general principle concerning the use of AI in the health sector, any information and communication relating to data processing should be provided in a way that ensures that patients are in a position to clearly understand how their data is used, as well as the potential risks posed by the specific circumstances of the processing.  This principle is more easily upheld during the deployment stage of an AI system, where healthcare professionals and research institutions utilize the system for specific cases and feed it targeted datasets. However, significant challenges arise during the training stage of the AI system, where the system is trained on extensive datasets sourced from various origins. The diversity of data sources makes it difficult to provide detailed privacy notices to all affected individuals.

These challenges become even more pronounced when the training of the AI system only serves as a secondary purpose of data processing data.  In such cases, data originally collected for a primary purpose (e.g., provision of healthcare services, conduction of a specific clinical trial) is repurposed for training, creating a secondary use that complicates the process of re-informing patients about their data processing in the context of the AI development. This issue is particularly acute in the healthcare sector, where the patient’s identifying details are not held by the AI provider, especially since much of the health data is pseudonymised.

Opacity of the AI system’s function

As said, AI systems within the health sector require large volumes of diverse data to function accurately, including highly sensitive personal information such as genetic, health, and biometric data.

Regulations require fairness and transparency in the processing of health data for health data processing. This becomes challenging when applied to “black box” AI models, where the logic behind data processing is opaque, even to the developers. AI’s inherently complex processing can hinder transparency, as data subjects and even controllers may struggle to grasp the full extent of an AI system’s data use, making it difficult for the primary actors in the health sector to communicate effectively with patients about the scope and implications of AI-driven data processing, ultimately limiting patients' ability to make informed choices about their personal data.

At the same time, the purpose limitation consideration - a cornerstone of data protection frameworks- emphasises the importance of processing personal data only for specified and legitimate purposes, ensuring that individuals' privacy is safeguarded against misuse or overreach. This approach aims to prevent “mission creep,” where data collected for diagnostics or treatment could be repurposed for unrelated applications. Nevertheless, in practice, while developing an AI, the exact definition of the purpose can be under debate or influenced by the outcomes of the training process, creating a heightened risk of infringing the purpose limitation principle, as it becomes challenging to ensure that AI systems adhere strictly to the original purpose for which the data was collected. Additionally, AI systems may identify unforeseen correlations in the data, potentially leading to unanticipated uses of the information.

Furthermore, the accountability considerations outlined above are closely intertwined with data protection challenges, as data protection regulations require data controllers—the entities primarily responsible for safeguarding personal data—to demonstrate how data was processed and to provide clear explanations for the decisions made using that data. However, the inherent opacity of AI systems—stemming from their reliance on algorithms that analyse vast amounts of data to identify correlations—makes fulfilling this requirement practically complex.

Biased AI decisions

Bias in AI models used within the health sector is a critical concern, often stemming from underrepresentation of minority groups in the datasets used to train these models. The phenomenon is further compounded when AI models derive correlations from seemingly unrelated data points, which could inadvertently lead to discriminatory outcomes. For instance, AI models trained on proxy attributes—variables that may unintentionally correlate with sensitive categories like race or socioeconomic status—can lead to unintended bias in outcomes, infringing on the fundamental right’s principle of the prohibition of discrimination.

For example, Optum’s Predictive Health Algorithm, designed to identify patients in need of complex health interventions, was found to systematically favor White patients over Black patients. This bias arose because the algorithm used healthcare costs as a proxy for healthcare needs, inadvertently disadvantaging Black patients, who historically spend less on their health due to inequalities, despite having equal or greater health care needs. This bias occurred because the algorithm used healthcare costs as a stand-in for healthcare needs, unintentionally disadvantaging Black patients.

Additionally, Pulse oximetry, a commonly used tool in COVID-19 screening algorithms to measure blood oxygen levels, has shown biases against darker-skinned individuals, leading to underestimations of oxygen saturation. This discrepancy impacted the accuracy of COVID-19 screening algorithms and decisions about patient care, as studies revealed that darker skin tones can interfere with the sensor readings.

III. Meeting the data protection requirements under EU law

The existing data protection regulations have been deemed adequate and well-suited to protect the individuals’ fundamental rights and freedoms in terms of data protection, due to the technological neutrality that was envisioned from their inception. Considering the main challenges presented above, this section will focus on the key practical aspects of ensuring compliance with the GDPR.

EU – GDPR Compliance

1.Accountability

For AI to safely and effectively advance health-related services, accountability mechanisms must be embedded in its development, deployment, and application, by clearly delineating the data protection responsibilities of each stakeholder involved in the AI lifecycle, from developers to end-users. In this context, key components for data protection compliance when developing and deploying AI systems in the health sector among others include data mapping exercises, clear documentation of data flows, establishing and maintaining a current Record of Processing Activity (RoPA) and perform regular data audits to re-evaluate the conditions of processing and the need to retain or delete the processed data.

Under the GDPR, all those tasks must be documented in the Data Protection Impact Assessment (DPIA) where rights and freedoms of data subjects are balanced with the AI development. The DPIA must evaluate both source data processing (development) and routine use of the AI (commercialisation) while highlighting specific risks for each use. As AI development is an ongoing process, the DPIA should be regularly reevaluated. A good start to conduct the DPIA for companies is to use authorities' soft law on the regulation of AI as an “evaluation grid” for the conclusions of the analysis. Thus, by complying with AI soft law regulations, even not dedicated to data protection, AI developers should be able to achieve major milestone in their AI development.

In this perspective the conduction of Data Protection Impact Assessments may serve to respond proactively to unforeseen technological challenges and anticipate and/or mitigate the respective risks.

Another key aspect of GDPR compliance is identifying the appropriate legal basis for processing personal data through AI systems. In its recently released opinion on AI and the GDPR, the EDPB affirmed that the legitimate interest of the data controller can serve as an appropriate legal basis. However, this choice requires a thorough and well-documented assessment of the conflicting interests involved—namely, those of the data controller and the affected data subjects. This balancing test, often referred to as a legitimate interest assessment (LIA), must carefully determine that the controller's legitimate interests are not overridden by the interests, fundamental rights, and freedoms of the data subjects.

2.Provision of Information to Individuals

The GDPR places significant emphasis on informing individuals about how their data is used. More specifically, the GDPR enforces a level of transparency by requiring clear communication about the types of personal data used, the AI system's purpose, and the specific reasons for each decision. This is also in line with the EU AI Act, which, as detailed further in the next section of this article, now emphasises transparency as a requirement, mandating that developers make AI decision-making processes interpretable. In this framework, the data sources and rationale behind specific decisions should be understandable to clinicians and patients, as a critical component for informed consent and patient autonomy.

In practice, this means that a company developing an AI system to be used in the health sector is expected to notify patients about the use of their data and to have documentation available, either on request or all the time, to explain how the algorithms work. However, as explained above, this becomes particularly difficult when the training of an AI system serves as a secondary purpose of processing, distinct from the original intent, and the company only has access to pseudonymised data, with no direct identifiers of the patients whose data were used for the training of the AI system. Furthermore, there may be an obligation to obtain the individual's consent again if consent was the legal basis for the initial processing of personal data. In some European countries, such as Italy, consent is the sole legal basis permitted for processing health data.

To overcome this issue, different scenarios need to be considered:

For source data (model training), AI developers can shift the obligation to inform individuals about the reuse of their data to the data provider. This approach addresses the practical difficulty of re-informing patients whose data are pseudonymised about those who have the information. Similarly, when consent is required, it is the responsibility of the data provider to collect it, especially when general prior consents are permitted. Additionally, AI solution developers can also create, notably through a dedicated page on their website, an information page to inform about the data processing conducted as part of the AI solution. This possibility is particularly encouraged in France.

For data used in the commercialisation of the solution, two hypotheses need to be considered:

When the AI does not have a direct interface with the individual concerned, the contract with the solution user (License Agreement) must stipulate that the latter is responsible for informing individuals about their data being processed by the AI solution. The same applies when consent is required. When a direct interface exists (HealthApp, Medical Devices), it must provide the required information and collect consent when required.

Finally, it should be noted that research actors are seeking to promote the research exception to the authorities to reduce obligations regarding information and individual consent. The current trend shows a reluctance of the authorities, as evidenced by the Resolution of the 97th Conference of Independent Federal and State Data Protection Supervisory Authorities (German Authorities) on the interpretation of the term “certain areas of scientific research” dated April 3, 2019, which tends to reinforce the obligation to specify prior consent for the reuse of data for scientific research.

3.Data Accuracy

The examples of bias by AI systems presented in section II. of the current article, highlight the importance of ensuring diversity in training datasets and implementing rigorous testing across demographic groups to mitigate algorithmic bias in healthcare AI—an issue which raises the question of the quality of data. Bias in the output of AI systems often arises from unforeseen correlations or causations (proxy attributes) within the data, as well as from unrepresentative training datasets.

Ensuring high quality of data is also fundamental to upholding the principle of accuracy of personal data, which is assessed according to the purpose and nature of the data processing. In the healthcare sector, accuracy of data is critical, as inaccurate or incomplete data can have direct consequences on patient outcomes.

Consequently, in line with the bias considerations discussed, it is crucial to focus on data quality rather than merely its quantity—a stance endorsed by international organisations such as the World Health Organisation and the FRA.

IV. Regulatory responses to AI in health

In addition to the existing regulations that broadly govern data protection across all industries, the growing need to address the unique risks associated with the development and use of AI has triggered a wave of regulatory responses. States and countries are now actively developing legislation to regulate AI use across various fields, including the health sector.

Although some regulatory frameworks imposing specific obligations on key stakeholders in the development and deployment of AI systems (a hard law approach) have already been adopted or proposed by governmental working groups, most jurisdictions currently rely on a soft law approach, which is characterised by adherence to guiding principles and non-binding recommendations.

Regardless of the approach taken, the extensive list of legislation and guidelines adopted for the regulation of AI underscores the intensifying international efforts to regulate AI, highlighting the need for a thorough and ad hoc examination of the scope and applicability of various regulatory frameworks and the specific obligations they impose. Simultaneously, it is evident that international regulatory efforts increasingly prioritise cross-sectorial rather than sector-specific approaches, rendering the aforementioned thorough examination within the healthcare sector even more imperative.

Although many of these compliance requirements overlap, a detailed overview of the structure and main provisions of the EU AI Act is valuable, given its significant influence on regulatory approaches in other jurisdictions—a phenomenon often referred to as the 'Brussels effect.’

EU: What does the EU AI Act regulate, and what obligations does it establish?

With the adoption of the AI Act, the EU establishes the world’s first comprehensive legal framework for Artificial Intelligence. Similar to the GDPR, the EU AI Act has an extraterritorial scope, applying not only to entities established within the EU but also to those that place or utilise AI systems within the EU market. The entry into force of the AI Act in August 2024 designates the starting point of the 24-months countdown for its full effect and enforcement. More specifically, the different sets of obligations around the lawful development and deployment of AI systems, are strongly interrelated with the classification of AI systems into four distinct categories, depending on the potential risks posed by them to the health, safety and fundamental rights of individuals.

→ Prohibited AI practices: the use of specific AI practices which are considered particularly harmful and abusive is prohibited. By referring to “practices”, it is clear that attention is given to the potential use rather than the type of algorithm of the AI system, covering both the objective and the effect of an AI system in materially distorting human behaviour and causing significant harms to individuals, by having sufficiently important adverse impacts on their physical and psychological health.

In the health sector, prohibited AI practices hold significant implications, particularly in research and, more specifically, in clinical trials. Certain AI practices within medical research warrant classification as prohibited due to their potential risks and ethical concerns, such as:

• AI systems that manipulates patients’ decision-making in health, by using personal data, such as psychological profiles, behavioural tendencies, or medical histories to manipulate patients into adhering to a treatment regimen by exploiting their cognitive biases.
• Health apps or AI systems used in clinical trials that deploy deceptive interfaces or dark patterns to trick users into sharing more personal data than necessary or into enrolling in trials without fully understanding the implications.

→ High-risk AI systems: the EU AI Act defines high-risk AI systems as those AI systems used as products or safety components of products falling under specific EU legislation as listed the EU AI Act (Annex I) and at the same time requiring the performance of a third-party conformity assessment. Apart from those, the development of AI systems for their use in specific areas listed in the EU AI Act (Annex III) would also qualify them as high-risk AI systems (stand-alone).

• In this framework, health AI systems considered as medical devices of Class IIa – Class III within the meaning of the Regulation EU 2017/745 or in as in vitro diagnostic medical devices of Class B-D according to the Regulation EU 2017/746 would be first of all considered as being of high-risk. For example, such case would be an AI system that assists or autonomously performs certain surgical procedures, such as precision-based surgeries.
• Concerning stand-alone AI systems, AI systems influencing critical medical decisions based on which access to healthcare services is denied or granted should also be characterised as high-risk AI systems. Multiple use-cases of AI systems would meet this requirement: AI systems analysing patient data, including genetic information, to suggest personalised treatment protocols for diseases; AI systems assisting in the recruitment of patients for clinical trials; AI systems for predicting drug efficacy or used for dose optimisation. All these examples present AI systems which influence the medical decision-making process concerning the eligibility of natural persons for healthcare services of different types, and which therefore potentially meet the criteria for characterisation of stand-alone high-risk AI systems.
• On the contrary, AI systems and models specifically developed and put into service for the sole purpose of scientific research and development are excluded from the scope of the EU AI Act (”AI systems and models, including their output, specifically developed and put into service for the sole purpose of scientific research and development”). However, a strict interpretation of this exemption should be promoted, given that any AI system surpassing the exclusive scientific research purpose by initially or subsequently serving commercialisation objectives would not fall under this exception.

In any case, under certain circumstances, the provider may prove that a specific AI system initially meeting the requirements of a stand-alone high-risk AI system should not be ultimately considered as such, for the reason of not materially influencing the outcome of decision making. Such an assessment should be documented before placing that system on the market or put into service.

The date when the provisions regarding obligations for the development and deployment of high-risk AI systems come into effect varies based on the type of system. For stand-alone high-risk AI systems in the health sector, the obligations for relevant operators will become enforceable in August 2026. This allows sufficient time for assessing and organising the necessary compliance measures. Meanwhile, for high-risk AI systems governed by the medical device regulations, considering that the relevant products are already regulated by European legislation, a longer timeframe is provided, with the relevant obligations coming into force in August 2027.

The actor bearing the most burdensome obligations is the provider of the high-risk AI system, as the entity which, whether by developing an AI system, or having the AI system developed for them, places the AI systems in the market or puts it into service under their own name or trademark. The provider is mainly required to:

• Design and develop AI systems in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and to be able to use it appropriately.
• Incorporate human-machine interface tools to ensure effective human oversight over the high-risk AI system.
• Achieve an appropriate level of accuracy, robustness and cybersecurity in the development and design of high-risk AI systems.
• Establish, implement, document and maintain a risk management system throughout the whole lifecycle of the high-risk AI system, to appropriately identify and tackle the risks posed by the system.
• Put in place data governance and data management practices for the training, validation and testing used for training the AI model.
• Draw-up technical documentation mainly describing compliance with the above obligations, the main elements of the AI system and of the process for its development, as well as the methods for monitoring and controlling the AI system.
• Support compliance with the aforementioned obligations by keeping the automatically generated logs of the AI system.
• Appoint an authorised representative within the Union for providers established in third countries and not established in the EEA / EU.
• Register the high-risk AI system in the relevant EU database established for this purpose, prior to placing the AI system in the market or putting it into service.
• Affix the CE marking in a visible, legible, indelible or digitally accessible manner for digital systems.
• Ensure that the high-risk AI system undergoes the relevant conformity assessment procedure. For stand-alone high-risk AI systems, the conformity assessment procedure is based on internal controls put in place by the provider of the system, without the involvement of a notified body (type of self-assessment). For AI systems incorporated in products listed under Annex I, such as medical devices, the provider should follow the relevant conformity assessment required by the relevant Union harmonization legislation, such as the medical device regulations.
• Draft a written machine-readable, physical or electronically signed EU declaration of conformity of the AI system with the above requirements.

Apart from the obligations of the provider described above, the deployer of high-risk AI systems—defined as the entity operating an AI system under its authority—must also adhere to several obligations to ensure safe and compliant use of the system. These include implementing technical and organisational measures to operate the AI system in line with its provided instructions. Additionally, human oversight must be assigned to qualified individuals who possess the necessary competence to monitor and manage the system effectively. Finally, the deployer must exercise control over input data, ensuring that the data used is both relevant and sufficiently representative to maintain the system's reliability and accuracy.

→ General-purpose AI models (’GPAI model’): These are the AI models that display significant generality and are capable of competently performing a wide range of distinct tasks, in a way that they could be further integrated into a variety of downstream systems or applications. GPAI that are used exclusively for research, activities before they are placed on the market, are excluded from the scope of the EU AI Act. These models could serve a variety of different purposes in the health area. For instance, an AI model initially designed for image recognition or pattern detection could be adapted to detect patterns in medical imaging, such as identifying tumors in MRI scans or abnormalities in X-rays. In the healthcare sector, this GPAI model can be repurposed to assist radiologists in interpreting medical images more efficiently. Similarly, it could be used in clinical trials to assess disease progression or treatment efficacy by analyzing imaging data. Moreover, Natural Language Processing systems, such as those used for question-answering, text summarization, or sentiment analysis can be adapted to process and analyze electronic health records (EHRs), extracting valuable information such as patient histories, symptoms, or drug interactions. There could be also predictive analytics systems that are repurposed to predict patient outcomes, treatment efficacy, or even patient dropout rates during trials. Such systems could analyze past trial data to forecast success rates and optimize the design of ongoing trials.

The obligations of providers of general-purpose AI models also focus on ensuring transparency and on facilitating the safe and responsible use of AI models for the providers who may subsequently integrate the GPAI model into their AI system. More specifically, providers of general-purpose AI models are mainly expected to:

• Create and maintain up-to-date technical documentation detailing the model's training and testing processes, along with evaluation results. This documentation must be provided upon request to the AI Office and national competent authorities.
• Prepare, update, and supply necessary information and documentation to AI system providers who intend to integrate the general-purpose AI model into their systems. This information should enable downstream providers to understand the model's capabilities and limitations and ensure compliance with the AI Act.
• Establish a policy to ensure adherence to EU copyright laws, particularly focusing on identifying and respecting reservations of rights as expressed under Article 4(3) of Directive (EU) 2019/790.
• Compile and publicly disclose a sufficiently detailed summary of the content used to train the AI model, following a template provided by the AI Office.

It is important to note that the obligations related to technical documentation and information for downstream providers do not apply to AI models released under a free and open-source license, provided that the model's parameters, architecture, and usage information are publicly accessible. However, this exemption does not extend to general-purpose AI models that pose systemic risks.

The latter are classified as such if they possess high impact capabilities evaluated using appropriate technical tools, methodologies, indicators, and benchmarks, or if they are designated as having a systemic risk by a decision of the European Commission, if the model's capabilities or impact are equivalent to those with high impact capabilities, considering the criteria set out in Annex XIII of the EU AI Act. Annex XIII specifies criteria such as the number of parameters, quality or size of the dataset, computational resources used for training, input and output modalities, and the number of registered end-users.

Providers of general-purpose AI models classified as having systemic risk have additional obligations, apart from those described above. In particular, they should also:

• Conduct evaluations using standardised protocols and tools reflecting the state of the art, including adversarial testing, to identify and mitigate systemic risks.
• Assess and mitigate potential systemic risks at the Union level that may arise from the development, market placement, or use of these models.
• Monitor, document, and promptly report serious incidents and possible corrective measures to the AI Office and, as appropriate, to national competent authorities.
• Ensure an adequate level of cybersecurity protection for the AI model and its physical infrastructure.

→ Basic AI systems: Finally, AI systems, regardless of their risks, that are intended to interact directly with natural persons, will be considered as basic AI systems and should be developed and deployed in such a way that natural persons concerned are informed that they are interacting with an AI system or that a specific content has been artificially generated.

V. Conclusion

The rapid advancement of artificial intelligence in the health sector over the last years has profoundly transformed the industry, revolutionising diagnostics, personalised treatment, and patient monitoring. While these innovations hold immense promise for improving clinical outcomes and research efficiency, they also introduce complex ethical, regulatory, and data protection challenges that must be addressed.

Although global regulatory efforts are increasingly focused on balancing the acceleration of AI-driven advancements with stringent compliance measures, significant hurdles remain. Many regulatory frameworks adopt cross-sectoral approaches rather than specific guidelines for the health sector, complicating their application in a medical context. Additionally, challenges related to varying regulatory standards across jurisdictions persist, demanding a comprehensive and adaptable compliance strategy for effective AI integration in health.

Despite these complexities, the emerging regulatory landscape consistently emphasises the importance of transparency, accountable governance, and ethical AI model design. As AI continues to shape healthcare, a thorough and nuanced understanding of these frameworks will be crucial to ensure that AI technologies are developed and deployed responsibly and compliantly, ultimately benefiting both patients and the broader healthcare ecosystem.

Part 2 of this article series will delve into the regulatory requirements for AI in healthcare and research under U.S. law. It will further offer detailed tables providing a comprehensive global overview of existing AI regulations and guidelines that may impact the healthcare and research sectors worldwide.

‍