Relevance of this topic to Digital Health
An increasing number of digital health tools already incorporate, and will in the future incorporate, AI components. These are sometimes targeted at patients using a digital health tool such as an app, at home, to better manage a health condition. The app may provide fine-tuning guidance on lifestyle and treatment, including possible dose changes, that can bring tremendous benefits to the stability of a health condition and the quality of life of the patient in between clinic visits.
AI powered digital health tools might also be used by healthcare professionals when evaluating the status of their patient including the reported health outcomes, and planning the trajectory of care. Sometimes AI components will be used as escalation tools, to help detect patterns of deterioration or disease flare up, enabling the patient and clinician to quickly reconnect and modify the treatment to intercept the issue as early as possible.
In Europe AI solutions are being increasingly regulated, and there are expectations within health systems of greater transparency in how the AI has been developed and tested, on which patients it might apply accurately, and on how ethics and governance principles have been followed.
This policy digest highlights some of the most relevant and useful instruments that developers of AI should be mindful of and make use of when developing solutions.
Current focus of policy, legislation, standards, emerging practices in this landscape
Three instruments are summarised here. The first, of greatest impact, is the new EU AI Act, which will come into force in a couple of years and for which it is wise for developers to prepare now. It introduces a risk stratification and stipulates measures that a developer mistake according to their risk level.
The ALTAI checklist is a formalisation of the European AI Ethical Principles that were published and highly regarded a few years ago. It offers a useful framework that a developer could utilise in order to establish quality processes and quality documentation practices, when developing AI. It is likely that any future conformity assessment of AI under the Act, will align with this checklist (although CE marking may go further).
The DECIDE-AI reporting guideline is not a statutory instrument, but the result of a European group of experts. It goes beyond ethical and governance matters into scientific validity of the development methodology and the valuation methodology, and therefore presents recommended good practice for high quality AI development and its documentation to potential adopters.
Implications for digital health uptake
Developers
- These instruments collectively present a pathway for developers of AI to construct a process and documentation practices that will place them in a good position to be able to conform in future to any CE marking introduced for AI, and be able to present a convincing case for the quality and trustworthiness of their products to potential adopters
Enablers
- n/a
Payers and procurers
- The same framework for developers could equally be used by decision-makers when considering the adoption of an AI powered solution, or comparing the strengths and weaknesses of multiple product offerings.
- It provides a structure and frames potential expectations that should allow more confident about whether to trust a particular solution or to determine which one is best suited to the intended needs.
Users
- Although most end users will not examine detailed assessment criteria and documentation, the knowledge that good practices have been adopted and that safety measures have been applied, and are applied on a continuous basis during use, should improve confidence in using Although most end users will not examine detailed assessment criteria and documentation, the knowledge that good practices have been adopted and that safety measures have been applied, and are applied on a continuous basis during use, should improve confidence in using the product, and relying on the product.
Remaining gaps and issues
- Regulators and HTA agencies need better formal ways of assessing AI based innovations, given their traditional waterfall method of assessment versus the evolving nature of machine learning algorithms.
- Procurement officers need formal guidance and quality/safety criteria from their health systems for judging what is a good AI solution to purchase for their organisation: what are the reliable quality and safety declarations they should seek, how to verify the suitability of the AI to their intended patient population, and what kind of contracts and service level agreements are appropriate.
- Health care professionals and patients need investment in education in knowing when and how far to trust AI recommendations and actions, how to supervise the operation of AI technologies.
Legislative, regulatory, policy or standardisation instrument, or good practice
European AI Act
Summary of the instrument
This Act will come into force by 2026 at the latest (the exact enforcement date has not been set). It requires developers of AI to undertake a risk assessment according to a stratification defined in the Act. Its amin obligations for high risk AI are:
- have risk assessment and mitigation systems
- ensure high quality of the datasets feeding the system to minimise risks and discriminatory outcomes
- conduct logging of activity to ensure traceability of results
- maintain detailed documentation providing all information necessary on the system and its purpose for authorities to assess its compliance
- have appropriate human oversight measures to minimise risk
- ensure a high level of robustness, security and accuracy
- provide clear and adequate information to the user
There are also obligations in the Act for users of AI:
- operate AI system in accordance with instructions of use
- ensure human oversight when using of AI system
- monitor operation for possible risks inform the provider or distributor about any serious incident or any malfunctioning
Implication for digital health stakeholders
- Know how to undertake an AI risk assessment and know the level of your intended solution
- If it is expected to need MDR Certification it is also likely to be high risk, because it influences human behaviour (HCP, patient – e.g. by making recommendations) or has some degree of autonomy over patient health (e.g. by directly changing the administration of a medication or by triggering an action by others)
- Conformity assessment schemes are being developed and CE marking will be required
- You must have a robust and documented process to ensure the AI performs as intended, and maintain an evidence trail
- You must systematically collect data throughout the AI lifetime of use on its reliability, performance and safety, and have measures in place to respond to issues
- A statutory reporting system for safety issues will be put in place
1. Ethics Guidelines for Trustworthy AI
2. Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment
Instrument status:
European Commission guidelines
Publisher or source:
European Commission
Summary of the instrument
The ALTAI checklist is a structured compliance-oriented representation of the Ethics Guidelines, and therefore an easier form in which to verify ethical compliance. The following is a non-exhaustive list of its main topics:
- Human interaction model and decision taking involvement
- AI user awareness
- Potential for interference with human decision making
- Oversight
- Ability to “emergency stop” the AI from proceeding and taking actions
- Cybersecurity: integrity, robustness, security., resilience from attack
- Risk management
- Reliability and stability testing, fault tolerance, failsafe mechanisms
- Accuracy of the training data, monitoring of accuracy during use, handling of uncertainty, missing data etc.
- Continual learning e.g. protection measures against risks of learning from unsuitable patterns, traceability of the learning cycles and the data inputs used
- Information governance
- Bias (mostly related to the quality and characterisation of the training and validation data)
- Accessibility
Implication for digital health stakeholders
Although its publication preceded the AI Act, it may be used as a more detailed checklist for conformance to the Act. For each of the areas summarised as a bullet point above, there is a description of the expectations not formally framed as quality standards but employing them, that a developer should fulfil in order to ensure that the development is ethical and of trustworthy quality.
It is therefore a recommended framework for developers to adopt as part of the process of demonstrating conformance to the AI Act, until more detailed conformity assessment criteria are published.
Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI
Summary of the instrument
This open access publication consolidates the work of a European consortium of experts to define a checklist of AI development and validation criteria that should be reported in order to present the evidence that the AI has been developed in a scientifically robust as well as ethical way. This guideline therefore covers scientific aspects of the development methodology and the evaluation protocol and the reporting of results from the evaluation, in addition to some of the same ethical points that the ALTAI checklist covers. It also includes reporting on the assessments and approvals that have been obtained for it.
The main areas it covers are:
- Provenance of the algorithm, version, date etc.
- Training data sources and their data quality assessment
- Algorithm development methodology
- Supervision method
- Validation evidence
- Safety data
- Interoperability (which standards e.g. EHR)
- Effectiveness evidence
- Certification e.g. MDR
- Approvals e.g. HTA (in which countries)
Implication for digital health stakeholders
- This is a guide, offering good practice, and does not have any enforcement. However, it should be considered for voluntary adoption by AI developers as a structure to guide their development and evaluation process and for reporting this to potential adopting stakeholders.
- It is likely in the coming years that standard reporting templates will surface for use across Europe, as a means of documenting the quality and safety process by which an AI solution has been developed. It would be expected that any such template will incorporate much of the same areas of content.