The Guide for evaluating digital health technologies that incorporate artificial intelligence has been published

The Agency for Healthcare Quality and Evaluation of Catalonia (AQuAS) has published the Guide to evaluating digital health technologies that incorporate artificial intelligence (IA), developed by professionals at the same entity, the Health/IA Program and the TIC Salut Social Foundation. The document is aimed at developer companies or professionals of AI solutions in health and aims to help them generate evidence to facilitate the evaluation and adoption process of its technology.

The guide presents the domains, dimensions and sub-dimensions that need to be evaluated, based on the AQuAS methodological framework for evaluating digital health technologies, and the evidence standards necessary to demonstrate the value of the technology in the Health System.

The document is part of the evaluation action strategy of the Health/AI Programme, and as mentioned therein, it will be reviewed once the new specific methodological framework for the evaluation of technologies that incorporate AI, commissioned by the Ministry of Health to AQuAS, is published.

13 domains to evaluate

The following domains are specified for the evaluation:

1) Description of the health problem and target population: includes the definition and description of the health problem (prevalence and incidence, pathophysiology, etc., in a contextualised manner) and the target population in which the intervention is applied or is expected to be applied (average age, risk factors, needs, etc.), as well as information related to the standard therapeutic approach (diagnosis, treatment, etc.) and the specific context to which it is applied.

2) Description of the technology: It considers the main characteristics that define it, the needs it seeks to address, the regulation or licences required, the technology requirements (e.g. basic material and human resources, space, training, etc.) or the potential added value with respect to existing alternatives.

3) Technical aspects: It includes the evaluation of the technological elements that configure and characterise the technology.

4) Content: provides for the evaluation of the suitability, completeness, accuracy and personalisation of the content, both in terms of written, visual and auditory information and in terms of the therapeutic intervention in question for the person using the technology. Evaluation of the scientific evidence on which the content is based or that supports it.

5) Safety: addresses the identification and evaluation of risks, harms and unwanted side effects derived from the use of the intervention that may be inherent to the technology or external factors (e.g. inappropriate user or incorrect use).

6) Clinical efficacy and effectiveness: includes the evaluation of the degree to which the technology contributes to improving the healthcare impact, health status and quality of life of users in relation to the use of the technology and the expected benefit.

7) Economic aspects: includes the evaluation of the economic costs of acquisition, maintenance and use from a payer and/or social perspective, and the cost-effectiveness ratio in comparison with existing alternatives.

8) Human and sociocultural aspects: includes the evaluation of human and sociocultural aspects that may have an impact on the use of technology (e.g. acceptability, ease of use, digital health literacy, commitment to the intervention, perceived benefit, patient empowerment, etc.) and evaluation of the sociocultural impact that technology may have (e.g. accessibility to the service or healthcare, changes in work flows and roles, changes in the doctor-patient relationship, etc.).

9) Ethical aspects of using the device: encompasses the evaluation of ethical concerns related to the use of the device or technology from the point of view of all stakeholders and the context in which it is implemented or intended to be used.

10) Legal and regulatory aspects: encompasses the assessment of the degree to which the technology complies with applicable regulations and standards at the national/regional level (as applicable).

11) Organisational aspects: includes the assessment of the impact of technology at an organisational level (e.g. changes in workflows or roles of professionals). The infrastructure and resources necessary for the implementation of the technology (human resources, training, etc.) also need to be considered.

12) Environmental aspects: evaluates the direct and indirect environmental impact associated with the development and implementation of the technology. Environmental impact can be measured, for example, as estimated carbon emissions, raw material use, energy consumption, or as environmental benefits obtained.

13) Description of post-implementation monitoring: includes the description of the mechanisms established for the post-commercialisation evaluation of the performance of the technology by the developers or those responsible for its management.

For each of the domains to be assessed, the guide details its dimensions and sub-dimensions, if any, and the necessary evidence standards. Finally, the document includes an annex with additional information on the current legislation applicable to the use of AI in healthcare.

For more information, you may refer to the 2023 Report at this link.