Moscow ranked its AI health services

It was published as part of an experiment on the use of computer vision based on artificial intelligence for the analysis of medical images and further use in the Moscow healthcare system.

The publication of the ranking in the public domain will help to stimulate competition between AI developers to develop promising technologies and objectively evaluate existing solutions before implementation in the healthcare system of other regions, for example. Also, patients themselves will be able to use the rating data outside of the experiment: many developers allow users to upload their research to get a second opinion from the AI service, and an independent comparison will help them to adequately consider the results obtained. The list of the best innovative technology solutions will be updated monthly.

The rating was initiated by the Moscow Centre for Diagnostics and Telemedicine, which organized the experiment. The results of AI work in the field of medicine in the world are commonly evaluated according to three parameters: sensitivity, specificity and diagnostic accuracy of the service.

Based on lessons learned during the experiment, Moscow specialists supplemented the existing parameters. The rating includes assessment not only of the service, but also of its adaptability to the unified city information system and the ability to work with large research flows. In the future, such a rating scale may be useful for the global scientific community to scale the obtained results and implement evolving AI algorithms.

The new methodology for evaluating artificial intelligence includes comparing service results with radiologists' protocols, the doctor's agreement with the localization of pathological findings, the proportion of examinations without defects and the speed of data processing. Some of the parameters are evaluated automatically and others are manually verified by a group of doctors - experts from the Diagnostic and Telemedicine Centre. For each criterion, the service receives a certain number of points, which are added up to a final score.

Services for automatic analysis of radiology research have been tested in the city's healthcare system since spring 2020. Artificial intelligence-based solutions are connected to the system that connects all radiology departments of Moscow polyclinics and hospitals - the unified radiology information service of the Unified Medical Information and Analytical System.

As part of the experiment, a funnel of intelligent algorithms was organized - the rules of selection, testing, validating and commissioning artificial intelligence-based services in order to apply the highest quality solutions in the clinical practice.

Currently, 19 services for the diagnosis of oncological diseases (lung and breast cancer), COVID-19, osteoporosis and coronary heart disease are connected to the main circuit of the system, which is used by all radiologists of urban health facilities.

A research study using promising innovative technologies has become possible due to a unified digital health platform developed by the city's social development complex and the Moscow Department of Information Technology.

Samuel Daniel

Update 18 October 2021