Evidence-based decision-making: technology as a solution?

It was with the collaboration of 20 professionals from the Health Region of Central Catalonia: for the first time, we were able to approximate the impact of using eConsulta on face-to-face visits. We did it in the least technological way possible – we asked them. “In relation to this case you dealt through eConsulta: do you think the interaction avoided a face-to-face visit?” Thus, 2,268 times. In 88% of cases, they answered yes. We took advantage of the fact that we “annoyed” them by also asking what kind of consultation it was and if, in the absence of virtual communication, the query was done (to understand what proportion of “new cases” we were treated: 30%).

We were very pleased with the way the experiment went, but we knew we couldn’t repeat it (it’s really grateful that the professionals reviewed, one by one, hundreds of cases!). That is why we decided to use Artificial Intelligence: we applied a classifier algorithm (actually, twenty different types) based on (Supervised) Machine Learning so that, having learned from the “case by case” classification of the faculties, able to do the same job.

We applied five different techniques of vector text representation (something like turning words into zeros) and four classifying algorithms (twenty different combinations), and tested how they did. The result: It seems that the algorithm has learned very well to distinguish, based on the text of the message, whether it will prevent a face-to-face visit or not. But it still can’t quite tell “what’s going” on the message (query type). This is, we believe, because the sample we analyzed is very small, and most of the cases we analyze are of type 1 “Test management”. If they are all of the same type, it is harder for them to learn how to distinguish them: we would like a more “balanced” sample.

We did not expect spectacular results (with such a small sample, in relation to the possibilities of the tool!) And we are not quite sure of what we have obtained (you must go with leaded feet!): With this second experiment, what we wanted to try is that Artificial Intelligence is a feasible (at a technical level) tool to understand how a service that already has 300,000 conversations is working. Does eConsult help to solve a case and prevent the citizen from moving? What kind of queries are made? Are these queries already done before, or since it is easier to communicate with the professional, do we ask them more?

Fortunately, little by little, technology will help us to answer these questions rigorously so that we can make evidence-based decisions.