Ask Aletta now reads images, flowcharts and tables in clinical guidelines

We're pretty excited about this one.

Clinical guidelines are not made up of text alone. Some of their most useful information is contained in images, flowcharts and tables: a decision tree showing the next treatment step, a dosing table organised by age or weight, or a diagram comparing two clinical approaches.

That information can be essential to answering a question correctly. Yet it is also the kind of information AI systems can easily miss.

Ask Aletta can now retrieve and use the information contained in these visual elements - and show the original table, diagram or flowchart directly in its answer.

Why visual information is difficult for AI

When a PDF is processed only as a layer of text, visual elements can become effectively invisible. The paragraphs surrounding a flowchart may be available, while the decisions, arrows and relationships inside the flowchart are not.

Simply extracting the words from an image does not necessarily solve this problem. A table is more than a collection of separate numbers and labels: their position in rows and columns determines what they mean. In a flowchart, the arrows and branches are often just as important as the text inside the boxes.

If that structure is lost, an AI system may overlook the part of the guideline that contains the actual answer.

Processing the complete guideline

When we add a new guideline to Ask Aletta's knowledge base, we now process both its written and visual content.

Images, tables and flowcharts are converted into a representation that Ask Aletta can retrieve alongside the surrounding text. The original visual element remains connected to its source and page, allowing Ask Aletta to use it when answering and display it directly to the healthcare professional.

This has two advantages. First, information contained only in a visual element is no longer automatically excluded from the answer. Second, the user can inspect the original table or flowchart and verify how it relates to the response.

From a question to the relevant flowchart

Consider this question:

Which steps should I follow when treating functional constipation in a child under 1 year old?

The relevant Dutch paediatric guideline contains a flowchart showing the treatment pathway. It covers alarm symptoms, initial advice, evaluation moments, first-choice medication and the circumstances in which further investigation or referral should be considered.

Instead of relying only on the surrounding paragraphs, Ask Aletta can use the information in the flowchart and show it in the answer.

The same applies to diagrams that compare different approaches. When asked about the top-down and bottom-up variants of a low-FODMAP diet, Ask Aletta explains the distinction and displays the schematic overview from the underlying guideline.

Showing the evidence, not hiding it

Understanding visual information is only half of the feature. We also want healthcare professionals to see the evidence for themselves.

A written summary can help someone find an answer quickly, but it should not replace the underlying guideline. That is why Ask Aletta displays the original visual element together with its source and page reference. The healthcare professional can review the image in context and decide how it applies to the situation in front of them.

This builds on a principle that has guided Ask Aletta from the beginning: every answer should be traceable to its source. That now includes the parts of a guideline that cannot be captured in paragraphs alone.

Want to try it yourself? Ask:

Which steps should I follow when treating functional constipation in a child under 1 year old?

Or:

What are the top-down and bottom-up approaches to a low-FODMAP diet?