Applying A/B testing to clinical decision support systems (CDSS)
As clinical decision support systems (CDSS) become common in healthcare, IT teams face new challenges. They must deliver CDSSs that are user-friendly, accurate, and optimizable—goals that A/B testing can help achieve.
Here’s an overview of how A/B testing can be applied to the CDSS lifecycle and other healthcare application testing efforts.
What is a clinical decision support system (CDSS)?
A clinical decision support system (CDSS) is an IT solution that assists providers with decision-making across the healthcare journey. Using patient data and clinical knowledge, these systems work to improve care outcomes. As governments and healthcare networks continue to push for electronic health record management, we’re seeing CDSSs become more mainstream.
So, where does A/B testing fit in the CDSS development and lifecycle?
How does A/B testing improve a CDSS?
A CDSS is a complex IT build, involving various interfaces, algorithms, and interactions. A/B testing helps evaluate different approaches to these components, leading to better design, clinical, and user experience decisions.
A CDSS using good usability practices sees higher adoption. A better UX also offers better integration into clinical workflows, supporting better patient outcomes.
With A/B testing, you can improve the UX of a CDSS by:
Optimizing CDSS user flows
Users often have “mental models” of how they expect a solution to work. In the context of a CDSS, a mental model refers to the way physicians understand and anticipate the system's behavior.
For a CDSS to be user-friendly, they must align with physicians' mental models of care flows. These conditions make the system easier for them to use and integrate with their workflow.
In their paper on human-centered CDSS design, Fraunhofer, the team behind the research project MED2ICIN (a proof-of-concept of a human-centered CDSS), suggests co-creating your CDSS with real stakeholders, like the physicians and healthcare professionals who will use it, to ensure the “flows” are right.
The team used “co-design sketching” with actual users (gastroenterologists) to create design solutions. They had target physicians outline the desired CDSS flow, which the design team then translated into the actual system.
Here is the physicians’ mental model for the first step:
Here is the team's proposed implementation:
Adding A/B testing to your CDSS DevOps allows you to test high-quality prototypes to identify the most user-friendly versions. The MED2ICIN team also emphasizes the importance of experimentation methods, like feature experimentation, in achieving optimized co-designed outcomes.
Improving visual communication
CDSS dashboards for web, mobile, and other devices need to communicate relevant or high-risk information within five seconds or less. To design these ‘at-a-glance’ user experiences, you need to be able to test different configurations of text, images, and user flows.
A/B testing helps design teams find the best placement for visual elements to optimize information flow. And, by testing specific audience segments, you gain these insights without a full UX launch.
Optimizing critical communications
A CDSS generates many alerts, most of which are irrelevant - leading physicians to ignore about 95% of them. This behavior increases the risk of caregivers missing critical messages.
A/B testing can help identify optimal alert settings to minimize the risk of missing critical CDSS alerts. There are several solutions to reducing CDSS alert fatigue and they can be tested with A/B testing:
- Route low-risk alerts to administrative staff.
- Route some alerts directly to patients, for a more collaborative approach.
- Test changes to CDSS programming.
To combat alert fatigue, one CDSS team conducted A/B tests with different variations of their alert system. Their first test had little impact, but a follow-up test, informed by initial insights, reduced daily alerts from 23.1 to 7.3.
Another paper highlights how CDSS teams run tests on alert fatigue post-implementation:
“Monitoring CDS alerts post implementation and robust testing allow the identification of malfunctions and optimization opportunities, and these processes are considered to be essential in maintaining reliable and effective CDS alerts.”
Boosting accuracy with better algorithms
At the core of CDSS systems are medical algorithms. A/B testing lets you evaluate different algorithms to find the best performers:
- You can use A/B testing to compare different diagnostic algorithms for better accuracy. For instance, test various decision tree algorithms to identify the most accurate one.
- You can test multiple predictive algorithms to determine which provide the most accurate estimations of disease likelihood.
- You can experiment with various machine learning methods to aid early diagnosis and support along the care continuum. For example, a CDSS could test Naive Bayes against Perceptron to predict postpartum depression likelihood.
A/B testing not only helps you find the best algorithms for your CDSS but also lets you improve them by testing updated versions against your defaults.
Enabling collaborative decision-making
With a growing focus on patient-centered care and patients taking active roles in their healthcare, shared decision-making is another aspect a CDSS should handle.
In shared decision-making, a CDSS must handle Personal Health Records (PHRs) in addition to EHRs. Patients often manage PHR data, which can include wearable data, coverage details, and symptom tracking. Effective patient-physician communication is another key feature a CDSS must manage.
A/B testing can optimize your patient CDSS experience, from UI to messaging. Here’s a prototype a CDSS team proposed for its self-management patient module:
What to look for in an A/B testing solution for optimizing a clinical decision support system
Building an effective CDSS and other healthcare application testing is an organization-wide effort, not just an IT project. Dialing in these systems requires collaboration from UX, IT/engineering, clinical teams, and legal and regulatory stakeholders.
When choosing an A/B testing platform for optimizing a CDSS, look for one that supports collaboration across all these teams. You need an "all-team" experimentation solution with full-stack capabilities, as A/B testing for a CDSS goes beyond the interface layer. HIPAA compliance is also essential.
Kameleoon is a HIPAA-compliant, full-stack A/B testing solution that works organization-wide, helping healthcare organizations build systems to improve their quality of care.
