Evaluation of AI-enabled transcription for hospital use through simulation testing

Opportunity

Regional health services stand to improve productivity and reduce costs through AI-enabled documentation tools. However, the "capability-reliability gap" — the difference between what AI tools can do in controlled conditions and how they perform in complex real-world environments — makes pre-implementation testing critical to successful adoption.

Intervention

The Centre’s research team conducted simulation studies to verify the workflow, safety, accuracy, and acceptability of an AI-enabled transcription tool with healthcare providers at a regional health service, prior to real-world implementation.

Impact of simulation

Pre-implementation simulation provided key evidence around the workflow and performance of the AI-enabled transcription tool.

The process produced 11 changes to the tool itself and two key workflow changes, substantially increasing the likelihood of successful adoption.

The study also highlighted that public hospitals are complex, dynamic environments requiring dedicated pre-implementation support.