RP38 - An Audiovisual Agent for Orchestration of Advanced AI Models and Seamless Human-Machine Interaction in Clinical Settings

Current medical practice is overwhelmed by excessive data, fragmented IT systems, and limited time, hindering efficient clinical decision-making and coordinated patient care. AI-driven virtual assistants could mitigate this problem by seamlessly integrating, filtering and interpreting data from various sources, streamlining information access, and providing real-time, intelligent support to optimise clinical decision-making and patient care [1]. This project aims to develop a virtual assistant for healthcare professionals that enhances human-machine interaction through a multimodal interface equipped with a microphone, loudspeakers, and a camera, enabling audiovisual communication. The assistant will serve as a central orchestrator, capable of integrating and managing a variety of AI models, including those developed in related projects such as RP17, RP21 and RP15. It will also incorporate 3rd party models and services, e.g., providing access to electronic textbook information and guidelines. The assistant’s design emphasises modularity and extensibility, allowing for the continuous addition of new models, thereby enhancing its capacity to support a wide range of clinical tasks over time. The assistant will focus on seamless interaction capabilities, incorporating natural speech and responses and computer vision to recognise gestures or visual cues and analyse documents and photos using specialised submodels. This project will work closely with RP35 to assess how the design and anthropomorphised elements of the assistant influence clinician engagement and workflow efficiency, providing valuable insights into the benefits and potential challenges of such systems in clinical environments. Through usability studies and workflow analysis, the project will evaluate the assistant’s performance and benchmark it against existing tools, identifying areas for improvement and optimisation. Expected outcomes include a fully functional prototype of the multimodal virtual assistant, comprehensive integration of multiple AI models, and detailed assessment reports on its impact on clinical performance.

[1] E. J. Topol, “High-performance medicine: the convergence of human and artificial intelligence,” Nat. Med., vol. 25, no. 1, pp. 44–56, Jan. 2019, doi: 10.1038/s41591-018-0300-7.