RP36 - Multimodal Dashboard

Patient data is collected in centralised data stores to simplify analysis and guarantee access for all departments. This information is presented to medical professionals in configurable dashboards adaptable to current needs and personal preferences. Dashboards thus provide structure in a world driven by data from various sources. But screen real estate constrains the displayed information and requires prioritisation of data. Dashboards are thus often organised into sections, widgets, or tabs that visualise some and hide other data. Users are tasked with navigating partially hidden information, which increases cognitive load, distracts from the current task, and requires memorising the placement of (hidden) information. In this project, we aim to introduce additional modalities into medical dashboards to move beyond the limitations of a computer screen. This requires identifying a suitable software architecture to incorporate additional modalities and allow the extension of existing workflows. Building upon results from the first (RP09) and second (RP16) cohorts, as well as prior work [1], we aim to implement a more fine-grained distinction between the responsibilities of different processing components. Different modalities also introduce the desire for more individual configurations, but end-users often cannot create individual widgets and configurations for different modalities. However, end-users have the necessary knowledge to define workflows and structure dashboard parts; thus, suitable tools (e.g., LLM-powered description of widgets in natural language trained with existing widget specifications) to define individual data representations on the dashboard are necessary. However, displaying more data on dashboards introduces the risk of displaying data that distracts users and thus impedes efficient working. Mathematical models (e.g., as proposed by [1]) can help to identify potential distractors but require an extension for multimodal dashboards and dedicated studies with potential users.

[1] S. Meißner and A. Degbelo, “User Performance Modelling for Spatial Entities Comparison with Geodashboards: Using View Quality and Distractor as Concepts,” in Companion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, in EICS ’24 Companion. New York, NY, USA: Association for Computing Machinery, Jun. 2024, pp. 7–14. doi: 10.1145/3660515.3661325.