Multimodal clinical dashboard

Practice overview

Provider	Avg. Patient Talk Ratio	Avg. Patient Satisfaction	Avg. Interruptions

Multimodal clinical dashboard

Individual patient visits

Select a patient visit to analyze data

Analysis dashboard

Please select a visit from the list above to see its analysis. Results will appear here.

Simulating Multimodal Healthcare Data: Building an Interactive Svelte Dashboard for Clinical Insights

Summary

This project introduces an interactive Svelte dashboard designed to simulate and analyze rich, multimodal clinical encounters, addressing the critical scarcity of privacy-safe healthcare data for AI research and informatics education. Inspired by the "Observer" dataset, the tool utilizes a custom Python engine to generate 100 synthetic primary care visits, complete with simulated video timelines, patient-provider talk ratios, and structured clinical metadata. Through intuitive Chart.js visualizations, the two-tiered application provides both high-level aggregated insights and deep-dive analyses of individual visits, offering an invaluable resource for modeling clinical communication.

1. Introduction: The Unmet Need for Clinical Data

In the modern healthcare landscape, innovation in artificial intelligence, user experience, and patient-provider communication is often bottlenecked by one critical resource: data. Real-world clinical encounters are rich with multimodal information—the nuance of a patient’s tone, the physician’s interaction with the Electronic Health Record (EHR), and the non-verbal cues that build or break patient trust. However, this data is justifiably locked away behind stringent privacy regulations like HIPAA.

This creates a significant barrier for researchers, data scientists, and informatics educators who need to understand, model, and improve clinical interactions.

How can we develop an AI that summarizes a visit if it has never “seen” one?
How can we teach medical students to improve their “digital bedside manner” without real examples to analyze?

This project was born from this challenge. Inspired by pioneering work in clinical data collection, we set out to create a high-fidelity simulation of a multimodal clinical dataset and build a powerful, interactive web-based dashboard to explore it.

This article provides a comprehensive walkthrough of the project, from its conceptual inspiration to the final, running SvelteKit application.

2. The Inspiration: The “Observer” Dataset

This project is a direct response to the groundbreaking research detailed in the 2025 Journal of the American Medical Informatics Association article:

“Observer: creation of a novel multimodal dataset for outpatient care research”
Johnson, K. B., Alasaly, B., Jang, K. J., Eaton, E., Mopidevi, S., & Koppel, R. (2025).
https://doi.org/10.1093/jamia/ocaf182

The “Observer” paper details the immense effort to create a shareable, privacy-preserving dataset of real-world outpatient primary care visits. The authors’ goal was to create a resource to “support ambulatory care innovation” and enable future research.

Their dataset, which we use as a “gold standard” for our simulation, includes:

360-degree room-view video and audio
Egocentric (head-mounted) video from both the patient and provider
EHR-derived audit logs, diagnoses, and clinical notes
Patient and provider post-visit surveys (e.g., satisfaction ratings)
Structured metadata like exam room layout and patient demographics

The authors explicitly designed this resource to serve three main goals:

AI research
Communication studies
Informatics education

Our project adopts these three goals as its core pillars.
We are not just building a “chart viewer”; we are building a tool that serves these specific, high-level research purposes.

3. Phase 1: Synthetic Data Generation

Since we cannot access the raw “Observer” dataset, our first task was to create a large, varied, and realistic simulation of its analyzed output.

A dashboard with only 4 or 5 data points is a toy.
A dashboard with 100+ data points, each with unique properties, becomes a valid tool for observing trends.

To achieve this, we used a Python script to generate a 101-file JSON dataset, mimicking the structure of the data types described in the paper.

The Data Generator Script

This script, data_generator.py, is the engine of our project. It creates two types of files:

visits_metadata.json: A single master file containing the 100 “visits.” This file includes the high-level metadata needed for our global dashboard.
visit_XXX_analysis.json: 100 individual files, one for each visit. This file simulates the analysis of the raw video/audio.

import json
import random
import os

# --- Configuration ---
NUM_VISITS = 100
OUTPUT_DIR = "./output_data"
PROVIDERS = ["Dr. Aris", "Dr. Basu", "Dr. Chen", "Dr. Patel", "Dr. Yilmaz"]
DIAGNOSES = [
    "Hypertension", "Type 2 Diabetes", "Annual Checkup", "Anxiety",
    "Musculoskeletal Pain", "Follow-up", "Hyperlipidemia", "Fatigue"
]
ROOM_LAYOUTS = [
    "computer_facing_patient", "computer_on_side_wall", "computer_in_corner"
]

TOPICS = {
    "rapport": (30, 90),
    "education": (120, 400),
    "ehr_use": (60, 300),
    "admin": (60, 180),
    "physical_exam": (100, 300)
}

def generate_visit_timeline(total_duration_sec):
    timeline = []
    current_duration = 0
    topic_keys = list(TOPICS.keys())
    
    while current_duration < total_duration_sec:
        topic = random.choice(topic_keys)
        min_sec, max_sec = TOPICS[topic]
        duration = random.randint(min_sec, max_sec)
        
        if current_duration + duration > total_duration_sec:
            duration = total_duration_sec - current_duration
            if duration <= 0:
                break
        
        timeline.append({
            "topic": topic,
            "duration_sec": duration
        })
        current_duration += duration
        
    return timeline

def main():
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    all_metadata = []
    
    print(f"Generating {NUM_VISITS} synthetic visits...")
    
    for i in range(1, NUM_VISITS + 1):
        visit_id = f"visit_{i:03d}"
        total_duration_min = random.randint(10, 35)
        total_duration_sec = total_duration_min * 60
        
        patient_talk_ratio = round(random.uniform(0.3, 0.7), 2)
        interruptions = random.randint(0, 7)
        
        # --- Metadata ---
        meta = {
            "visit_id": visit_id,
            "provider": random.choice(PROVIDERS),
            "patient_satisfaction": round(random.uniform(2.5, 5.0), 1),
            "provider_satisfaction": round(random.uniform(2.5, 5.0), 1),
            "duration_minutes": total_duration_min,
            "diagnoses": random.sample(DIAGNOSES, random.randint(1, 3)),
            "room_layout": random.choice(ROOM_LAYOUTS),
            "patient_talk_ratio": patient_talk_ratio,
            "interruptions": interruptions
        }
        all_metadata.append(meta)
        
        # --- Analysis Data ---
        timeline = generate_visit_timeline(total_duration_sec)
        
        analysis = {
            "visit_id": visit_id,
            "timeline": timeline,
            "metrics": {
                "patient_talk_ratio": patient_talk_ratio,
                "provider_talk_ratio": round(1.0 - patient_talk_ratio, 2),
                "interruptions": interruptions
            }
        }
        
        filepath = os.path.join(OUTPUT_DIR, f"{visit_id}_analysis.json")
        with open(filepath, 'w') as f:
            json.dump(analysis, f, indent=2)
            
    # --- Master Metadata File ---
    meta_filepath = os.path.join(OUTPUT_DIR, "visits_metadata.json")
    with open(meta_filepath, 'w') as f:
        json.dump(all_metadata, f, indent=2)
        
    print(f"Successfully generated {NUM_VISITS} visits in '{OUTPUT_DIR}' directory.")
    print("Please move these files to your 'static/data/json/Multimodal/' folder.")

if __name__ == "__main__":
    main()

4. Phase 2: The Svelte Component Architecture

With our 101 data files generated and moved to static/data/json/Multimodal/, we began building the frontend using SvelteKit.

We used Chart.js for visualizations and structured the app into four key components.

Component 1: `AggregateAnalyticsTable.svelte` (The “Innovation” Goal)

<script>
  export let allMetadata = [];

  let providerStats = {};

  $: {
    const stats = {};
    const providers = [...new Set(allMetadata.map(visit => visit.provider))];
    
    providers.forEach(provider => {
      if (provider) {
        stats[provider] = { totalRatio: 0, totalSatisfaction: 0, totalInterruptions: 0, count: 0 };
      }
    });

    allMetadata.forEach(visit => {
      if (visit && visit.provider && stats[visit.provider]) {
        const p = stats[visit.provider];
        p.totalRatio += visit.patient_talk_ratio;
        p.totalSatisfaction += visit.patient_satisfaction;
        p.totalInterruptions += visit.interruptions;
        p.count += 1;
      }
    });

    Object.keys(stats).forEach(provider => {
      const p = stats[provider];
      if (p.count > 0) {
        stats[provider] = {
          avgRatio: (p.totalRatio / p.count * 100).toFixed(1) + '%',
          avgSatisfaction: (p.totalSatisfaction / p.count).toFixed(1),
          avgInterruptions: (p.totalInterruptions / p.count).toFixed(1)
        };
      } else {
        stats[provider] = { avgRatio: 'N/A', avgSatisfaction: 'N/A', avgInterruptions: 'N/A' };
      }
    });
    
    providerStats = stats;
  }
</script>

<style>
  .stats-table { width: 100%; border-collapse: collapse; margin-bottom: 20px; }
  .stats-table th, .stats-table td { text-align: left; padding: 10px; border-bottom: 1px solid #ddd; }
  .stats-table th { background-color: #f4f4f4; }
</style>

<table class="stats-table">
  <thead>
    <tr>
      <th>Provider</th>
      <th>Avg. Patient Talk Ratio</th>
      <th>Avg. Patient Satisfaction</th>
      <th>Avg. Interruptions</th>
    </tr>
  </thead>
  <tbody>
    {#each Object.entries(providerStats) as [provider, stats]}
      <tr>
        <td>{provider}</td>
        <td>{stats.avgRatio}</td>
        <td>{stats.avgSatisfaction} / 5</td>
        <td>{stats.avgInterruptions}</td>
      </tr>
    {/each}
  </tbody>
</table>

Component 2: `TimelineChart.svelte` (The “Communication Study”)

<script>
  import { onMount, onDestroy } from 'svelte';
  import { Chart } from 'chart.js/auto';

  export let visitData;

  let canvasElement;
  let chartInstance = null;

  const pastelColors = {
    rapport: '#A7C7E7',
    education: '#C1E1C1',
    ehr_use: '#FDFD96',
    admin: '#FFB347',
    physical_exam: '#FFC0CB'
  };

  const options = {
    indexAxis: 'y',
    responsive: true,
    maintainAspectRatio: false,
    plugins: { legend: { position: 'top' } },
    scales: { x: { stacked: true }, y: { stacked: true } }
  };

  function updateChart(data) {
    if (!chartInstance || !data) return;
    chartInstance.data = {
      labels: ['Visit Timeline'],
      datasets: data.timeline.map(segment => ({
        label: segment.topic,
        data: [segment.duration_sec],
        backgroundColor: pastelColors[segment.topic] || '#E0E0E0'
      }))
    };
    chartInstance.update();
  }

  $: updateChart(visitData);

  onMount(() => {
    if (canvasElement) {
      chartInstance = new Chart(canvasElement, {
        type: 'bar',
        data: { labels: [], datasets: [] },
        options: options
      });
      if (visitData) updateChart(visitData);
    }
  });

  onDestroy(() => {
    if (chartInstance) chartInstance.destroy();
  });
</script>

<canvas bind:this={canvasElement}></canvas>

Components 3 & 4: `TalkDoughnutChart.svelte` & `SatisfactionBarChart.svelte`

These follow the same pattern:

Import Chart from chart.js/auto
Use onMount / onDestroy
Update chart reactively with $:
Use doughnut and bar types respectively

5. Phase 3: The Final `.svx` Page & Web Display

We used an .svx file to combine markdown, Svelte logic, and CSS in one place.

Full Page Code (`clinical-dashboard.svx`)

<script lang="ts">
  import TalkDoughnutChart from '$lib/components/common/data-viz/TalkDoughnutChart.svelte';
  import SatisfactionBarChart from '$lib/components/common/data-viz/SatisfactionBarChart.svelte';
  import AggregateAnalyticsTable from '$lib/components/common/data-viz/AggregateAnalyticsTable.svelte';
  import TimelineChart from '$lib/components/common/data-viz/TimelineChart.svelte';
  import { onMount } from 'svelte';

  let allMetadata = [];
  let selectedVisitData = null;
  let selectedMeta = null;

  async function handleVisitSelect(visit) {
    selectedMeta = visit;
    selectedVisitData = null;
    const res = await fetch(`/data/json/Multimodal/${visit.visit_id}_analysis.json`);
    const data = await res.json();
    selectedVisitData = data;
  }

  onMount(async () => {
    const res = await fetch('/data/json/Multimodal/visits_metadata.json');
    allMetadata = await res.json();
  });
</script>

<style>
  .visit-list { display: flex; flex-wrap: wrap; gap: 10px; margin-bottom: 20px; max-height: 200px; overflow-y: auto; border: 1px solid #eee; padding: 10px; }
  .visit-btn { padding: 8px 12px; border: 1px solid #ccc; border-radius: 4px; cursor: pointer; }
  .visit-btn:hover { background-color: #f0f0f0; }
  .selected { background-color: #A7C7E7; font-weight: bold; border-color: #5b7a97; }
  .dashboard-grid { display: grid; grid-template-columns: 300px 1fr; gap: 30px; }
  .chart-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 20px; }
</style>

<hr class="my-12 border-t-[3px] border-primary/30 dark:border-primary/40 rounded-full" />

### Global Innovation Dashboard

<AggregateAnalyticsTable {allMetadata} />

<hr class="my-12 border-t-[3px] border-primary/30 dark:border-primary/40 rounded-full" />

### Select a Visit to Analyze (100 visits)

<div class="visit-list">
  {#each allMetadata as visit (visit.visit_id)}
    <button 
      class="visit-btn"
      class:selected={selectedMeta?.visit_id === visit.visit_id}
      on:click={() => handleVisitSelect(visit)}
    >
      {visit.visit_id}
    </button>
  {/each}
</div>

<hr class="my-12 border-t-[3px] border-primary/30 dark:border-primary/40 rounded-full" />

### Analysis Dashboard

{#if selectedVisitData}
  <h4>Displaying: {selectedMeta.visit_id}</h4>
  <div class="dashboard-grid">
    <div>
      <strong>Visit Details</strong>
      <ul>
        <li><strong>Provider:</strong> {selectedMeta.provider}</li>
        <li><strong>Duration:</strong> {selectedMeta.duration_minutes} min</li>
        <li><strong>Layout:</strong> {selectedMeta.room_layout}</li>
        <li><strong>Diagnoses:</strong> {selectedMeta.diagnoses.join(', ')}</li>
      </ul>
      
      <strong>Visit Scores</strong>
      <ul>
        <li><strong>Patient Satisfaction:</strong> {selectedMeta.patient_satisfaction} / 5</li>
        <li><strong>Provider Satisfaction:</strong> {selectedMeta.provider_satisfaction} / 5</li>
        <li><strong>Interruptions:</strong> {selectedVisitData.metrics.interruptions}</li>
        <li><strong>Patient Talk Ratio:</strong> {selectedVisitData.metrics.patient_talk_ratio * 100}%</li>
      </ul>
    </div>
    
    <div>
      <strong>Visit Timeline</strong>
      <div style="height: 250px; margin-bottom: 20px;">
        <TimelineChart visitData={selectedVisitData} />
      </div>

      <div class="chart-grid">
        <div>
          <strong>Talk Time Analysis</strong>
          <div style="height: 250px;">
            <TalkDoughnutChart metrics={selectedVisitData.metrics} />
          </div>
        </div>
        <div>
          <strong>Satisfaction Scores</strong>
          <div style="height: 250px;">
            <SatisfactionBarChart meta={selectedMeta} /> 
          </div>
        </div>
      </div>
    </div>
  </div>

{:else if selectedMeta}
  <p>Loading data for {selectedMeta.visit_id}...</p>
{:else}
  <p>Please select a visit from the list above to see its analysis.</p>
{/if}

6. Conclusion

This project successfully demonstrates the power of simulation and modern web technologies to tackle complex, real-world problems.

By standing on the shoulders of the “Observer” research paper, we were able to define a clear set of goals for a clinical data dashboard.

Using Python for data generation and SvelteKit for the frontend, we built a two-tiered application:

Global Dashboard – Aggregates data from 100 visits to provide high-level, actionable insights for clinic managers or AI researchers.
Individual Dashboard – Allows for a deep, multimodal analysis of any selected encounter.

Built in a fraction of the time it would take to gain access to real data, this tool provides an invaluable resource for training, research, and development — perfectly mirroring the objectives of the original “Observer” dataset.

Multimodal clinical auditor

Multimodal clinical dashboard

Multimodal clinical dashboard

Simulating Multimodal Healthcare Data: Building an Interactive Svelte Dashboard for Clinical Insights

1. Introduction: The Unmet Need for Clinical Data

2. The Inspiration: The “Observer” Dataset

3. Phase 1: Synthetic Data Generation

The Data Generator Script

View Source Code

4. Phase 2: The Svelte Component Architecture

Component 1: `AggregateAnalyticsTable.svelte` (The “Innovation” Goal)

View Source Code

Component 2: `TimelineChart.svelte` (The “Communication Study”)

View Source Code

Components 3 & 4: `TalkDoughnutChart.svelte` & `SatisfactionBarChart.svelte`

5. Phase 3: The Final `.svx` Page & Web Display

Full Page Code (`clinical-dashboard.svx`)

View Source Code

6. Conclusion

Multimodal clinical auditor

Multimodal clinical dashboard

Multimodal clinical dashboard

Simulating Multimodal Healthcare Data: Building an Interactive Svelte Dashboard for Clinical Insights

1. Introduction: The Unmet Need for Clinical Data

2. The Inspiration: The “Observer” Dataset

3. Phase 1: Synthetic Data Generation

The Data Generator Script

View Source Code

4. Phase 2: The Svelte Component Architecture

Component 1: AggregateAnalyticsTable.svelte (The “Innovation” Goal)

View Source Code

Component 2: TimelineChart.svelte (The “Communication Study”)

View Source Code

Components 3 & 4: TalkDoughnutChart.svelte & SatisfactionBarChart.svelte

5. Phase 3: The Final .svx Page & Web Display

Full Page Code (clinical-dashboard.svx)

View Source Code

6. Conclusion

Component 1: `AggregateAnalyticsTable.svelte` (The “Innovation” Goal)

Component 2: `TimelineChart.svelte` (The “Communication Study”)

Components 3 & 4: `TalkDoughnutChart.svelte` & `SatisfactionBarChart.svelte`

5. Phase 3: The Final `.svx` Page & Web Display

Full Page Code (`clinical-dashboard.svx`)