Project Visuospatial: The Augmented Anatomy Mirror

A Mixed-Reality Interface for Clinical Skills Acquisition

This is a live test of the AR Clinical Skills tool.

1. Executive Summary

The Augmented Anatomy Mirror is a web-based Augmented Reality (AR) tool designed to bridge the gap between theoretical anatomical knowledge and practical clinical application. By leveraging computer vision to superimpose accurate pathology and skeletal structures onto live video feeds of standardized patients (or peers), this tool transforms any laptop into a high-fidelity clinical simulation station. It addresses the critical need for accessible, repeatable, and visuospatial training in medical education without the cost of physical mannequins.

2. The Clinical Problem: The “Visuospatial Deficit”

Medical education currently suffers from a fundamental disconnection in how anatomy is taught versus how it is practiced.
· The Problem: Students memorize anatomy from 2D textbooks and static diagrams, but real patients are dynamic, 3D biological systems. Novice students often struggle to mentally map the “textbook diagram” onto a real, breathing human standing in front of them. This “Visuospatial Deficit” leads to errors in physical exams and a lack of confidence during initial patient encounters.
· The Solution: This tool forces the student to “see through the skin.” By projecting pathology (e.g., Chapman’s Reflex Points, organ locations) onto a real human video feed, we train the student’s brain to instantly locate clinical landmarks on a living subject.

3. How It Works (Student Workflow)

The application is designed for “dorm room simulation,” allowing students to practice clinical skills outside of the lab.

1. Launch & Calibrate: The student opens the URL on their laptop. They can choose to use the built-in “Standardized Patient” (high-fidelity video loops) or activate their Webcam to practice on a roommate or themselves.
2. Case Presentation: The system generates a clinical scenario (e.g., “Patient reports sharp lower right quadrant pain”).
3. Active Palpation: Using the mouse (or touch screen), the student must locate the correct anatomical landmark associated with that pathology (e.g., McBurney’s Point).
4. Real-Time Feedback:
   · Success: If they identify the correct area, the system confirms the diagnosis (“Appendicitis”), reveals the underlying clinical reasoning, and plays the associated auscultation sound (e.g., bowel sounds or friction rubs).
   · Correction: If they miss, the system provides “Hot/Cold” hints relative to anatomical anchors (e.g., “Too high. Look below the 12th rib”).
5. X-Ray Reinforcement: Students can toggle a procedural “X-Ray Mode” that draws a dynamic skeleton over the patient in real-time, visualizing the relationship between the skin surface and the bone structure underneath.

4. Institutional Value & Efficiency

This project serves as a “Force Multiplier” for the educational team, offering significant ROI compared to traditional methods:
· Replaces Expensive Models: High-fidelity simulation mannequins cost upwards of $10,000–$50,000 and are limited to simulation centers. This tool replicates the locational utility of a mannequin for $0 hardware cost.
· Saves Lecture Hours: Instead of spending valuable faculty time demonstrating surface anatomy landmarks repeatedly, students can perform “pre-work” using this tool, entering the lab already proficient in landmark identification.
· Unlimited “Reps”: Unlike a scheduled lab session, this tool is available 24/7. A student can diagnose 50 cases in 10 minutes before an exam, building muscle memory that textbooks cannot provide.

5. Technical Architecture

The system is built on a modern, lightweight stack designed for performance and privacy.
· Frontend Framework: SvelteKit. chosen for its reactive DOM updates and high performance on mobile devices.
· Computer Vision: Google MediaPipe Pose. This runs entirely client-side (in the browser). No video data is ever sent to a server, ensuring 100% HIPAA/FERPA compliance and zero latency.
· Procedural Graphics: HTML5 Canvas API. The skeletal overlays and Chapman points are drawn mathematically in real-time using Bezier curves and vector geometry, allowing the “bones” to stretch and rotate naturally as the patient moves.
· State Management: JavaScript/Svelte Stores. Manages the logic for 40+ clinical cases, randomizing presentations and tracking user success metrics.
· Assets: AI-Generated video loops (Veo) for standardized patients and open-source clinical audio files for auscultation training.

6. Future Improvements & Roadmap

To further enhance realism and educational depth, the following features are proposed for Phase 2:

1. “Sterile Field” Voice Control: Integration of the Web Speech API to allow students to control the interface (e.g., “Next Case”, “Show X-Ray”) via voice commands, simulating the hands-free environment of a sterile operating room.
2. LLM-Powered History Taking: Integration of a local LLM (Large Language Model) to give the patient a “voice.” Students would have to interview the patient via microphone to uncover symptoms before palpating, merging history-taking with physical exam skills.
3. Multi-User Remote Precepting: A “Proctor Mode” where a faculty member can watch a student’s webcam feed remotely and digitally “point” to areas on the student’s screen to guide their hands during telemedicine training.

7. Conclusion

The Augmented Anatomy Mirror is not just a study aid; it is a shift in pedagogical approach. By overlaying digital clinical data onto physical human reality, we remove the abstraction of the textbook and provide students with a direct, interactive, and safe environment to master the geography of the human body. This project demonstrates a capability to deliver high-impact educational technology that aligns with the modern medical curriculum’s focus on active learning and early clinical exposure.