Multi-Modal Diagnostic Assistant
Using CNN as a tool for clinical dermantology cases.
Project Overview
Summary
This project introduces a prototype Multi-Modal Diagnostic Assistant for veterinary dermatology, blending machine learning with an expert logic layer. A Convolutional Neural Network trained on skin lesion images provides baseline predictions, which are then refined by a clinical rules engine incorporating symptoms and patient history. This hybrid approach effectively emulates the nuanced diagnostic process of a veterinarian without requiring a massive multi-modal dataset.
I am developing a prototype “Multi-Modal Diagnostic Assistant” for veterinary dermatology. This tool aims to go beyond simple image recognition by emulating a veterinarian’s diagnostic process. It will analyze skin lesion images alongside essential clinical context (symptoms, patient history) to deliver intelligent and nuanced assessments.
Methodology (Our Approach)
To achieve this with available resources, I will employ a hybrid methodology combining machine learning with an expert-driven rules engine:
Machine Learning Core
- Convolutional Neural Network (CNN): Train a CNN in Google Colab using a public dataset of dog skin images to serve as the “vision expert,” providing baseline predictions based on visual data.
Expert Logic Layer
- Differential Diagnosis Engine: Build within the Streamlit application, this engine will use rules and logic based on clinical veterinary knowledge to refine diagnoses.
Intelligent Fusion
- Process: The app first retrieves the image-based prediction from the AI model. It then incorporates user-entered clinical data (e.g., itch level, lesion location) into the expert logic to refine the prediction, add context, and provide clinically relevant explanations.
- Advantage: This approach bypasses the need for a comprehensive multi-modal dataset while creating a prototype that functions as a sophisticated, expert-guided system.
Project Phases
Phase 1 (This Weekend’s Goal - MVP)
- Train the base image model in Google Colab.
- Develop and deploy a functional Streamlit web app with:
- UI for clinical data entry.
- Core rule-based logic for at least 4 dermatological conditions.
- Deliverable: A live, shareable web link for the app.
Phase 2 (Future Refinement)
- Enhance the image model with additional data.
- Expand the logic engine to include more conditions and complex diagnostic rules.
- Improve the user interface and experience.
Phase 3 (Future Expansion)
- Explore developing a true multi-modal neural network if a suitable dataset is found or created.
- Integrate Explainable AI (XAI) techniques to visually highlight areas of focus in images.
Why & How This Project is Useful
This project stands out as an exceptional portfolio piece for the following reasons:
Demonstrates Advanced Thinking
- Showcases creative problem-solving by addressing the challenge of missing datasets with a practical, intelligent solution.
Showcases a Hybrid Skillset
- Combines technical expertise (Python, TensorFlow, Streamlit) with domain knowledge in veterinary science and medicine, a rare and valuable skill combination.
Tells a Compelling Story
- This is not just another image classifier; it’s a narrative of bridging technology and real-world clinical applications, highlighting a deep understanding of the problem and a unique ability to address it effectively.
Multi-modal diagnostic tool
