ClinicalMirror: Auditing AI Fairness and Explainability in Clinical Decision Support

2. Introduction: The Problem with AI in Healthcare

I have been thinking about this problem for a while. AI is moving into healthcare faster than our ability to audit it. Radiology AI reads scans. Sepsis algorithms trigger alerts. Readmission models allocate resources. These are not experimental tools — they are in production, influencing real clinical decisions for real patients today.

And yet, most clinicians using these tools have no idea how they work. They see a number — “74% readmission risk” — and they either trust it or they don’t. There is no middle ground, no explanation, no way to interrogate the prediction. That is a problem.

It became a documented crisis in 2019 when Ziad Obermeyer and colleagues published a landmark paper in Science showing that a widely-used commercial health algorithm — deployed across hundreds of hospitals — was systematically underestimating illness severity in Black patients. The algorithm used healthcare costs as a proxy for health needs. Because Black patients historically received less care due to systemic inequities, they had lower costs, and the algorithm interpreted lower costs as lower need. The result: Black patients who were equally sick as White patients were assigned lower risk scores and received fewer resources. The bias was not intentional. It was structural. And it was invisible until someone looked.

That paper changed how I think about AI in healthcare. The question is not just “does the model work?” The question is “does the model work equally well for everyone?” And beyond that: “can a clinician understand why it made a specific prediction for a specific patient?”

These two questions — fairness and explainability — are what ClinicalMirror is built to answer.

3. Why Fairness and Explainability Are Not Optional

4. What I Built: ClinicalMirror

ClinicalMirror is a single-page web application that does four things:

1. Runs a trained XGBoost readmission model in the browser using ONNX Runtime Web — no server, no API call, no data transmission.
2. Explains each prediction using pre-computed SHAP values, visualized as an interactive waterfall chart.
3. Audits for demographic bias using counterfactual analysis and a population-level fairness chart.
4. Synthesizes all signals into a per-patient Trust Score (green / yellow / red) that tells a clinician whether to rely on the prediction.

The live dashboard, as shown in my portfolio, demonstrates this with patient P0002 — a 48-year-old Black or African American female with COPD. The model assigns her a 36% readmission risk. The counterfactual analysis reveals that if she were identified as White, that risk would drop to 15% — a 21-percentage-point gap driven almost entirely by Race/Ethnicity (SHAP value: +0.576). The Trust Score is LOW TRUST. The dashboard is doing exactly what it was designed to do: surfacing a prediction that should not be used without clinical scrutiny.

5. Technical Architecture

┌─────────────────────────────────────────────────────────┐
│                    BROWSER (Client)                      │
│                                                          │
│  ┌──────────┐   ┌──────────────┐   ┌─────────────────┐  │
│  │  Svelte  │   │ ONNX Runtime │   │  Apache ECharts │  │
│  │   UI     │──▶│     Web      │   │   (Charts)      │  │
│  │          │   │  (Inference) │   │                 │  │
│  └────┬─────┘   └──────────────┘   └─────────────────┘  │
│       │                                                  │
└───────┼──────────────────────────────────────────────────┘
        │ Firestore SDK
        ▼
┌─────────────────────────────────────────────────────────┐
│              Firebase (Google Cloud)                     │
│                                                          │
│  ┌──────────────────┐    ┌──────────────────────────┐   │
│  │    Firestore     │    │    Firebase Hosting       │   │
│  │  (Patient data,  │    │  (Static site delivery)  │   │
│  │   SHAP values,   │    │                          │   │
│  │   counterfactuals│    └──────────────────────────┘   │
│  └──────────────────┘                                    │
└─────────────────────────────────────────────────────────┘
        ▲
        │ (one-time upload)
┌─────────────────────────────────────────────────────────┐
│              Google Colab (Offline)                      │
│                                                          │
│  scikit-learn · XGBoost · SHAP · skl2onnx               │
│  (Train model, compute SHAP + counterfactuals,          │
│   export to ONNX + JSON)                                │
└─────────────────────────────────────────────────────────┘

Key design decision: All ML inference happens in the browser via ONNX Runtime Web. The SHAP values and counterfactual predictions are pre-computed in Colab and stored in Firestore — they are not computed in real time. This is architecturally honest: production clinical AI systems do offline explainability, not real-time SHAP computation. The dashboard retrieves and visualizes pre-computed results, which is both faster and more realistic.

Stack summary:

6. The Data: Designing Bias Intentionally

I used synthetic patient data for this project. This was a deliberate choice, not a compromise. Synthetic data eliminates HIPAA concerns, makes the project immediately shareable, and — critically — allows me to design the bias intentionally so the fairness analysis is meaningful and interpretable.

The dataset contains 1,000 synthetic patients with the following features:

The Risk Formula

The readmission risk formula uses threshold-based binary terms rather than small continuous multipliers. This is a critical design choice. XGBoost builds decision trees that split on thresholds — “prior_admissions > 1.5” — not on continuous gradients. Threshold-based signal gives the model clean, learnable decision boundaries.

risk = (
    0.12                                                      # ~12% base rate (realistic)
    + 0.25 * (prior_admissions >= 2)                         # strong: 2+ prior admissions
    + 0.18 * (prior_admissions >= 3)                         # even stronger: 3+
    + 0.15 * (comorbidity_score >= 4)                        # high comorbidity
    + 0.12 * (comorbidity_score >= 7)                        # very high comorbidity
    + 0.10 * (age >= 75)                                     # elderly
    + 0.08 * (age >= 85)                                     # very elderly
    + 0.14 * (race == 'Black or African American')           # documented disparity
    + 0.10 * (insurance == 'Medicaid')                       # SES proxy
    + 0.08 * (insurance == 'Uninsured')                      # access barrier
    + 0.12 * (diagnosis == 'Heart Failure')                  # high-risk diagnosis
    + 0.08 * (diagnosis == 'COPD')
    + 0.20 * ((age >= 70) & (comorbidity_score >= 4))        # interaction: elderly + sick
    + 0.15 * ((prior_admissions >= 2) & (insurance == 'Medicaid'))  # compounding disadvantage
    - 0.008 * length_of_stay                                 # protective: more treatment time
).clip(0.04, 0.82)                                           # hard cap prevents runaway rates

The .clip(0.04, 0.82) at the end is essential. Without it, additive terms stack and push the overall readmission rate above 60%, which is clinically unrealistic and destroys model signal. Real-world 30-day readmission rates are 15–20%. I target ~30–35% to ensure enough positive cases for the model to learn from while remaining plausible.

Resulting readmission rates by race:

The 14-percentage-point gap between White and Black patients is intentional and mirrors documented real-world disparities. It is what makes the fairness analysis meaningful.

7. The Model: XGBoost Readmission Prediction

I trained an XGBoost classifier on 800 patients (80/20 train/test split, stratified). The model parameters were chosen to match the dataset size and signal strength:

from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score

model = XGBClassifier(
    n_estimators=200,       # more trees = better generalization at this data size
    max_depth=4,            # shallow enough to avoid overfitting 1,000 samples
    learning_rate=0.05,     # slow learning rate with more trees = better calibration
    subsample=0.8,          # row subsampling reduces variance
    colsample_bytree=0.8,   # feature subsampling reduces correlation between trees
    random_state=42,
    eval_metric='auc',
    verbosity=0
)
model.fit(X_train, y_train)

auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
# Result: AUC = 0.732

Test AUC: 0.732 — comparable to published 30-day readmission models in the literature (typical range: 0.68–0.78). This is not a coincidence: the signal structure I designed mirrors the clinical factors that genuinely predict readmission.

Feature importance (from XGBoost):

Prior admissions is the dominant predictor — as it should be clinically. Race/Ethnicity ranks second, which is the fairness signal the dashboard is designed to surface.

A Note on Hyperparameter Tuning

I want to address this directly because it is a common mistake. When the model AUC is low (e.g., 0.58–0.60), the instinct is to tune hyperparameters — adjust n_estimators, learning_rate, max_depth. This is the wrong approach. Hyperparameter tuning on weak data gives marginal gains at best (0.575 → 0.584 in my testing). The signal lives in the data generation formula, not the model parameters. Fix the data first. The model follows.

8. Explainability: SHAP Values

SHAP (SHapley Additive exPlanations) is the gold standard for explaining tree-based model predictions. It assigns each feature a value representing its contribution to the prediction for a specific patient — positive values push the prediction toward readmission, negative values push it away.

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)

feature_names = [
    'Age', 'Sex', 'Race/Ethnicity', 'Diagnosis',
    'Prior Admissions (12mo)', 'Length of Stay (days)',
    'Comorbidity Score', 'Insurance Type'
]

# Store per-patient SHAP values for Firestore
shap_records = []
for i in range(len(df)):
    shap_records.append({
        name: round(float(val), 4)
        for name, val in zip(feature_names, shap_values[i])
    })
df['shap_values'] = shap_records

I use shap.TreeExplainer rather than the model-agnostic KernelExplainer because it is exact for tree-based models (not an approximation) and orders of magnitude faster. For 1,000 patients, TreeExplainer runs in seconds. KernelExplainer would take hours.

Why pre-compute? SHAP values are computed once in Colab and stored in Firestore. The dashboard retrieves them per patient. This is architecturally correct — real production systems do not compute SHAP values in real time for every prediction. They compute them in batch and serve them on demand.

9. Fairness: Counterfactual Analysis

The counterfactual analysis is the most clinically intuitive fairness measure in the dashboard. For each patient, I ask: what would the model predict if this patient’s race were “White”?

# Use the same encoder fit on training data — critical for consistency
white_code = float(encoders['race'].transform(['White'])[0])

X_counterfactual = X.copy()
X_counterfactual[:, 2] = white_code   # column index 2 = race_enc

cf_proba = model.predict_proba(X_counterfactual)[:, 1]

df['predicted_risk']               = model.predict_proba(X)[:, 1].round(4)
df['counterfactual_risk_if_white'] = cf_proba.round(4)
df['counterfactual_delta']         = (cf_proba - df['predicted_risk']).round(4)

For patient P0002 — the case shown in my portfolio — this produces:

• Actual predicted risk: 36%
• Counterfactual risk (if White): 15%
• Delta: −21%

A 21-percentage-point drop from a single demographic change is a stark finding. It means the model is not just using race as a minor adjustment — it is a dominant driver of the prediction. The SHAP value for Race/Ethnicity for this patient is +0.576, confirming this: race is pushing the prediction up by 57.6 percentage points relative to the base rate.

This is exactly the kind of finding that should trigger clinical review before acting on the prediction.

10. The Trust Score

The Trust Score synthesizes three signals into a single clinician-facing indicator:

1. Counterfactual gap — how much does the prediction change with a demographic swap?
2. Prediction uncertainty — how close is the prediction to 0.5 (the decision boundary)?
3. Combined signal — green / yellow / red

def compute_trust_score(row):
    risk        = row['predicted_risk']
    delta       = abs(row['counterfactual_delta'])
    # Uncertainty: predictions near 0.5 are least confident
    # A model saying "50% risk" is essentially saying "I don't know"
    uncertainty = 1 - abs(risk - 0.5) * 2

    if delta > 0.15 or uncertainty > 0.6:
        return 'red'      # large demographic gap OR very uncertain
    elif delta > 0.08 or uncertainty > 0.35:
        return 'yellow'   # moderate gap OR moderate uncertainty
    else:
        return 'green'    # consistent and confident

Trust Score Thresholds

The thresholds are not arbitrary. A 15% counterfactual gap means the model assigns meaningfully different risk based on race alone — that is a clinically significant disparity. An uncertainty above 60% means the model is essentially guessing. Either condition warrants a red flag.

11. How to Use the Dashboard

P0002 — Black or African American, 48yo Female (COPD)

ClinicalMirror Audit Dashboard

12. Chart Legends and How to Read Them

Feature                    Impact on Risk Score
─────────────────────────────────────────────────
Race/Ethnicity    ████████████████████  +0.576  ← increases risk by 57.6pp
Sex               ██████                +0.156  ← increases risk by 15.6pp
Age               ███                   +0.077  ← increases risk by 7.7pp
Prior Admissions  ██                    +0.031  ← small positive contribution
Comorbidity       █                     +0.012
Length of Stay    ▌                    -0.008  ← slightly reduces risk
Insurance Type    ▌                    -0.003
Diagnosis         ▌                    -0.001

Avg Predicted Risk by Demographic Group

1.0 ┤
0.8 ┤
0.6 ┤
0.4 ┤  ████  ████  ████  ████
    │  0.28  0.42  0.33  0.37
0.2 ┤
0.0 ┤
    └──────────────────────────
       White  Black  Hisp  Asian

┌─────────────────────────────────────┐
│  🔴  LOW TRUST                      │
│                                     │
│  Base Prediction:    35.7%          │
│  Demographic Variance: 20.7%        │
└─────────────────────────────────────┘

13. EU AI Act Compliance

14. Discussion and Limitations

15. Conclusion

I built ClinicalMirror because I believe the most important question in healthcare AI right now is not “how accurate is the model?” but “who does the model fail, and why?” Accuracy is a population-level metric. Fairness is a patient-level question. Explainability is what connects the two.

The dashboard I built makes this concrete. For patient P0002 — a 48-year-old Black woman with COPD — the model assigns a 36% readmission risk. The counterfactual analysis shows that if she were White, that risk would be 15%. The SHAP analysis shows that Race/Ethnicity is the dominant driver of her prediction, contributing +0.576 to the risk score. The Trust Score is RED.

That is not a model doing its job. That is a model encoding demographic disparity as clinical risk. And without the tools I built into ClinicalMirror, no clinician using this model would ever know.

The technical stack — Svelte, Firebase, ONNX, Apache ECharts — is entirely free and runs in the browser. The concepts — SHAP, counterfactual fairness, calibrated uncertainty — are well-established in the research literature. The regulatory framework — the EU AI Act — is already in force. There is no technical barrier to responsible AI in healthcare. There is only the choice to build it.

16. Future Directions

Near-Term (Next 3–6 months)

1. Multi-attribute counterfactuals. Extend the counterfactual analysis to all protected attributes — age, sex, insurance type — and compute intersectional counterfactuals (e.g., “if this patient were a young White male with private insurance”).

2. Calibration curve panel. Add a reliability diagram showing whether the model’s predicted probabilities match observed readmission rates. A well-calibrated model saying “70% risk” should be right 70% of the time.

3. Bias mitigation toggle. Add a “what if we applied bias correction?” toggle that re-weights predictions using post-processing fairness constraints (e.g., equalized odds) and shows the updated Trust Score.

4. Real MIMIC-IV data. Apply for MIMIC-IV access through PhysioNet and retrain the model on real clinical data. This would transform ClinicalMirror from a demonstration into a genuine research tool.

Medium-Term (6–18 months)

5. Federated learning integration. Explore whether the model can be trained across multiple hospital datasets without centralizing patient data — using federated learning to improve generalization while preserving privacy.

6. Temporal drift monitoring. Add a model monitoring panel that tracks prediction distributions over time and alerts when the model’s behavior shifts — a critical requirement for production clinical AI.

7. Multi-model comparison. Allow users to load and compare multiple models side-by-side — e.g., a model trained on one hospital’s data vs. another — to surface institutional variation in algorithmic bias.

8. Natural language explanations. Replace the SHAP waterfall chart with a plain-language narrative explanation generated from the SHAP values — making the explainability panel accessible to non-technical clinical staff.

Long-Term Vision

The long-term vision for ClinicalMirror is a clinical AI audit layer — a standardized interface that sits between any clinical AI model and its users, providing real-time explainability, fairness auditing, and regulatory compliance documentation. As the EU AI Act and equivalent regulations in the US and UK come into full enforcement, every hospital deploying clinical AI will need exactly this capability. ClinicalMirror is a prototype of what that looks like.

17. References

1. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342

2. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.

3. Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31(2).

4. European Parliament. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act). Official Journal of the European Union.

5. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

6. Barocas, S., Hardt, M., & Narayanan, A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press. https://fairmlbook.org

7. Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28(1), 31–38.