Dyna-Lipid: A Full-Stack, N-of-1 Adaptive Trial Platform for Severe Hypertriglyceridemia Using a Heuristically-Governed Reinforcement Learning Agent

Abstract

Background: The management of severe hypertriglyceridemia (SHTG) is complex, with high inter-patient variability in therapeutic response. Standard-of-care protocols are often static and fail to adapt to an individual’s dynamic responses to diet, adherence, and specific pharmacotherapies.

Objective: To design, build, and deploy a full-stack, N-of-1 adaptive trial platform, “Dyna-Lipid,” capable of ingesting live patient data and providing real-time therapeutic recommendations generated by a Reinforcement Learning (RL) agent.

Methods: A “Digital Twin” (a gymnasium environment) was developed in Python to simulate a patient with SHTG. A Reinforcement Learning agent (PPO, Stable Baselines3) was trained against this environment in Google Colab to learn an optimal therapeutic policy. This trained agent was wrapped in a Flask API and deployed as a web service on Render. A SvelteKit frontend, hosted on Firebase, provides a dashboard that reads live patient data from a Firebase Firestore database. The frontend calls the Render API to get a live prediction, which is then displayed to the clinician. The dashboard features several D3.js visualizations to display historical data and model performance.

Results: Initial model training (V2, V3) resulted in “lazy” or “greedy” policies (always choosing “Fibrate” or “Volanesorsen,” respectively). A “Balanced” V4 model was developed by introducing a DRUG_COST penalty to the reward function. This produced a more dynamic policy, which was further refined by adding a heuristic override in the Flask API to act as a “safety net,” preventing the model from recommending powerful drugs for non-critical TG levels. The final system successfully demonstrates a complete, closed loop: (1) Svelte reads from Firebase, (2) Svelte sends data to Render, (3) Render returns a safe, dynamic prediction, (4) Svelte displays the result.

Conclusion: The Dyna-Lipid project serves as a successful, end-to-end proof-of-concept for the architecture of a live, adaptive N-of-1 trial. It demonstrates the feasibility of integrating a live-data frontend (Svelte/Firebase) with a productionized RL model (Python/Flask/Render) to provide dynamic, personalized therapeutic recommendations.

1. Introduction

Severe Hypertriglyceridemia (SHTG) represents a significant clinical challenge, predisposing patients to an increased risk of acute pancreatitis and cardiovascular disease. The management of SHTG is multifaceted, involving lifestyle modifications and pharmacological interventions with agents such as fibrates, omega-3 fatty acids, and novel therapeutics.

The clinical context for this work is grounded in the established research of leaders in the field, such as Dr. Robert Hegele, whose work at the Robarts Research Institute and the Lipid Genetics Clinic has been foundational in identifying the molecular genetic basis for severe dyslipidemias, including familial chylomicronemia syndrome (FCS) [1, 2]. This research has directly informed the development and trial of new therapies, including the antisense oligonucleotide Volanesorsen, which targets APOC3 and is a key therapeutic option in our simulated model [1].

Despite these advances, patient response is highly heterogeneous. A static, one-size-fits-all protocol is suboptimal. An ideal system would adapt its recommendations based on a patient’s unique, real-time data—a concept known as an N-of-1 trial.

1.1. The Dyna-Lipid Hypothesis

We hypothesize that an adaptive, N-of-1 clinical trial, managed via a Svelte/Firebase web application, can identify the optimal therapeutic strategy (drug choice, dose, and timing) for a patient with SHTG more effectively than a standard-of-care approach.

We posit that a Reinforcement Learning (RL) agent, developed in Colab, can create a “digital twin” for each patient, learning their unique response patterns to interventions (e.g., fibrates, omega-3s, volanesorsen) and self-reported inputs (diet, adherence, side effects). This trained agent, deployed as a live API, can then be queried by a clinician-facing dashboard to provide real-time, optimized therapeutic policies.

This paper details the design, implementation, failures, and final success of the “Dyna-Lipid” platform, a full-stack proof-of-concept for this hypothesis.

2. Materials and Methods

The project was executed in distinct phases: (1) Environment Simulation, (2) RL Agent Training, (3) API Deployment, and (4) Frontend Integration.

2.1. System Architecture

The Dyna-Lipid platform is a decoupled, full-stack application. The architecture is designed for scalability and separation of concerns: data, logic, and presentation are handled by distinct services.

graph TD
    subgraph Browser
        A[SvelteKit Frontend]
    end

    subgraph Firebase (Google Cloud)
        B[Firebase Hosting] --> A
        C[Firestore Database]
    end

    subgraph Render.com
        D[Python/Flask API]
        E[RL Model (ppo_v4.zip)]
    end

    subgraph Google Colab (Development)
        F[Jupyter Notebook]
        G[Digital Twin (Gymnasium)]
        H[RL Training (Stable Baselines3)] --> E
    end

    A -- (1. Reads Live Data) --> C
    A -- (2. User Updates Data) --> C
    A -- (3. Sends Obs for Prediction) --> D
    D -- (4. Loads Model) --> E
    D -- (5. Returns Recommendation) --> A

2.2. Technology Stack

• RL Training: Google Colab Pro, Python, gymnasium, stable-baselines3
• Backend API: Python 3.12, Flask, gunicorn, flask-cors
• API Hosting: Render.com (Web Service)
• Frontend: SvelteKit (Svelte 5)
• Database: Google Firebase (Firestore)
• Frontend Hosting: Firebase Hosting
• Visualization: D3.js (v7)

2.3. Phase 1 & 2: The Digital Twin & RL Agent (V4)

The project’s core logic resides in the “Balanced” V4 model, which was the result of multiple failed iterations (V2, V3). The V2 “Slacker” model (stuck on Fibrate) and V3 “Greedy” model (stuck on Volanesorsen) proved that the reward function is the most critical component.

The V4 environment (SHTG_DigitalTwin_V2_4) introduced two key concepts: a strong linear penalty for high TG and a drug cost.

This code snippet from the Colab notebook defines the “Balanced” reward function that produced a successful, dynamic policy.

# Critical Code: The "Balanced" Reward Function (SHTG_DigitalTwin_V2_4)

# [!!] NEW: Define a "cost" for each drug [!!]
# Fibrate=3, Omega-3=1, Volanesorsen=10
# This teaches the model that Volanesorsen is "expensive"
self.DRUG_COST = np.array([3.0, 1.0, 10.0])

def step(self, action):
    # ... (Physiological simulation code) ...

    # --- [!!] "BALANCED" REWARD FUNCTION [!!] ---
    reward = 0
    
    # 1. Strong reward for being in the target zone
    if self.current_tg < self.TG_TARGET:
        reward += 50
    
    # 2. Strong *linear* penalty for being over target
    else:
        # Strong, but not exponential (which caused V3 failure)
        reward -= (self.current_tg - self.TG_TARGET) * 15 
    
    # 3. Increased side effect penalties
    reward -= (self.current_myalgia * 10)
    reward -= (self.current_gi_distress * 5)
    
    # 4. Drug "Cost" Penalty (The KEY)
    # This penalizes the *choice* of drug
    drug_choice_cost = self.DRUG_COST[therapy_idx]
    reward -= drug_choice_cost
    
    # 5. Dose Penalty
    reward -= (dose * 2)
    
    return obs, reward, terminated, truncated, info

This model was trained for 500,000 timesteps using the PPO algorithm and saved as ppo_dyna_lipid_v4.zip.

2.4. Phase 3 & 4: API Deployment and Heuristic Override

The trained ppo_dyna_lipid_v4.zip model was deployed on a Render web service running a Flask API. However, even the V4 model was found to be overly aggressive at specific edge cases (e.g., recommending Volanesorsen for a TG of 9.0).

To ensure patient safety (a primary constraint), a Heuristic Override was implemented directly in the app.py file. This logic acts as a “safety catch” after the model provides its recommendation, using simple if statements to enforce clinical common sense.

# Critical Code: The Heuristic Override in app.py

@app.route("/predict", methods=["POST"])
def predict():
    try:
        # ... (Get data, run model prediction) ...
        obs = data["observation"] 
        obs_np = np.array(obs, dtype=np.float32)
        current_tg = obs_np[0] # Get the TG level

        action, _states = model.predict(obs_np, deterministic=True)

        therapy_idx = int(round(np.clip(float(action[0]), 0, 2)))
        dose = float(np.clip(float(action[1]), 0.1, 1.0))
        therapy_name = THERAPY_MAP.get(therapy_idx, "Unknown")

        # --- [!!] 6. HEURISTIC OVERRIDE (FINAL LOGIC) [!!] ---
        #
        # If the model is being "greedy" and picking the
        # strongest drug (Volanesorsen) for a non-critical
        # TG level, override it.
        #
        if current_tg < 10.0 and therapy_name == "Volanesorsen":
            print(f"OVERRIDE: TG={current_tg} is not critical. Model's 'Volanesorsen' " \
                  f"recommendation was overridden to 'Fibrate'.")
            
            # Change the recommendation
            therapy_name = "Fibrate"
            dose = 0.5 # A standard maintenance dose
        else:
            print(f"POLICY: Model's recommendation of '{therapy_name}' was " \
                  f"approved for TG={current_tg}.")
        # --- End of Override ---

        return jsonify({
            "recommendation_therapy": therapy_name,
            "recommendation_dose": round(dose, 2)
        })
    # ... (Error handling) ...

2.5. Phase 4: Frontend and Database

A SvelteKit frontend provides the dashboard. The application is built around a central, reactive Svelte store (patientStore) that creates a real-time, two-way binding with a document in the Firestore database.

File: src/lib/api/Dyna_Lipid.js (abridged)

import { initializeApp } from "firebase/app";
import { getFirestore, doc, onSnapshot, updateDoc } from "firebase/firestore";
import { readable } from 'svelte/store';

// ... (firebaseConfig) ...
const app = initializeApp(firebaseConfig);
const db = getFirestore(app);

// Creates a reactive Svelte store that stays
// in sync with a single Firestore document.
function docStore(path) {
  return readable(null, (set) => {
    const unsubscribe = onSnapshot(doc(db, path), (snapshot) => {
      set(snapshot.data());
    });
    return unsubscribe;
  });
}

// Helper function to update a patient document
export async function updatePatient(patientId, data) {
  const docRef = doc(db, `patients/${patientId}`);
  await updateDoc(docRef, data);
}

// Create a live, readable store for our patient
export const patientStore = docStore('patients/P1000');

When the user submits the “Update Patient Live Data” form, updatePatient() is called. This changes the data in Firestore. The onSnapshot listener in the docStore immediately detects this change, which updates the patientStore. This, in turn, triggers the LivePrediction component to re-run, fetching a new prediction from the Render API. This completes the live-data loop.

3. Results

The project successfully produced a fully functional, end-to-end platform. The “Results” are the visualizations and the interactive loop itself. The dashboard successfully renders five key visualization components, all powered by D3.js, which parse a 90-day historical simulation JSON (rl_predicted_trajectory_p1000.json).

3.1. Figure Legends

• Figure 1: System Architecture. A Mermaid diagram illustrating the flow of data between the SvelteKit Frontend (on Firebase Hosting), the Firestore Database, the Render.com API, and the Colab training environment. It details the closed-loop nature of the application.
• Figure 2: Live Interaction Dashboard. A composite of three UI components.
  • (A) Live Data: A card displaying the current patient state, read in real-time from the patients/P1000 Firestore document.
  • (B) Live RL-Agent Recommendation: A card displaying the output from the Render API. This component shows “Loading…” and updates asynchronously, demonstrating the “cold start” of the free-tier API.
  • (C) Patient Input Form: A form that allows the user to modify the patient’s live data. Submitting this form calls updatePatient(), which writes to Firestore and triggers the entire loop, resulting in a new recommendation.
• Figure 3: Cohort Comparison Chart. A D3 bar chart comparing the primary outcome metric, “Time to Target (Days),” between the “Adaptive Trial (P1000)” and a “Historical Control” group. This chart directly visualizes the project’s core value proposition.
• Figure 4: Triglyceride Trajectory. A D3 line chart plotting three series over a 90-day period: “Actual” (simulated patient log), “RL Predicted” (the V4 model’s trajectory), and “Target” (a static 5.0 mmol/L line). This chart demonstrates the model’s ability to successfully bring the simulated TG level down to the target.
• Figure 5: Input & Policy Explorer. A multi-panel D3 bar chart. It visualizes three distinct time-series, all sharing a common date axis: “RL Agent Dose,” “Patient Diet Score,” and “Patient Adherence.” This allows a clinician to visually correlate the agent’s actions with the patient’s self-reported behaviors.
• Figure 6: Temporal Side-Effect Map. A D3 heatmap visualizing the intensity (scaled by radius and a pastel PuBu color gradient) of “Myalgia” and “GI Distress” over the 90-day trial. Dots are only rendered for intensity > 0, providing a clear, de-noised view of adverse events.

4. Discussion

The development of Dyna-Lipid yielded several critical insights. The primary challenge was not the technology stack but the “tuning” of the RL agent’s reward function. Our initial V2 model was a “slacker,” learning that the lowest-penalty action was to do the minimum (prescribe Fibrate), as it was not punished sufficiently for failing to reach the TG target. Conversely, our V3 model, which used an exponential penalty for high TG, became a “greedy” agent, learning that the most powerful drug (Volanesorsen) was the only solution, even for minor TG elevations.

The V4 “Balanced” model, which introduced a DRUG_COST, was the solution. By making Volanesorsen “expensive” (a high-penalty action), the model was forced to learn an economical policy: use Fibrate for small problems, but pay the cost for Volanesorsen when the TG penalty became greater than the drug’s cost.

The final addition of the app.py heuristic override is, in our view, the most important finding. It demonstrates that a pure, black-box RL model is insufficient for a clinical application. A “human-in-the-loop,” in the form of simple, hard-coded safety rules, is necessary to govern the model’s output and ensure patient safety.

4.1. Limitations

This project is a proof-of-concept of an architecture, not a clinically-validated tool. The limitations are significant:

1. Synthetic Digital Twin: The SHTG_DigitalTwin environment is an assumption. It is a simple, rules-based simulation, not a complex physiological model. Its “reactions” to drugs are hard-coded.
2. Single Patient: The model was trained and evaluated on data from a single, simulated patient (P1000).
3. Static Historical Data: The D3 charts are (currently) populated from a static JSON file generated during one evaluation run, not from a live, growing log of all patients.

5. Conclusion

The Dyna-Lipid project successfully demonstrates the design, development, and deployment of a full-stack, closed-loop platform for N-of-1 adaptive trials. It proves the feasibility of integrating a SvelteKit/Firebase frontend with a live, Python-based Reinforcement Learning API on Render. The final V4 model, governed by a heuristic safety layer, provides a dynamic, plausible, and safe set of recommendations.

Future work would involve replacing the simple Digital Twin with a more complex, data-driven model and expanding the Firestore database to manage a full cohort of patients, not just a single document.