The evolution of drug resistance presents a challenge in cancer therapy, leading to treatment failure and tumor progression. Mathematical oncology has emerged as a vital approach to model the evolutionary dynamics of cancer cells, combining computer modeling with physical and biological insights, helps understand cancer's evolution and drug interactions. Yet, the growing complexity of drug targets and the integration of omics data challenge these models. Advancements in deep learning have revolutionized cancer analysis by integrating genomic and clinical data, improving diagnosis and treatment strategies. However, the opacity of such models poses interpretability issues in medical decisions.

To harness the strengths and mitigate the limitations of both mathematical modeling and machine learning techniques, we proposed the Learning-Enhanced Mathematical Oncology Model (Len-MoM) framework. This framework synergizes the precision of mathematical modeling with the predictive prowess of machine/deep learning algorithms. It is tailored for the development of personalized cancer treatment regimens. The integration of these methodologies significantly augments the efficacy of drug scheduling and dosing accuracy, leading to improved patient outcomes, and minimizing adverse effects through optimized dosage reduction.

Furthermore, our initiative extends to the fusion of deep learning with mathematical modeling to discover the intricate patterns within multimodal data. We advanced the Len-MoM framework by incorporating multi-omics data and pre-trained-BERT generated embeddings using a deep neural networks as encoders. This is combined with an interpretable neural ordinary differential equations (neural-ODE) model to articulate the dynamics of tumor evolution comprehensively. This enhancement amplifies the framework's descriptive and predictive capacities, facilitating a more refined and personalized approach to cancer therapy. In applying our methodology to an extensive collection of patient-derived xenograft (PDX) data, which encompasses a broad spectrum of treatments and drugs combinations across tumors from diverse origins, we demonstrate its capability to capture the individualized tumor dynamics. And we also quantify the contribution of the multimodal data to the model's prediction ability.

Moreover, as the next step of the Len-MoM, we proposed a reinforcement-learning-enabled individualized treatment framework for intermittent androgen depreviation therapy for prostate cancer, namely, I²ADT, to learn the patient-specific tumor dynamics and derive the optimal drug administration policy, enhancing the suppression of resistant cells by leveraging the competitive advantage of responsive cells. This approach is adaptable for optimizing treatments across various cancer types due to the inherent flexibility of the reinforcement learning. Cancer treatment typically involves multiple lines of therapies and the use of multiple drugs. Every patient has a unique cancer phenotype and tumor microenvironment. In the context of ADT in prostate cancer, our proposed I²ADT yielded a highly personalized dosing policy that maximizes the competition advantage of responsive cancer cells to suppress resistant cancer cells.

Our results show the transformative potential and applicability of the Len-MoM framework in clinical settings, highlighting its capacity to revolutionize personalized cancer treatment through enhanced drug scheduling, dosage precision, and the integration of complex multimodal data. This positions the Len-MoM framework as a beacon of innovation in the realm of mathematical oncology, promising significant advancements in the precision and personalization of cancer therapy.

Though limitations and challenges are presented in the current work, as we look to the future, we believe the collaboration of data scientists, pharmacologists, and oncologists could further optimize the proposed Len-MoM methods and other adaptive therapy strategies. Such interdisciplinary efforts are critical to harnessing the full potential of personalized medicine to enhance cancer treatment outcomes.

In summary, this thesis underscores the significant progress in integrating mathematical oncology with advanced machine learning techniques for personalized cancer therapy. It exemplifies the potential of mathematical modeling, reinforcement learning, and deep learning to revolutionize cancer treatment, moving towards more personalized and efficacious regimens.