Design of Machine Learning Models for Medical Data Analytics in the Assisted Reproductive Technology Field


Student thesis: Doctoral Thesis

View graph of relations


Related Research Unit(s)


Awarding Institution
Award date10 Oct 2022


Infertility is an important problem in the world as nearly 48.5 million people have infertility-related diseases. In China, the infertility rate is estimated to be 10%–15% in women and men of reproductive age (15–45 years old) (Logan et al. 2019). The development of assisted reproductive technologies (ARTs), such as in vitro fertilization (IVF), has helped people achieve their dream of having a child. However, the success rate of IVF remains relatively low. For women under 35, the clinical pregnancy rate is 44.5% after IVF. The live birth rate after IVF is 38.9%.

An IVF cycle often involves extracting eggs from a woman, fertilizing them in the laboratory, and then transferring a fertilized egg (also called embryo) into the woman’s uterus. Selecting a viable embryo to transfer is an important step in IVF. Conventional embryo selection is based on morphological evaluation (giving an embryo a grade) by skilled embryologists at the blastocyst stage (day 5 of the embryo) (Tran et al. 2019). As manual grading is quite subjective and the criteria are ambiguous, this method is rather time-consuming and has low effectiveness. Moreover, prediction of the pregnancy results of embryo implantation is challenging.

Conventional machine learning models, such as decision tree, support vector machine (SVM), naive Bayes, logistic regression, and multi-layer perceptron (MLP), are widely adopted in many medical data analytics studies. Examples include when extracting research topics from the COVID-19-related literature (Cheng et al. 2020), predicting pregnancy via logistic regression (Sufriyana et al. 2020), and predicting the survival time of cancer patients (Walczak and Velanovich 2018). However, conventional machine learning methods in medical image analysis highly depend on task-specific feature extraction, such as texture feature (Mall et al. 2019) and local binary pattern (LBP; Morales et al. 2008) extraction, which highly relies on researchers’ expertise (Xiao et al. 2018). In addition, these methods often use images manually captured by embryologists, which makes the feature construction work very time-consuming. Further, the performance of these features is not remarkable.

Deep learning is a new generation of machine learning. Prevalent deep learning models, such as the convolutional neural network (CNN), recurrent neural network (RNN), and transformer, are end-to-end architectures. The feature extraction and classifier are combined into a single model without requiring handcrafted feature selection anymore. Recently, deep learning has been applied to the medical data analytics field in tasks such as lung cancer tumor detection with computed tomography (CT) data (Tekade and Rajeswari 2018), skin cancer classification (Esteva et al. 2017), and breast cancer classification (Chen et al. 2021). One of the advantages of deep learning is the automatic extraction of latent features without human supervision. However, deep learning methods typically require a large volume of human-labeled data, and the training time of a deep learning model is quite long. Inspired by transfer learning, in which knowledge learned from different but related source domains is transferred, some researchers are attempting to use pre-trained deep learning to extract general image features from data to help improve performance.

A time-lapse monitoring (TLM) video is a new type of medical data that could monitor the growth of embryos automatically without human intervention (Campbell et al. 2018). It not only provides high-quality embryo images but also offers growing information about the embryo. A high-quality embryo image could help embryo selection. Time-series information could facilitate the early prediction of viable embryo formation. Integrated with clinical records, predicting the pregnancy result of embryo implantation is also possible.

As selecting viable embryos and predicting pregnancy after IVF are two challenges for researchers, we aim to answer the following question in this thesis:

How to design new machine learning models to utilize TLM data for improving the efficiency of viable embryo selection and embryo implantation result prediction?

There are two research paradigms in the information systems (IS) discipline: behavioral science and design science (Hevner et al. 2004). The behavioral science paradigm aims to develop and test theories to explain human or organizational behavior. The design science paradigm aims to create innovative artifacts to extend human and organizational capabilities. The information systems design theory (ISDT) is a prescriptive theory that integrates normative and descriptive theories into design activities to help produce more effective artifacts in the IS field (Walls et al. 1992). There is a considerable amount of design science research that follows the ISDT to create new artifacts, including algorithms, ISs, and models, to help solve business problems (Abbasi et al. 2012; X. Liu et al. 2020; Yang et al. 2022).

To answer the research question mentioned above, we designed three studies based on the ISDT to create our new machine learning models that help improve the effectiveness of IVF.

In the first study, guided by the design science research paradigm, we utilized the kernel theory of data (feature-level) fusion to build a new machine learning model for viable embryo selection. The proposed machine learning model integrates general image features extracted by transfer learning to task-specific features extracted using the histogram of oriented gradients (HOG) (Mizuno et al. 2012) method. We compared our model with several machine learning models, as well as deep learning models in previous studies, using static image frames extracted from time-lapse video data. We found that the proposed machine learning model outperforms other machine learning models in viable embryo selection. We also found that the proposed machine learning model outperforms several deep learning models reported in previous studies.

In the second study, guided by the design science research paradigm, we utilized the kernel theory of the attention mechanism to build a deep learning model (LSTM-Attention) for viable embryo prediction based on TLM video data. We first used a pretrained CNN (VGG16) to extract features from image frames. Long short-term memory (LSTM), an RNN model, dealt with the time information. The attention mechanism assigned an “attention score” to the input features, which could be used to weigh the significance of different parts of the input data. The integration of LSTM and the attention mechanism helped the prediction with time-lapse video data. Several experiments were conducted to evaluate the performance of the proposed method. We found that 1) our proposed deep learning model with TLM data outperforms other machine learning models with static image data, indicating that time-lapse video data could provide more valuable information than a single static image. 2) Our proposed deep learning model with TLM data outperforms other machine learning models with TLM data. This suggests that including temporal information processing along with the attention mechanism could improve viable embryo prediction. 3) The image size and number of frames have quite little effect on the proposed deep learning model’s performance. The dataset size factor has a larger impact on deep learning prediction compared to the other two factors.

In the third study, guided by the design science research paradigm, we utilized the kernel theory of data (joint) fusion to build a hybrid deep learning model, CNN-LSTM-MLP, to explore its effect on predicting the pregnancy result after embryo implantation using multi-source medical data. We conducted several experiments to examine the performance of our deep learning model. We found that hybrid deep learning models perform better than the CNN only, LSTM only, and CNN+LSTM models, indicating that multi-source data, especially when integrating medical indicators into image-video data, could provide more valuable information.

This study makes a significant contribution to the fields of medical data analytics, machine learning, and health information technology (HIT) application in the IVF field. The proposed machine learning models could help improve the performance of viable embryo selection as well as pregnancy prediction after IVF. In addition, our models could also be extended to other types of medical data, such as CT-based lung cancer classification and breast cancer classification.

    Research areas

  • Medical Data Analytics, Deep Learning, Attention Mechanism, In Vitro Fertilization