Embedding learning has been a popular approach to process unstructured data for learning a low-dimensional representation, which maintains some specific characteristics of data. Despite the successes of existing methods, there are still several challenges in embedding learning remaining unsolved. First, how to utilize covariates to design a varying embeddings method. Second, deep neural network always acts as a generic framework for learning embedding, especially in recommender system. Nevertheless, few attempt has been made to quantify the corresponding statistical property. In this thesis, we first introduce the background of embedding learning and then we move on to our contributions in two relevant topics relating to embedding learning, including word embedding in sentiment analysis and recommender system.

Sentiment analysis measures inclination of textual documents, aiming to extract and quantify their subjective sentiment polarity. Generally, most sentiment analysis methods first numericalize textual documents through certain word embeddings framework, and then formulate sentiment analysis as an ordinal regression or classification task. Yet it is often ignored that different people may have different preference of wording, and thus a uniform word embeddings often leads to suboptimal performance. To accommodate the heterogeneity among individual persons, a covariate-assisted word embeddings method in a margin-based ordinal regression framework is developed. Particularly, covariates are incorporated through scaling factors to adjust the word embeddings. Moreover, we employ a block-wise coordinate descent scheme to tackle the resultant large-scale optimization task, and establish theoretical results to quantify the asymptotic behavior of the proposed method, guaranteeing its fast convergence rate in terms of prediction accuracy. Finally, we demonstrate the advantages of the proposed method over its competitors in both the Yelp Challenge dataset and synthetic datasets.

Recommender system refers to predicting preferred items for a user by integrating information from similar users or items. One of its key challenges is to leverage the available high-dimensional covariates by learning their low-dimensional embeddings to improve personalized prediction by capturing complex interaction between users and items. To tackle this problem, a deep recommender system is developed as an extension to the
classical collaborative filtering method for learning low-dimensional representation of high-dimensional covariates. Specifically, the proposed deep recommender system employs two deep neural networks to embed users and items into a low-dimensional space, and predicts rating by resorting to the structure of matrix factorization. More importantly, we establish
some asymptotic results of the deep recommender system in terms of generalization error, showing that it achieves fast convergence rate. To the best of our knowledge, this is among the first attempts to establish statistical guarantees for the generalization error of the deep recommender system. Through numerical experiments, we also demonstrate that the deep recommender system is capable of capturing effects of covariates on ratings, leading to higher prediction accuracy over its competitors in both simulated examples and a real application data set.

At the end of this thesis, a conclusion including some promising future work and a short summary are provided. In the future work, we introduce some possible extension of the deep network embeddings methods, including varying embeddings method with respect to continuous covariates and network embeddings for specific structure of neural network like convolutional neural network.