PupilRec : Leveraging Pupil Morphology for Recommending on Smartphones

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

11 Scopus Citations
View graph of relations


  • Xiangyu Shen
  • Hongbo Jiang
  • Daibo Liu
  • Kehua Yang
  • John C. S. Lui
  • Jiangchuan Liu
  • Schahram Dustdar
  • Jun Luo


Original languageEnglish
Pages (from-to)15538-15553
Journal / PublicationIEEE Internet of Things Journal
Issue number17
Online published9 Jun 2022
Publication statusPublished - 1 Sept 2022


As mobile shopping has gradually become the mainstream shopping mode, recommendation systems are gaining an increasingly wide adoption. Existing recommendation systems are mainly based on explicit and implicit user behaviors. However, these user behaviors may not directly indicate users’ inner feelings, causing erroneous user preference estimation and thus leading to inaccurate recommendations. Inspired by our key observation on the correlation between pupil size and users’ inner feelings, we consider using the change of pupil size when browsing to model users’ preferences, so as to achieve targeted recommendations. To this end, we propose PupilRec as a computer-vision-based recommendation framework involving a mobile terminal and a server side. On the mobile terminal, PupilRec collects users’ pupil size change information through the front camera of smartphones; it then pre-processes the raw pupil size data before transmitting them to the server. On the server side, PupilRec utilizes the Tsfresh package and Random Forest algorithm to figure out the key time series features directly implying user preferences. PupilRec then trains a neural network to fit a user preference model. Using this model, PupilRec predicts user preference to obtain a user-product matrix and further simplifies it by singular value decomposition. Finally, real-time recommendation is achieved by a collaborative filtering module that retrieves recommended contents to users’ smartphones. We prototype PupilRec and conduct both experiments and field studies to comprehensively evaluate the effectiveness of PupilRec by recruiting 67 volunteers. The overall results show that PupilRec can accurately estimate users’ preference, and can recommend products users interested in.

Research Area(s)

  • Cameras, Computational modeling, Energy Optimization, Internet of Things, multilayer perceptron (MLP), Optimization, Physiology, Pupillary Response, Pupils, Random Forest, Recommendation, Smart phones, User Preference Model

Citation Format(s)

PupilRec: Leveraging Pupil Morphology for Recommending on Smartphones. / Shen, Xiangyu; Jiang, Hongbo; Liu, Daibo et al.
In: IEEE Internet of Things Journal, Vol. 9, No. 17, 01.09.2022, p. 15538-15553.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review