Semantic analysis and annotation of histological and natural images


Student thesis: Master's Thesis

View graph of relations


  • Feiyang YU

Related Research Unit(s)


Awarding Institution
Award date15 Feb 2007


Semantic analysis of image content is an emerging but challenging field which is attracting more and more research interests, especially in view of the drawbacks of content-based image retrieval (CBIR). Compared with CBIR, annotation-based image retrieval moves towards bridging the semantic gap. Textual description can more precisely capture and present users’ information need as well as describe the semantic content of images. In this thesis we address the problem of modeling the semantic content of natural and medical images in some specific domains. We present a two-dimensional stochastic method for recognition and annotation of images. Our method is a generalization of the traditional hidden Markov model (HMM). We regard visual features extracted from images as observations, and the semantic concepts are hidden states which govern the underlying probabilistic characteristics of observations. With the aim to model the context of semantic features, we employ a second-order neighborhood system. The spatial relationships of features are encoded in the transitions between hidden states in our model. Hence we term our approach Spatial Hidden Markov Model or SHMM. It is well studied that a fully-connected 2-D HMM will lead to a NP-hard problem. To break the exponential complexity barrier, we assume the conditional independence of vertical and horizontal transition. That is, the vertical dependence of a hidden state on its upper hidden state, and the horizontal dependence of a hidden state on its previous hidden state are probabilistically independent. Our Markov assumption is that the hidden state of a block, given all its previous stats in a raster scanning, is only dependent on the states of its two neighbors---the upper one and the left one. This “past” notion is actually based on the non-symmetric half plane (NSHP), and is different from the definition generally adopted in 2D hidden Markov mesh random field (HMMRF). Since our model is a generalization of HMM, the three basic problems, namely, the generation probability of an observation, the state decoding problem, and the parameter estimation problem, associated with our SHMM should also be solved. For the first two problems, we extended the traditional Viterbi algorithm and Forward-Backward algorithm for SHMM. In addition, we derive our solution to the parameter estimation problem by use of the well-established Expectation-Maximization (EM) algorithm. Our semantic analysis is based on image blocks with equivalent sizes. Visual features such as color and texture characteristics are automatically extracted from those blocks. A bank of well-tuned Gabor filters are employed to analyze texture features in different frequency bands and orientations. Meanwhile, mean value of RGB color components are also used for representation of the content of images. Two sets of experiments on four categories of COREL images and histological images, which are obtained from human gastrointestinal tract, were conducted to verify our method. The experiment on histological images was carried on the same database of the I-Browse project. The results showed our method ameliorates many drawbacks of the previous I-Browse system and achieved a high recognition and annotation accuracy. Performance comparison with HMM also validate the superiority of our method. The second experiment on COREL images demonstrated that our model is superior to HMM in both recognition and annotation accuracy. Performance comparison with a state-of-the-art second-order hidden Markov mesh model-based image classification approach, namely the 2D Hierarchical HMM, also testifies the validity of our model. Based on our annotation results, i.e., a semantic label matrix for an image, we carried out semantic-based retrieval on those two image collections. The PR graphs showed that the performance of SHMM-based retrieval achieve retrieval rates which are much superior to that of HMM-based retrieval.

    Research areas

  • Image analysis, Image processing, Digital techniques, Semantic networks (Information theory)