Abstract
This thesis proposes a framework that jointly uses textual and visual features of user generated social media data for mining the distribution of user interests. The mined distribution can serve for personalized ads recommendation or social content recommendation. The proposed framework consists of three steps: feature extraction, model training, and user interest mining.We choose boards from popular users on Pinterest to collect training and test data. For each pin we extract the term-document matrices as textual features, bag of visual words (BoVW) as low-level visual features, and attributes as mid-level visual features to bridge the semantic gap between low-level visual feature and textual descriptions. After feature extraction, a word selection process is applied to filter out words with an ambiguous distribution. The new term-document matrices of three types of features are then used to train topic models using discriminative latent Dirichlet allocation (DLDA). Finally, a representative distribution selection method is performed to choose the final topic distribution of each input document. In the prediction phase, pins from other popular user are used to evaluate the classification accuracy and pins from other common users are used to evaluate the recommendation performance. Our experimental results shows the efficacy of the proposed method. Also, the image recommendation demonstration verifies the feasibility of our method applied on real data.