Research Theme:

Neuromediomics: the Interface of Brain Imaging and Multimedia


Traditionally, neuroimaging and multimedia are conceived as two completely different disciplines. The first one deals with structural and functional mapping of the brain, while the latter concentrates on the representation and analysis of digital multimedia such as video and audio streams. We made initial effort in integrating these two fields in order to bridge the gaps between low-level multimedia features and high-level semantics via fMRI brain imaging under natural stimulus of watching video or listening audio streams. Our results show that: 1) there are meaningful couplings between brain’s fMRI responses and video/audio stimuli, suggesting the validity of linking semantics and low-level features via fMRI; 2) The computationally learned low-level feature sets from fMRI-derived semantic features can significantly improve the classification of video/audio categories in comparison with that based on original low-level features.

This theme of neuromediomics research aims to, on one hand, bridge semantic gaps in multimedia representation and analysis via brain imaging. On the other hand, quantitative representation of functional brain responses to multimedia stimuli via neuroimaging will fundamentally advance scientific understanding of the human brain and its functions. In the long-run, we hope that the advancements of neuromediomics will facilitate wider adoption of applying neuroimaging to assist multimedia analysis and using multimedia as neuroimaging natural stimuli to better understand the brain.

Representative Publications:

Yimin Hou, Ting Xiao, Shu Zhang, Xi Jiang, Xiang Li, Xintao Hu, Junwei Han, Lei Guo, L Stephen Miller, Richard Neupert, and Tianming Liu, Predicting Movie Trailer Viewer’s “Like/Dislike” via Learned Shot Editing Patterns, IEEE Transactions on Affective Computing, 2015. in press PDF

Junwei Han, Xiang Ji, Xintao Hu, Lei Guo, and Tianming Liu, Arousal Recognition Using Audio-Visual Features and FMRI-based Brain Response, IEEE Transactions on Affective Computing, 2015. vol. 6(4), pp. 337-347.PDF

Xintao Hu, Lei Guo, Junwei Han*, Tianming Liu*, Decoding Semantics Categorization during Natural Viewing of Video Streams, IEEE Transactions on Autonomous Mental Development, 2015. vol. 7(3), pp. 201-210. *Co-corresponding authors.PDF

Junwei Han, Changyuan Chen, Xintao Hu, Lei Guo and Tianming Liu, Learning Computational Models of Video Memorability from FMRI Brain Imaging, IEEE Transactions on Cybernetics, 2014. vol. 45(8), pp. 1692-703.PDF

Xiang Ji; Junwei Han; Xi Jiang; Xintao Hu; Lei Guo; Jungong Han; Ling Shao; Tianming Liu, Analysis of music/speech via integration of audio content and functional brain response, Information Sciences, 2015. vol. 297, pp. 271–282. PDF

Junwei Han, Kaiming Li, Xintao Hu, Sheng He, Lei Guo, Tianming Liu. Video abstraction based on fMRI-driven visual attention model, accepted, Information Sciences, 2014. vol. 281, pp. 781–796. PDF

Tianming Liu, Xintao Hu, Xiaojin Li, Mo Chen, Junwei Han, Lei Guo. Merging Neuroimaging and Multimedia: Methods, Opportunities and Challenges, accepted, IEEE Transactions on Human-Machine Systems, 2014. vol. 44(2), pp. 270-280. PDF

Junwei Han, Sheng He, Xiaoliang Qian, Dongyang Wang, Lei Guo, Tianming Liu. An object-oriented visual saliency detection framework based on sparse coding representations, IEEE Transactions on Circuits and Systems for Video Technology, 2013. vol. 23(12), pp. 2009 - 2021. PDF

Junwei Han, Xiang Ji, Xintao Hu, Dajiang Zhu, Kaiming Li, Xi Jiang, Guangbin Cui, Lei Guo, and Tianming Liu. Representing and Retrieving Video Shots in Human-Centric Brain Imaging Space, IEEE Transactions on Image Processing, 2013. vol. 22(7), pp. 2723 - 2736.PDF

Xintao Hu, Kaiming Li, Junwei Han, Xian-Sheng Hua, Lei Guo, Tianming Liu. Bridging the Semantic Gap via Functional Brain Imaging. IEEE Transactions on Multimedia, 14(2):314-325, 2012.PDF

Jiehuan Sun*, Xintao Hu*, Xiu Huang, Yang Liu, Kaiming Li, Xiang Li, Junwei Han, Lei Guo, Tianming Liu**, Jing Zhang**, Inferring Consistent Functional Interaction Patterns from Natural Stimulus FMRI Data,NeuroImage, 2012. vol. 61(4), pp. 987–999. *Joint first authors, **Joint corresponding authors.PDF

Fan Deng, Dajiang Zhu, Lei Guo and Tianming Liu, FMRI Signal Analysis Using Empirical Mean Curve Decomposition, IEEE Transactions on Biomedical Engineering, 2013. vol. 60(1), pp. 42-54.PDF

Related Methodological and Algorithmic Publications:

Sheng He, Junwei Han, Xintao Hu, Ming Xu, Lei Guo, Tianming Liu, A Biologically Inspired Computational Model for Image Saliency Detection, ACM Multimedia, 2011. pp. 1465-1468.PDF

Xintao Hu, Fan Deng, Kaiming Li, Tuo Zhang, Hanbo Chen, Xi Jiang, Jinglei Lv, Dajiang Zhu, Li Xie, Carlos, Faraco, Degang Zhang, Arsham Mesbah, Junwei Han, Xian-Sheng Hua, Stephen Miller, Lei Guo, Tianming Liu. Bridging Low-level Features and High-level Semantics via fMRI Brain Imaging for Video Classification. ACM Multimedia, 2010. pp. 451-460.PDF

Kaiming Li, Tuo Zhang, Xintao Hu, Dajiang Zhu, Hanbo Chen, Xi Jiang, Fan Deng, Jinglei Lv, Carlos, Faraco, Degang Zhang, Arsham Mesbah, Junwei Han, Lie Lu, Xian-Sheng Hua, Lei Guo, Stephen Miller, Tianming Liu, Human-friendly Attention Models for Video Summarization, ICMI, 2010.PDF

Tianming Liu, Hong-Jiang Zhang, and Feihu Qi. A systematic rate controller for MPEG-4 FGS video streaming. ACM/Springer Multimedia Systems. 2004;8(5):369-379.PDF

Tianming Liu, Hong-Jiang Zhang, and Feihu Qi. A Novel Video Key-Frame-Extraction Algorithm Based on Perceived Motion Energy Model. IEEE Transactions on Circuits and Systems for Video Technology. 2003;13(10):1006-1013.PDF