Fasih Haider
ABSTRACT
In social media, vlogs (video blogs) are a form of unidirectional communication, where the vloggers (video bloggers) convey their messages (opinions, thoughts, etc.) to a potential audience which cannot give them feedback in real time. In this kind of communication, the non-verbal behaviour and personality impression of a video blogger tends to influence viewers' attention because non-verbal cues are correlated with the messages conveyed by a vlogger. In this study, we use the acoustic and visual features (body movements that are captured by low-level visual descriptors) to predict the six different attitudes (amusement, enthusiasm, friendliness, frustration, impatience and neutral) annotated in the speech of 10 video bloggers. The automatic detection of attitude can be helpful in a scenario where a machine has to automatically provide feedback to bloggers about their performance in terms of the extent to which they manage to engage the audience by displaying certain attitudes. Attitude recognition models are trained using the random forest classifier. Results show that: 1) acoustic features provide better accuracy than the visual features, 2) while fusion of audio and visual features does not increase overall accuracy, it improves the results for some attitudes and subjects, and 3) densely extracted histograms of flow provide better results than other visual descriptors. A three-class (positive, negative and neutral attitudes) problem has also been defined. Results for this setting show that feature fusion degrades overall classifier accuracy, and the classifiers perform better on the original six-class problem than on the three-class setting.
- H. Akira, F. Haider, L. Cerrato, N. Campbell, and S. Luz. Detection of cognitive states and their correlation to speech recognition performance in speech-to-speech machine translation systems. In Sixteenth Annual Conference of the International Speech Communication Association, pages 2539--2543, 2015.Google Scholar
- J. Allwood and P. J. Henrichsen. Predicting the attitude flow in dialogue based on multi-modal speech cues. In NEALT Proceedings. Northern European Association for Language and Technology; 4th Nordic Symposium on Multimodal Communication; November 15-16; Gothenburg; Sweden, number 093, pages 47--53. Linköping University Electronic Press, 2013.Google Scholar
- J.-I. Biel, O. Aran, and D. Gatica-Perez. You are known by how you vlog: Personality impressions and nonverbal behavior in youtube. In ICWSM, pages 446--449, 2011.Google Scholar
- J.-I. Biel and D. Gatica-Perez. Vlogsense: Conversational behavior and social attention in youtube. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 7(1):33, 2011. Google ScholarDigital Library
- J.-I. Biel, L. Teijeiro-Mosquera, and D. Gatica-Perez. Facetube: predicting personality from facial expressions of emotion in online conversational video. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 53--56. ACM, 2012. Google ScholarDigital Library
- L. Breiman. Manual on setting up, using, and understanding random forests v3. 1. Statistics Department University of California Berkeley, CA, USA, 1, 2002.Google Scholar
- K. Chatfield, V. S. Lempitsky, A. Vedaldi, and A. Zisserman. The devil is in the details: an evaluation of recent feature encoding methods. In BMVC, volume 2, page 8, 2011.Google ScholarCross Ref
- J. A. De Moraes, A. Rilliard, B. A. de Oliveira Mota, and T. Shochi. Multimodal perception and production of attitudinal meaning in brazilian portuguese. Proc. Speech Prosody, paper, 340, 2010.Google Scholar
- F. Eyben, F. Weninger, F. Groß, and B. Schuller. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia, pages 835--838. ACM, 2013. Google ScholarDigital Library
- J. Fleiss. Statistical methods for rates and proportions. john wiley & sons; new york: 1981. The measurement of interrater agreement, pages 212--36.Google Scholar
- A. Liaw and M. Wiener. Classification and regression by randomforest. R news, 2(3):18--22, 2002.Google Scholar
- M. Liu, R. Wang, S. Li, S. Shan, Z. Huang, and X. Chen. Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 494--501. ACM, 2014. Google ScholarDigital Library
- N. Madzlan, J. Han, F. Bonin, and N. Campbell. Towards automatic recognition of attitudes: Prosodic analysis of video blogs. Speech Prosody, Dublin, Ireland, pages 91--94, 2014.Google ScholarCross Ref
- N. A. Madzlan, J. G. Han, F. Bonin, and N. Campbell. Automatic recognition of attitudes in video blogs-prosodic and visual feature analysis. In INTERSPEECH, pages 1826--1830, 2014.Google Scholar
- N. A. Madzlan, Y. Huang, and N. Campbell. Automatic classification and prediction of attitudes: Audio-visual analysis of video blogs. In International Conference on Speech and Computer, pages 96--104. Springer, 2015.Google ScholarCross Ref
- N. A. Madzlan, J. Reverdy, F. Bonin, L. Cerrato, and N. Campbell. Annotation and multimodal perception of attitudes: A study on video blogs. In Proceedings from the 3rd European Symposium on Multimodal Communication, Dublin, September 17-18, 2015, number 105, pages 50--54. Linköping University Electronic Press, 2016.Google Scholar
- D. McNeill. Hand and mind: What gestures reveal about thought. University of Chicago Press, 1992.Google Scholar
- D. McNeill. Gesture and thought. University of Chicago Press, 2008.Google Scholar
- F. Perronnin, J. Sánchez, and T. Mensink. Improving the fisher kernel for large-scale image classification. In European conference on computer vision, pages 143--156. Springer, 2010. Google ScholarDigital Library
- J. Uijlings, I. Duta, E. Sangineto, and N. Sebe. Video classification with densely extracted hog/hof/mbh features: an evaluation of the accuracy/computational efficiency trade-off. International Journal of Multimedia Information Retrieval, 4(1):33--44, 2015.Google ScholarCross Ref
- A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/, 2008.Google Scholar
- C. Vogel and L. Mamani Sanchez. Epistemic signals and emoticons affect kudos. In P. Baranyi, editor, 3rd IEEE Conference on Cognitive Infocommunications, pages 517--522, 2012.Google Scholar
Recommendations
Application of Emotion Recognition and Modification for Emotional Telugu Speech Recognition
AbstractMajority of the automatic speech recognition systems (ASR) are trained with neutral speech and the performance of these systems are affected due to the presence of emotional content in the speech. The recognition of these emotions in human speech ...
Efficiency of chosen speech descriptors in relation to emotion recognition
This research paper presents parametrization of emotional speech using a pool of common features utilized in emotion recognition such as fundamental frequency, formants, energy, MFCC, PLP, and LPC coefficients. The pool is additionally expanded by ...
Emotion Recognition from Audio and Visual Data using F-score based Fusion
CODS '14: Proceedings of the 1st IKDD Conference on Data SciencesEmotion recognition has been one of the cornerstones of human-computer interaction. Although decades of work has attacked the problem of automatic emotion recognition from either audio or video signals, the fusion of the two modalities is more recent. ...
Comments