Abstract: 【Purpose/significance】In this paper,multi-modal information of text and image is integrated for emotion recognition,aiming
at solving the problem that single modal information cannot accurately predict the polarity of emotion【. Method/process】Based on the microblog data published by Internet users on Sina Weibo,this paper proposes a multi-modal emotion recognition algorithm based on DR-Transformer model.The pre-trained DenseNet and RoBERTa models are used to extract the emotional features of image mode and text mode respectively.The Modal Embedding mechanism was introduced to identify the sources of different modal features.A shallow Transformer Encoder is used to integrate the emotional features of different modes,and the self-attention mechanism is used to dynami‐cally adjust the weight of information features of each mode【. Result/conclusion】Experiments on the Weibo dataset show that:The emo‐tion recognition accuracy of this model is 79.84%; Compared with the sentiment classification algorithm based on single text and im‐age modes,the accuracy of this model is improved by 4.74% and 19.05% respectively; Compared with the feature fusion method that di‐rectly concatenate the feature vectors of different modes,the accuracy of this model is improved by 1.12%.It is proved that this model is scientific,reasonable and effective in emotion recognition【. Innovation/limitation】Multimodal information can be effectively integrated by using Modal Embedding and Self-Attention mechanism.The dataset of microblog network public opinion needs to be further ex‐panded.