Holistic Visual-Textual Sentiment Analysis with Prior Models (2024)

III-A Visual-textual Branch

The visual-textual branch aims at extracting sentiment-related features directly from the datasets. Two networks are adopted to extract features from text and image, respectively.

Text Encoder. We employ BERTweet-large[7] as the text encoder. BERTweet is based on BERT architecture[13] and was pre-trained using $873$M English tweet. It achieves state-of-the-art performance in tweets classification tasks. Since the text of visual-textual sentiment datasets originated from social media which is similar to tweets, the text encoder can acquire a strong prior from pre-training.We fine-tune the model with the text of the training set to adapt it to sentiment analysis. Next, we remove the classification head of the BERTweet model, and utilize the last layer’s hidden state of the classification token as the text representation, which is an embedding of $1024$ dimensions.

Image Encoder. We use Swin Transformer[6] as the image encoder, specifically a Swin-base network pre-trained on ImageNet-21k at a resolution of $384\times 384$. This pre-training enables the model to more robustly handle diverse and noisy open-domain images in the datasets. Next, similar to the text encoder, we fine-tune the Swin-base model with only images of the datasets, remove the classification head, and use the last hidden-state layer’s average pooling to generate a $1024$-dimensional image embedding.

III-B Visual Expert Branch

The visual expert branch employs four pre-trained neural networks to selectively extract high-valued face, scene, object, and optical character recognition (OCR) features in images.

Face Recognition.Face images are of pivotal importance in sentiment analysis[4, 12]. As shown in TableI, portrait photos with faces account for over $40\%$ of images in the datasets. To extract effective face features, we first use the MTCNN face detector[8] to filter out images without faces, then identify and retain the largest face in each image. We then adopt a pre-trained Facenet[25] based on Inception-ResNet architecture as the face feature extractor, which was pre-trained on VGGFace2.The resulting $512$-dimensional embedding represents the face features.

Scene and Object Detection. Scenes and objects are pivotal in conveying sentiment in images[16, 26]. For example, a photo of a delicious meal in a fancy restaurant reveals enjoyment, while a solitary tree in a graveyard implies melancholy. To capture these elements, our visual expert branch employs YOLOv5[9] pre-trained on the MS COCO dataset for object detection. The output, i.e., the predicted logit for each category, is summed up into an $80$-dimensional vector to represent the number and types of objects in each image.For scene detection, we use a DenseNet161 pre-trained on the Place365 dataset[10]. This model can recognize $365$ types of scenes, such as “airfield ”, “campus” and “highway”. We use the output logit of each image as the scene features, which are $365$-dimensional.

OCR. Text within images, such as internet memes, event posters, and screenshots of messages, is prevalent on social media platforms like Twitter. To extract OCR features, we employ a pre-trained Sentence-Transformer based on MPNet[27] as our OCR encoder. The process begins with extracting English words from images using Tesseract-OCR Engine[28]. These words are then concatenated into a sequence and processed through the Sentence-Transformer. The output $768$-dimensional sentence embeddings serve as the OCR features. To ensure relevance, we only extract OCR features from images containing at least five words, minimizing the impact of trivial text.

III-C CLIP Branch

The CLIP branch leverages CLIP’s pre-trained image and text encoders to extract corresponding features. The image encoder is a $24$-layer, $16$-head vision transformer, while the text encoder is a $12$-layer, $12$-head transformer, both with an embedding dimension of $768$. CLIP[29] was pre-trained on a massive dataset of $400$ million image-text pairs using a contrastive loss to minimize the distance between the paired text and image. This training enables CLIP’s encoders to discern high-level correlations between images and text that unimodal feature extractors cannot capture.

III-D Multimodal Feature Fusion

We employ a $3$-layer, $512$-width BERT model (i.e., Transformer encoder) with $4$ attention heads to integrate all visual and textual features for sentiment prediction. As illustrated in Fig.2b, we first zero pad all the extracted features to the same size of $1024$ to prevent distortion before fusion. These features are arranged into a sequence and processed by the BERT model. This process effectively fuses the information from the visual-textual, visual expert, and CLIP branches to make the final sentiment prediction.

IV-A Datasets and Preprocessing

MVSA-Single and MVSA-Multiple.These datasets contain $4,869$ and $19,600$ image-text pairs from Twitter. Each pair is assigned a human-annotated sentiment label (positive, negative, neutral).We follow the preprocessing procedure of[11] to remove pairs with conflicting sentiments in two modalities or disagreement among annotators.This results in $4,511$ and $17,025$ pairs for MVSA-Single and MVSA-Multiple, respectively.

TumEmo. TumEmo[24] is a large visual-textual emotion dataset containing $195,265$ image-text pairs from Tumblr, classified into seven emotion classes (i.e., angry, bored, calm, fearful, happy, loving, and sad) if the hashtags in the text contain certain keywords related to that emotion class. We adhere to the original study’s methodology, removing all hashtags that could reveal the ground truth label.

We follow BERTweet’s normalization procedure for text preprocessing, where emojis, user mentions, and URLs are replaced with special tokens, and the use of punctuation and abbreviations is standardized across sentences. TableI shows the detailed statistics of the datasets after preprocessing.For each dataset, we adopt an 8:1:1 split ratio for the training, validation, and test sets in a stratified manner, where the distributions of annotation labels are consistent across these sets.Pre-trained features are extracted once and saved for reuse.

IV-B Model Training

In model training, the CLIP and visual expert branches are fixed. As shown in Fig.2, our training procedure consists of two steps.

Unimodal Training.We first train the visual-textual branch separately with either text or images in the datasets. This procedure enables the encoders to learn enough knowledge for sentiment analysis from unimodal data.For the visual encoder (Swin Transformer), we train exclusively with images for $15$ epochs, with a learning rate of $5\mathrm{e}{-5}$ for MVSA and $1\mathrm{e}{-4}$ for TumEmo, and batch sizes of $16$ for MVSA and $32$ for TumEmo. For the textual encoder (BERTweet), we train with only text for $20$ epochs, setting the learning rate to $8\mathrm{e}{-6}$ for MVSA and $5\mathrm{e}{-5}$ for TumEmo, and batch sizes of $16$ for MVSA and $64$ for TumEmo.

Multimodal Training.After unimodal training, we train the entire model with both images and text. Note the dataset splits remain the same in the two stages. We initialize the visual-textual branch with the best-performing parameters (those with the lowest loss on the validation set) in unimodal training. Next, we train the visual-textual branch and the final multimodal feature fusion module together end-to-end. We use the AdamW optimizer with learning rates of $5\mathrm{e}{-6}$ for MVSA and $1\mathrm{e}{-5}$ for TumEmo, a batch size of $16$ for all datasets, and a dropout rate of $0.5$ to mitigate over-fitting. The training lasts $30$ epochs, and the parameters that yield the highest accuracy on the validation set are then used for testing.

IV-C Baselines

We benchmark our method against three publicly available, state-of-the-art methods in visual-textual sentiment analysis:

MGNNS[26] introduces a multi-channel graph neural network to model object, scene, and text representations based on the global co-occurrence patterns across the dataset.

Se-MLNN[12] integrates several pre-trained image features with contextual text features extracted via RoBERTa to comprehensively represent multimodal contents.

CLMLF[21] introduces a contrastive learning framework to enable effective learning of generalized representations for both image and text in visual-textual sentiment analysis.

Among the compared methods, Se-MLNN relies only on pre-trained features and is lightweight with 1M parameters, while our method and compared CLMLF, MGNNS have comparable sizes (500M, 200M, 100M respectively).

IV-D Quantitative Comparison

TableII shows the quantitative comparison between our proposed method and established baselines.While baseline results from their respective papers vary due to different dataset splits, we ensure a fair comparison by training each model with our dataset splits and reporting the metrics. Results for the MVSA datasets are averaged over 10 folds.

On the MVSA and TumEmo datasets, the proposed method outperforms other strong baselines, highlighting its effectiveness.The proposed visual-textual branch is based on pre-trained Swin Transformer and BERTweet, which allow our framework to employ the power of advanced neural network architectures. In contrast, Se-MLNN relies solely on pre-trained features without learning useful sentiment-related features directly from the dataset, which may lead to its compromised performance on the more complex TumEmo dataset with fine-grained emotion annotations.Regarding the visual expert branch, our method has four different visual features while OCR and face features are not used by MGNNS[26]. It is hard for a trainable visual encoderto learn these missing visual features directly.

IV-E Ablation Study

In our ablation study, we assess the contribution of each pre-trained and learned feature, the significance of multimodal input, and compare different fusion network architectures.

The Importance of Different Branches.We analyze performance changes when individual encoders are removed. Additionally, we provide a qualitative illustration (Fig.4), showcasing how each branch contributes to accurate sentiment predictions.As detailed in TableIII,the most substantial decline occurs with the removal of BERTweet, underscoring the critical role of textual information in visual-textual sentiment analysis. We hypothesize the reason is that learning sentiment from text is easier than from images, since adjectives in text can convey sentiment more directly.In the MVSA datasets, the removal of Swin Transformer, or the fine-tuning of the visual-textual branch (i.e., using fixed pre-trained Swin and BERTweet) leads to a notable performance decrease.This indicates the significance of directly learning sentiment-related features from the datasets. Among visual expert features, removing OCR causes the largest performance drop. This is in line with our observation about the prevalence and importance of image text in social media, which has been ignored in previous studies. We show some example cases in Fig.3. When the input image contains text to deliver sentiment-related information, the OCR feature can indeed help make correct sentiment predictions. In general, removing any of the pre-trained features diminishes performance, highlighting the necessity of each encoder. We present examples in Fig.4 to illustrate the importance of these encoders, where their removal leads to incorrect results. Removing the CLIP image encoder results in a decrease in accuracy and a marginal increase in weighted F1 on MVSA-Single. This could probably be caused by the class imbalance present in the dataset, as shown in TableI.For TumEmo, the removal of the CLIP image encoder significantly reduces performance. This suggests that the CLIP encoder captures unique image features distinct from the learnable Swin Transformer and is particularly effective in handling the noisy, open-domain images in this complex dataset.

In the ablation study on TumEmo, we find that the OCR and scene features only marginally help sentiment analysis. This observation aligns with [26]’s ablation study, where scene features had a lesser impact on TumEmo compared to MVSA. A possible explanation is that the OCR and scene encoders might not be sufficiently robust for the complex task of seven-class emotion classification: (1) OCR may not detect all the text present in the images (last column in Fig.3), (2) the text information in images may be inconsistent with the input text, and (3) it is more difficult to infer seven-class emotional predictions from scene features.

The Necessity of Multimodal Input.TableIV shows that the pre-trained visual (CLIP image and visual expert features) and textual (CLIP text feature) elements enhance unimodal sentiment analysis over using Swin Transformer and BERTweet alone. This supports the effectiveness of pre-trained features.When using unimodal data only, text-based analysis generally outperforms image-based analysis. This is in line with previous findings[26, 21], and could be due to a higher level of abstraction and subjectivity in the visual modality.Multimodal models show consistently superior performance than unimodal models, particularly on the challenging TumEmo dataset, which demonstrates the necessity of multimodal input for comprehensive sentiment analysis.

Fusion Module Architecture.We compare our BERT-based fusion module to a lightweight $3$-layer MLP with a $768$-size hidden layer, where MLP is used by[12] as the feature fusion module. In this setup, visual and textual features are concatenated and fed into the MLP with ReLU activation for sentiment prediction. TableIII shows that MLP is comparable to BERT on the TumEmo but worse on two MVSA datasets, demonstrating that BERT is more effective in fusing multimodal multi-domain features than simple networks like MLP.