髙瀬悠太, 春日宥一郎, 大野正樹, 橋本泰一(RevComm)

本研究ではオンライン会議動画に対して、動画像と音声を用いた話者分離を行う。音声のみの話者分離では、ノイズやオーバーラップが多いオンライン会議動画における話者交代検出の精度の低さが課題である。オンライン会議動画では発話に応じて画面上の話者の映像も切り替わるため、発話区間に対応した話者の顔画像を活用することが可能である。本論文では、発話区間に対して顔検出を一秒ごとに行い、フレーム間の顔画像の類似度に着目し、話者交代検出の改良を試みた。オンライン会議動画199本からなるデータセットに対して評価を行い、平均Diarization Error Rateを29.1%から19.6%に改善することができた。

Jennifer Santoso, Kenkichi Ishizuka, Taiichi Hashimoto(RevComm)

The remarkable emergence of large language models (LLM) and their vast capabilities have opened a possibility for applications in various fields, including speech emotion recognition (SER). Despite the advancement of SER methods and the abundance of speech data, the requirement of having speech data labeled with emotions is a challenge to fulfill, considering the cost of human annotation. In this study, we propose utilizing LLM to annotate emotional speeches, investigating the use of conversation sequence transcription, and incorporating the textual acoustic feature descriptors into the prompt. Furthermore, we also examine the application of annotation results on emotional speeches as training and augmentation data. Our experiment using the IEMOCAP dataset shows that emotional speech annotation using LLMs can outperform human annotation with possibly lower annotation costs. The SER trained using the annotation result as a whole training data or augmentation data reaches a performance close to state-of-the-art SER methods.

Natsumi Kobayashi, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm)

Yurina Machida, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm)

小林夏美, 山田武志(筑波大学), 石塚賢吉, 橋本泰一(RevComm

音声感情認識（Speech Emotion Recognition: SER）は、音声から喜びや悲しみなどの感情を認識する技術で、応答生成や心理カウンセリングなど幅広い応用が期待されています。しかし、感情音声データの不足が課題となっており、自然な感情発話の収録やラベル付けは高コストです。これに対し、データ拡張が低コストで有効な手法として注目されています。本研究では、従来の音響特徴量だけでなく、発話内容テキストのバリエーションを増やすため、変分オートエンコーダ（VAE）を用いた音響およびテキスト特徴量のデータ拡張手法を提案しました。IEMOCAPデータセットを用いた実験で、その有効性を検証しました。

町田裕璃奈(筑波大) , 山田武志(筑波大) , 石塚賢吉(RevComm) , 橋本泰一(RevComm)

音声感情認識は、音声に含まれる感情を推定する技術で、近年は音響特徴だけでなく発話内容のテキストを活用する手法が注目されています。しかし、マイナー言語では感情ラベル付き音声データや書き起こしテキストが不足しているため、リソースが豊富なメジャー言語のモデルをマイナー言語へドメイン適応する方法が提案されています。この適応では言語の差異が課題であり、テキスト埋め込み空間の共通化が必要です。本研究では、機械翻訳や多言語BERTを用いて埋め込み空間を統一する手法を提案し、7言語の感情音声コーパスを用いた実験でその有効性を検証しました。

袁 培傑, 大野 正樹, 橋本 泰一(RevComm)

Large language models (LLMs), such as ChatGPT, have risen to prominence in text summarization tasks, primarily due to the advent of in-context learning. This paper delves into how in-context learning steers the outputs of LLMs based on different data demonstration configurations. Our pivotal findings reveal that ChatGPT's adaptability to target summarization tasks is enhanced when provided with paired text and summaries compared to when provided in isolation. Furthermore, the structured presentation of these pairs proves more influential than their precise content alignment. However, there are observable limitations: increasing the number of demonstrations yields diminishing returns, and the improvement of adaptability declines when tasked with more intricate news texts as opposed to simpler dialogues. This study comprehensively explains in-context learning's nuances in text summarization, highlighting its merits and demerits for future researchers.

Shuhei Kato, Taiichi Hashimoto

Speech-to-face conversion is the task of generating face images from speech signals. Many studies have been conducted to address this task, and achieved good performances. In this paper, we introduce denoising diffusion probabilistic models (DDPMs) to generate face images instead of generative adversarial networks (GANs) or autoencoders, which are used in most of the prior studies. Moreover, unlike prior studies, several components of our system are designed to use high-resolution face image datasets instead of audio-visual paired data. As a result, our system can generate high-resolution face images from speech signals with an architecture that is simpler and more flexible than the ones used in prior studies. In addition, introducing DDPMs enables us to utilize techniques that control out- puts of DDPMs or improve performance of them in succeeding studies.

加藤 集平，橋本 泰一(RevComm

音声は年齢や性別などの情報を伝えていることが知られている。さらに言えば，人間は音声からその話し手の顔をある程度想像することができるが，2019年頃から機械学習の技術を用いて同様のタスク (speech-to-face conversion) を行わせようという試みがなされている。これらの試みの多くは敵対的生成ネットワーク (generative adversarial network; GAN) を用いて顔画像を生成しているが，本稿ではGANに代えて拡散確率モデル (diffusion probabilistic model; DPM) を導入し，顔画像を生成する。拡散確率モデルは潜在変数モデルの一つで，ガウシアンノイズからノイズを有限回数だけ繰り返し除去することにより目的の分布に合致したサンプルを得るモデルであり，画像生成の分野で近年state-of-the-art の性能を達成している。さらに，条件付けモデルを訓練するための簡素で柔軟な手法も存 在する。本稿では，Ohらの手法をベースに拡散確率モデルを導入し，さらにそれを簡素化したシステムとあわせて性能を評価する。

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

Speech emotion recognition (SER) has been actively studied in the recent decade and has achieved promising results. Most state-of-the-art SER methods are based on a classification approach that ultimately outputs the softmax probability over different emotion classes. On the other hand, we have recently introduced an anomalous sound detection approach to improve the SER performance of the neutral class. It uses a neutral speech detector consisting of an autoencoder that reconstructs acoustic and text features in latent space and is trained using only neutral speech data. The experimental result confirmed that the reconstruction error could be successfully used as a cue to decide whether or not the class is neutral and suggested that it could be applied to other emotion classes. In this paper, we propose an SER method based on the reconstruction of acoustic and text features in latent space, in which the reconstructor for different emotion classes, including the neutral class, is used. The proposed method selects the emotion class with the lowest normalized reconstruction error as the SER result. Unlike the classifier approach, one reconstructor is dedicated to each emotion class and trained using only the data of the target emotion class. Therefore, the reconstructor can be trained without being affected by imbalanced training data and also facilitates the application of data augmentation to only a specific emotion class. Our experimental result obtained using the IEMOCAP dataset showed that the proposed method improved the class-average weighted accuracy by 1.7% to 77.8% compared with the stateof-the-art SER methods.

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

Speech emotion recognition (SER) is essential for understanding a speaker’s intention. Recently, some groups have attempted to improve SER performance using a bidirectional long short-term memory (BLSTM) to extract features from speech sequences and a self-attention mechanism to focus on the important parts of the speech sequences. SER also benefits from combining the information in speech with text, which can be accomplished automatically using an automatic speech recognizer (ASR), further improving its performance. However, ASR performance deteriorates in the presence of emotion in speech. Although there is a method to improve ASR performance in the presence of emotional speech, it requires the fine-tuning of ASR, which has a high computational cost and leads to the loss of cues important for determining the presence of emotion in speech segments, which can be helpful in SER. To solve these problems, we propose a BLSTM-and-self-attention-based SER method using self-attention weight correction (SAWC) with confidence measures. This method is applied to acoustic and text feature extractors in SER to adjust the importance weights of speech segments and words with a high possibility of ASR error. Our proposed SAWC reduces the importance of words with speech recognition error in the text feature while emphasizing the importance of speech segments containing these words in acoustic features. Our experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset reveal that our proposed method achieves a weighted average accuracy of 76.6%, outperforming other state-of-the-art methods. Furthermore, we investigated the behavior of our proposed SAWC in each of the feature extractors.

Jennifer Santoso, Takeshi Yamada(Univ. of Tsukuba), Kenkichi Ishizuka, Taiichi Hashimoto(RevComm), Shoji Makino(Waseda Univ./Univ. of Tsukuba)

In recent years, classification-based speech emotion recognition (SER) methods have achieved high overall performance. However, these methods tend to have lower performance for neutral speeches, which account for a large proportion in most practical situations. To solve the problem and improve the SER performance, we propose a neutral speech detector (NSD) based on the anomaly detection approach, which uses an autoencoder, the intermediate layer output of a pretrained SER classifier and only neutral data for training. The intermediate layer output of a pretrained SER classifier enables the reconstruction of both acoustic and text features, which are optimized for SER tasks. We then propose the combination of the SER classifier and the NSD used as a screening mechanism for correcting the class probability of the incorrectly recognized neutral speeches. Results of our experiment using the IEMOCAP dataset indicate that the NSD can reconstruct both the acoustic and textual features, achieving a satisfactory performance for use as a reliable screening method. Furthermore, we evaluated the performance of our proposed screening mechanism, and our experiments show significant improvement of 12.9% in the F-score of the neutral class to 80.3%, and 8.4% in the class-average weighted accuracy to 84.5% compared with state-of-the-art SER classifiers.

Jumon Nozaki, Tatsuya Kawahara(Kyoto Univ.), Kenkichi Ishizuka, Taiichi Hashimoto(Revcomm)

Conventional automatic speech recognition systems do not produce punctuation marks which are important for the readability of the speech recognition results. They are also needed for subsequent natural language processing tasks such as machine translation. There have been a lot of works on punctuation prediction models that insert punctuation marks into speech recognition results as post-processing. However, these studies do not utilize acoustic information for punctuation prediction and are directly affected by speech recognition errors. In this study, we propose an end-to-end model that takes speech as input and outputs punctuated texts. This model is expected to predict punctuation robustly against speech recognition errors while using acoustic information. We also propose to incorporate an auxiliary loss to train the model using the output of the intermediate layer and unpunctuated texts. Through experiments, we compare the performance of the proposed model to that of a cascaded system. The proposed model achieves higher punctuation prediction accuracy than the cascaded system without sacrificing the speech recognition error rate. It is also demonstrated that the multi-task learning using the intermediate output against the unpunctuated text is effective. Moreover, the proposed model has only about 1/7th of the parameters compared to the cascaded system.

Heran Zhang, Masato Mimura, Tatsuya Kawahara(Kyoto Univ.), Kenkichi Ishizuka(RevComm)

In this paper, we investigate a speech emotion recognition based on multi-task learning (MTL) considering the style attribute. We also hypothesize the neutral expression, which has the largest number of samples, is not affected by the style, and thus propose a selective MTL method that applies MTL to emotion categories except for the neutral category. Experimental evaluations using the IEMOCAP database and a call center dataset confirm the effect of the combination of the two corpora, MTL, and the proposed selective MTL.

Santoso Jennifer、Yamada Takeshi(Univ. of Tsukuba)、Ishizuka Kenkichi、Hashimoto Taiichi(RevComm)、Makino Shoji(Waseda Univ./Univ. of Tsukuba).

In recent years, there have been several deep-learning-based methods to im- prove the performance of speech emotion recognition (SER). However, in most of these studies, the recognition of neutral speeches, which is the most common type of speech in practical settings, tend to have poor performance. One of the reasons is the wide data distribution for neutral speeches, which is harder to generalize than emotional speeches.
In several practical settings, such as business conversation analysis, most conversations do not contain emotions. Emotional speeches, therefore, are considered an unusual occurrence and might be an indicator of trouble or unanticipated events in the conversation. Therefore, by taking advantage of a large number of neutral speeches available, it is possible to tackle problems in the SER from the anomaly detection approach, where neutral speeches are considered normal and emotional speeches are considered anomalous.