Abstract

In this paper, the authors propose a method of customizing automatic speech recognition models using editable term lists to increase recognition accuracy for phrases registered at the discretion of the user. The proposed method adds a text encoder and cross-attention layer to the conventional Connectionist Temporal Classification/Attention-based automatic speech recognition model to detect registered terms in speech signals. To increase registered term recognition accuracy, the authors propose an inference algorithm that integrates the probability of detected registered words and Connectionist Temporal Classification. Results of their evaluation using the in-house Japanese-language dataset and the English-language Librispeech corpus showed that using the proposed method lowered the character error rate and the word error rate for registered terms.

References

1. (1) Prabhavalkar, R., Hori, T., Sainath, T. N., Schlüter, R., Watanabe, S.: End-to-End Speech Recognition: A Survey, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 32, p. 325-351, (2023), doi: 10.1109/TASLP.2023.3328283
2. (2) https://global.honda/jp/stories/003.html, (accessed 2024/9/23) (in Japanese)
3. (3) https://www.jp.honda-ri.com/en/technical-review/activity/honda-ca-system/, (accessed 2024/9/23)
4. (4) Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, ICML ‘06: Proceedings of the 23rd international conference on Machine learning, p. 369-376, (2006), doi: 10.1145/1143844.1143891
5. (5) Peng, Y., Sudo, Y., Shakeel, M., Watanabe, S.: OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Vol. 1, p. 10192–10209, (2024), doi: 10.18653/v1/2024.acl-long.549
6. (6) Graves, A.: Sequence Transduction with Recurrent Neural Networks, International Conference of Machine Learning (ICML) 2012 Workshop on Representation Learning, arXiv preprint, (2012), doi: 10.48550/arXiv.1211.3711
7. (7) Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented Transformer for Speech Recognition, Proc. Interspeech 2020, p. 5036-5040, (2020), doi: 10.21437/Interspeech.2020-3015
8. (8) Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-Based Models for Speech Recognition, Advances in Neural Information Processing Systems, Vol. 28, (2015)
9. (9) Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust Speech Recognition via Large-Scale Weak Supervision, Proceedings of the 40th International Conference on Machine Learning, p. 28492-28518, (2023)
10. (10) Watanabe, S., Hori, T., Kim, S., Hershey, J. R., Hayashi, T.: Hybrid CTC/Attention Architecture for End-to-End Speech Recognition, IEEE Journal of Selected Topics in Signal Processing, Vol. 11, Issue 8, p. 1240-1253, (2017), doi: 10.1109/JSTSP.2017.2763455
11. (11) Sudo, Y., Shakeel, M., Yan, B., Shi, J., Watanabe, S.: 4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders, Proc. INTERSPEECH 2023, p. 3312-3316, (2023), doi: 10.21437/Interspeech.2023-1303
12. (12) Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., Zhao, D.: Deep Context: End-to-end Contextual Speech Recognition, 2018 IEEE Spoken Language Technology Workshop (SLT), (2018), doi: 10.1109/SLT.2018.8639034
13. (13) Huang, K., Zhang, A., Yang, Z., Guo, P., Mu, B., Xu, T., Xie, L.: Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network, Proc. INTERSPEECH 2023, p. 4933-4937, (2023), doi: 10.21437/Interspeech.2023-767
14. (14) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I.: Attention is all you need, NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems, p. 6000-6010, (2017)
15. (15) Shakeel, M., Sudo, Y., Peng, Y., Watanabe, S.: Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss, Proc. Interspeech 2024, p. 3909-3913, (2024), doi: 10.21437/Interspeech.2024-1257
16. (16) Huber, C., Hussain, J., Stüker, S., Waibel, A.: Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition, 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), (2021), doi: 10.1109/ASRU51503.2021.9687898
17. (17) Sudo, Y., Shakeel, M., Fukumoto, Y., Peng, Y., Watanabe, S.: Contextualized Automatic Speech Recognition With Attention-Based Bias Phrase Boosted Beam Search, ICASSP 2024 – 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 10896-10900, (2024), doi: 10.1109/ICASSP48485.2024.10447782
18. (18) Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: An ASR corpus based on public domain audio books, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 5206-5210, (2015), doi: 10.1109/ICASSP.2015.7178964
19. (19) Maekawa, K.: Corpus of Spontaneous Japanese: Its design and evaluation, Proc. ISCA/IEEE Workshop on Spontaneous Speech Processing and Recognition, (2003), https://www.isca-archive.org/sspr_2003/maekawa03_sspr.pdf, (accessed 2024/9/23)
20. (20) Kurematsu, A., Takeda, K., Sagisaka, Y., Katagiri, S., Kuwabara, H., Shikano, K.: ATR Japanese speech database as a tool of speech recognition and synthesis, Speech Communication, Vol. 9, Issue 4, p. 357-363, (1990), doi: 10.1016/0167-6393(90)90011-W
21. (21) Sudo, Y., Hata, K., Nakadai, K.: Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation, Proc. INTERSPEECH 2023, p. 491-495, (2023), doi: 10.21437/Interspeech.2023-1320

Author (organization or company)

• Yui SUDO（Honda Research Institute Japan Co., Ltd.）
• Shakeel MUHAMMAD（Honda Research Institute Japan Co., Ltd.）
• Naoaki SUMIDA（Honda Research Institute Japan Co., Ltd.）

Publication Information

Article of Honda R&D Technical Review Vol.38 (139 - 145)

PDF

Citation Information