Research Review of Recurrent Neural Networks

doi:10.13438/j.cnki.jdzk.2021.01.007

Abstract

Abstract: Recurrent neural network (RNN) is a kind of neural network with feedback connection in each layer. Because of its storage characteristics, it can process the sequence data which is related before and after input, and can be widely used in the field of text audio, video and so on. But when the input gap is large, RNN has a short-term memory problem, which can not process long input sequences, while long short-term memory (LSTM) can deal with the long-term dependence problem well. Almost all the exciting results based on RNNs have been realized by LSTM since LSTM was proposed, so LSTM has become the focus of deep learning. This review firstly introduces the basic working principle and characteristics of RNN, and then it introduces the principle and characteristics of LSTM and its variants, as well as the application of RNN and LSTM in various fields. Finally, the future research direction of RNN is proposed.

Key words: recurrent neural network, long short-term memory, sequential data, natural language processing

WANG Yuyan, LIAO Bolin, PENG Chen, LI Jun, YIN Yumin. Research Review of Recurrent Neural Networks[J]. Journal of Jishou University(Natural Sciences Edition), 2021, 42(1): 41-48.

References

[1] 任伟建,宋月,陈建玲,等.基于改进的卷积神经网络的钢号识别[J].吉林大学学报(信息科学版),2018,36(3):325-332.
[2] LIPTON ZACHARY C, BERKOWITZ JOHN, ELKAN CHARLES. A Critical Review of Recurrent Neural Networks for Sequence Learning[J]. Computer Science, 2015, 215: 15-25.
[3] ELMAN JEFFREY L. Finding Structure in Time[J]. Cognitive Science, 1990, 14(2): 179-211.
[4] JORDAN MICHAEL. AttractorDynamics and Parallelism in a Connectionist Sequential Machine[C]. Proceedings of the Eighth Annual Conference of the Cognitive Science Society. Piscataway, NJ: IEEE, 1986, 41: 531-546.
[5] CHEN TUNG-BO, SOO VON-WUN. A Comparative Study of Recurrent Neural Network Architectures on Learning Temporal Sequences[C]. Proceedings of IEEE International Conference on Neural Networks. Washington, DC, USA: IEEE, 1996: 1945-1950.
[6] TER BRANKO. Selective Recurrent Neural Network[J]. Neural Processing Letters, 2013, 38(1): 1-15.
[7] HOCHREITER SEPP, SCHMIDHUBER JRGEN. Long Short-Term Memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[8] SCHUSTER MIKE, PALIWAL KULDIP K. Bidirectional Recurrent Neural Networks[J]. IEEE Transactions on Signal Processing, 1997, 45(11): 2673-2681.
[9] CHO KYUNGHYUN, VAN MERRINBOER BART, GULCEHRE CAGLAR, et al. Learning Phrase RepresentationsUsing RNN Encoder-Decoder for Statistical Machine Translation[C]. Proceedings of the Empiricial Methods in Natural Language Processing. Doha, Qatar: Association for Computational Linguistics, 2014: 1179.
[10] WEISS GAIL, GOLDBERG YOAV, YAHAV ERAN. On the Practical Computational Power of Finite Precision RNNs for Language Recognition[J]. Association for Computational Linguistics, 2018, 2: 740-745.
[11] BRITZ DENNY, GOLDIE ANNA, LE QUOC, et al. Massive Exploration of Neural Machine Translation Architectures[EB/OL]. (2017-03-21)[2020-08-25] . https:∥arxiv.org/abs/1703.03906.
[12] CHUNG JUNYOUNG, GULCEHRE CAGLAR, CHO KYUNGHYUN, et al. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling[EB/OL]. (2014-12-11)[2020-08-25] . https:∥arxiv.org/abs/1412.3555.
[13] ZHOU Chunting, SUN Chonglin, LIU Zhiyuan, et al. A C-LSTM Neural Network for Text Classification[J]. Computer Science, 2015, 1(4): 39-44.
[14] HECK JOEL C, SALEM FATHI M. Simplified Minimal GatedUnit Variations for Recurrent Neural Networks[C]. Proceedings of the IEEE International Midwest Symposium on Circuits and Systems. Boston, MA, USA: IEEE, 2017: 1593-1596.
[15] RAHMAN LAMIA, MOHAMMED NABEEL, AZAD ABUL KALAM AI. A New LSTM Model by Introducing Biological Cell State[C]. Proceedings of the 3rd International Conference on Electrical Engineering and Information Communication Technology. Dhaka, Bangladesh: IEEE, 2016: 1-6.
[16] PULVER ANDREW, LYU SIWEI. LSTM with Working Memory[C]. Proceedings of the International Joint Conference on Neural Networks. Piscataway, NJ: IEEE, 2017: 845-851.
[17] 刘礼文,俞弦.循环神经网络(RNN)及应用研究[J].科技视界,2019,32:54-55.
[18] GRAVES ALEX, FERNNDEZ SANTIAGO, GOMEZ FAUSTINO, et al. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks[C]. Machine Learning, Proceedings of the Twenty-Third International Conference. Pittsburgh, Pennsylvania, USA: International Machine Learning Society, 2006: 369-376.
[19] GRAVES ALEX, MOHAMEDABDEL-RAHMAN, HINTON GEOFFREY. Speech Recognition with Deep Recurrent Neural Networks[C]. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, New York, USA: IEEE, 2013: 6645-6649 .
[20] GRAVES ALEX, NAVDEEP JAITLY. Towards End-To-End Speech Recognition with Recurrent Neural Networks[C]. Proceedings of the 31st International Conference on Machine Learning. Beijing, China, JMLR: W&CP, 2014: 1764-1772.
[21] AMODEI DARIO, ANUBHAI RISHITA, BATTENBERG ERIC, et al. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin[J]. Computer Science, 2015, 23: 216-229.
[22] BAHDANAU DZMITRY, CHO KYUNGHYUN, BENGIO YOSHUA. Neural Machine TranslationBy Jointly Learning to Align and Translate[EB/OL]. (2016-05-19)[2020-09-25] . https:∥arxiv.org/abs/1409.0473.
[23] ZHANG Wen, FENG Yang, MENG Fandong, et al. Bridging the GapBetween Training and Inference for Neural Machine Translation[C]. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence. Yokohama, Japan: International Joint Conferences on Artificial Intelligence, 2020: 659.
[24] LAI Siwei, XU Liheng, LIU Kang, et al. Recurrent Convolutional Neural Networks for Text Classification[J]. Association for the Advance of Artificial Intelligence, 2015, 333: 2267-2273.
[25] YANG Zichao, YANG Diyi, DYER CHRIS, et al. Hierarchical Attention Networks for Document Classification[C]. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California, USA: Association for Computational Linguistics, 2016: 1480-1489.
[26] HE Pan, HUANG Weilin, QIAO Yu, et al. Reading Scene Text in Deep Convolutional Sequences[J]. Association for the Advance of Artificial Intelligence, 2015, 16: 3501-3508.
[27] DONAHUE JEFF, HENDRICKS LISA ANNE, ROHRBACH MARCUS, et al. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2017: 677-691.
[28] KARPATHY ANDREJ, LI Feifei. Deep Visual-Semantic Alignments for Generating Image Descriptions[J]. IEEE Transactions on Pattern Analysisand Machine Intelligence, 2016, 23(2): 664-676.
[29] LEE BYUNGHAN, LEE TAEHOON, NA BYUNGGOOK, et al. DNA-Level Splice Junction PredictionUsing Deep Recurrent Neural Networks[J]. Computer Science, 2015, 32(9): 69-81.
[30] PARK SEUNGHYUN, MIN SEONWOO, CHOI HYUN-SOO, et al. DeepMiRGene: Deep Neural Network Based Precursor MicroRNA Prediction[EB/OL]. (2016-04-29)[2020-08-26] . https:∥arxiv.org/abs/1605.00017.
[31] SNDERBY SREN KAAE, SNDERBY CASPER KAAE, NIELSEN HENRIK, et al. Convolutional LSTM Networks for Subcellular Localization of Proteins[C]. Proceedings of the International Conference on Algorithms for Computational Biology. Cham, San Francisco, USA: Springer, 2016: 68-80.
[32] KORDMAHALLEH MINA MORADI, SEFIDMAZGI MOHAMMAD GORJI, HOMAIFAR ABDOLLAH. A Sparse Recurrent Neural Network for Trajectory Prediction of Atlantic Hurricanes[C]. Proceedings of the Genetic and Evolutionary Computation Conference 2016. New York, United States: Association for Computing Machinery, 2016: 957-964.
[33] FANG Kuai, SHEN Chaopeng, KIFER DANIEL, et al. Prolongation of SMAP to Spatio-Temporally Seamless Coverage of Continental US Using a Deep Learning Neural Network[J]. Geophysical Research Letters, 2017, 44: 1-12.
[34] 向秋红,廖柏林,马川.新型递归神经网络求解时变矩阵Moore-Penrose逆[J].吉首大学学报(自然科学版),2017,38(3):31-35.
[35] 张永胜,肖林.二次最小化问题的有限时间递归神经网络求解[J].吉首大学学报(自然科学版),2019,40(2):21-26.