The Proposition of Three Approaching Ways to Implement Tan-sigmoid Activation Function in FPGA

Ali, Manal Talib; Abed, Bassam H.

doi:10.30684/etj.v40i2.2160

	The Proposition of Three Approaching Ways to Implement Tan-sigmoid Activation Function in FPGA
Engineering and Technology Journal
Article 3, Volume 40, Issue 2, February 2022, Pages 311-321 PDF (892.76 K)
DOI: 10.30684/etj.v40i2.2160
Authors
Manal Talib Ali^* ; Bassam H. Abed
Electrical Engineering Dept., University of Technology-Iraq, Alsina’a street, 10066 Baghdad, Iraq.
Abstract
Hyperbolic tangents and Sigmoid are commonly used for Artificial Neural Networks as activation functions. The complex equation of the activation function is one of the most difficult to be implemented in hardware because containing division and exponential, which gives non-linear behavior. The challenge is building a tan-sigmoid function in hardware with efficient performance. Therefore, this work will focus on implementing the activation function in FPGA. To overcome this challenge, a different approach was proposed in this paper, efficient hardware-implemented for tan-sigmoid in terms of the number of slices occupied and the resources utilization are designed. In this work, three approaches are proposed: tan-sigmoid using the log approximation method, tan-sigmoid using segmentation method, and tan-sigmoid using the polynomial method. These approaches are efficiently implemented in the Xilinx Spartan-3A xc3s700a-4fg484 platform. Hardware synthesis and FPGA implementations illustrate that the proposed tan-sigmoid only takes up to 1% of logic resources in the first and second proposed approaches. While, 4% showed in the third proposed approach, with the best efficiency and significantly confirmed the lowest implementation costs than the traditional approach.
Highlights
The challenge is building a tan-sigmoid function in hardware with efficient performance. Proposed three approaches: tan-sigmoid using the log approximation, the segmentation, and the polynomial methods. These approaches are efficiently implemented in the Xilinx Spartan-3A xc3s700a-4fg484 platform. The resource utilization is about (1%) for slices and LUT.
Keywords
neural network; Field Programmable Gate Array (FPGA); Activation function; Tan- sigmoid

References
[1] H. F. Ridzuan, A. Ja’afar, A. K. Amali, A. J. S. H.-F. Syam, and J. Yulni, MLP Based Tan-Sigmoid Activation Function for Cardiac Activity Monitoring, MATEC Web Conf., 255 (2019) 3005. [2] L. Zhang, Artificial neural network model-based design and fixed-point fpga implementation of hénon map chaotic system for brain research, 2017 IEEE XXIV Int. Conf. Electron. Electr. Eng. Comput., (2017) 1–4. [3] M. Alas and S. I. A. Ali, Prediction of the high-temperature performance of a geopolymer modified asphalt binder using artificial neural networks, Int. J. Technol, 102 (2019) 417–427. [4] E. Srinivasan and S. Himavathi, Neural network implementation using FPGA: issues and application, Int. J. Inf. Technol., 2 (2008) 86–92. [5] S. R. Chiluveru and M. Tripathy, Non‐linear activation function approximation using a REMEZ algorithm, IET Circuits, Devices Syst. [6] A. Savran and S. Ünsal, Hardware implementation of a feed-forward neural network using fpgas, third Int. Conf. Electr. Electron. Eng. (ELECO 2003) (2003) 3–7. [7] K. M. Hamdia, X. Zhuang, and T. Rabczuk, An efficient optimization approach for designing machine learning models based on genetic algorithm, Neural Comput. Appl., 336 (2021) 1923–1933. [8] R. G. Biradar, A. Chatterjee, P. Mishra, and K. George, FPGA implementation of a multi-layer Artificial Neural Network using System-on-Chip design methodology,2015 Int. Conf. Cogn. Comput. Inf. Process., (2015) 1–6. [9] S. Himavathi, D. Anitha, and E. Srinivasan, Feedforward neural network implementation in FPGA using layer multiplexing for effective resource utilization, IEEE Trans. Neural Networks, 183 (2007) 880–888. [10] A. Tisan and J. Chin, An end-user platform for FPGA-based design and rapid prototyping of feed-forward artificial neural networks with on-chip backpropagation learning, IEEE Trans. Ind. Informatics, 123 (2016) 1124–1133. [11] Alçın, İ. Pehlivan, and İ. Koyuncu, Hardware design and implementation of a novel ANN-based chaotic generator in FPGA, Optik (Stuttg), 13 (2016) 5500–5505. [12] S. Mitra and P. Chattopadhyay, Challenges in implementation of ANN in embedded system, 2016 Int. Conf. Electr. Electron. Optim. Tech., (2016) 1794–1798. [13] S. Ngah, R. A. Bakar, A. Embong, and S. Razali, Two-steps implementation of sigmoid function for artificial neural network in Field Programmable Gate Array, ARPN J. Eng. Appl. Sci, 7 (2016) 4882–4888. [14] P. Ramachandran, B. Zoph, and Q. V Le, Searching for activation functions, arXiv Prepr. arXiv1710.05941, (2017). [15] S. Sawaguchi and H. Nishi, Slightly-slacked dropout for improving neural network learning on FPGA, ICT Express, 42 (2012) 75–80. [16] D. Misra, Mish: A self regularized non-monotonic neural activation function, arXiv Prepr. arXiv1908.08681, 4 ( 2019). [17] R. Sarić, D. Jokić, N. Beganović, L. G. Pokvić, and A. Badnjević, FPGA-based real-time epileptic seizure classification using Artificial Neural Network, Biomed. Signal Process. Control, 62 (2020) 102106. [18] I. Koyuncu, M. Alcin, P. Erdogmus, and M. Tuna, Artificial Neural Network-Based 4-D Hyper-Chaotic System on Field Programmable Gate Array, Int. J. Intell. Syst. Appl. Eng., 82 (2020) 102–108. [19] N. Zhang, X. Wei, H. Chen, and W. Liu, FPGA implementation for CNN-based optical remote sensing object detection, Electronics, 103 (2021) 282. [20] S. Ngah and R. A. Bakar, Sigmoid function implementation using the unequal segmentation of differential look-up table and second order nonlinear function, J. Telecommun. Electron. Comput. Eng., 9 (2017) 103–108. [21] I. A. D. Williamson, T. W. Hughes, M. Minkov, B. Bartlett, S. Pai, and S. Fan, Reprogrammable electro-optic nonlinear activation functions for optical neural networks, IEEE J. Sel. Top. Quantum Electron., 261 (2019) 1–12. [22] H. M. H. Al-Rikabi, M. A. M. Al-Ja’afari, A. H. Ali, and S. H. Abdulwahed, Generic model implementation of deep neural network activation functions using GWO-optimized SCPWL model on FPGA, Microprocess. Microsyst., 77 (2020) 103141. [23] H. K. Ali and E. Z. Mohammed, Design artificial neural network using FPGA, IJCSNS, 108 (2010) 88. [24] S. Gomar, M. Mirhassani, and M. Ahmadi, Precise digital implementations of hyperbolic tanh and sigmoid function, 2016 50th Asilomar Conf. Signals, Syst. Comput., (2016) 1586–1589. [25] L. Moreira, R. Vettor, and C. Guedes Soares, Neural Network Approach for Predicting Ship Speed and Fuel Consumption, J. Mar. Sci. Eng., 92 (2021) 119. [26] L. Li, S. Zhang, and J. Wu, An efficient hardware architecture for activation function in deep learning processor, 2018 IEEE 3rd Int. Conf. Image, Vis. Comput., (2018) 911–918.
Statistics Article View: 342 PDF Download: 446