|
Article information
2025 , Volume 30, ¹ 1, p.132-146
Sivak M.A., Timofeev V.S.
Effect of experimental design on feed-forward neural network accuracy
The paper highlights an application of the main ideas of optimal experimental design theory during collection of data for training of an artificial neural network. The main purpose of this work is to demonstrate possible effect of a matter of choice for the experimental design applied to model training process and model accuracy. It is necessary to determine the experimental design that is the closest to the optimal one. For this purpose, Q-optimality criterion is used. This criterion assumes using neural network response function which is introduced in this research. The study describes a process of initial dataset generation based on an experimental design that consists of two spectrum points. With the help of the obtained datasets, a binary classification pro blem is sorted out. To solve the problem a feed-forward neural network with one hidden layer is used. The research results revealed that the increase of distance between experimental design spectrum points leads to the increase of neural network accuracy. Also, the effect of data normalization on the model fit process is shown. For some experimental designs, data normalization allows significant speeding up neural network training. In addition, based on the obtained results it is concluded that choosing the experimental design that is the closest to the optimal one can obviously increase the artificial neural network accuracy and reduce its training time
[link to elibrary.ru]
Keywords: artificial neural network, experimental design, Q-optimality criterion
doi: 10.25743/ICT.2025.30.1.012
Author(s):
Sivak Maria Alekseevna
PhD. , Associate Professor
Position: Associate Professor
Office: Novosibirsk State Technical University
Address: 630073, Russia, Novosibirsk, 20, prospekt K. Marksa
E-mail: pepelyaeva@ami.nstu.ru
Timofeev Vladimir Semenovich
Dr. , Associate Professor
Position: Professor
Office: Novosibirsk State Technical University
Address: 630073, Russia, Novosibirsk, Marx avenue, 20
Phone Office: (383) 346-31-72
E-mail: v.timofeev@corp.nstu.ru
SPIN-code: 8627-9120 References:
1. Rosenblatt F. Principles of neurodynamics: perceptrons and the theory of brain mechanisms. Cornell Aeronautical Laboratory. Report No. VG-1196-G-8. Spartan Books. 1962. Available at: https://safari.ethz.ch/digitaltechnik/spring2018/lib/exe/fetch.php?media=neurodynamics1962rosenblatt.pdf (accessed on December 20, 2024).
2. McCulloch W.S., Pitts W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics. 1943; (5):115–133. DOI:10.1007/BF02478259.
3. Werbos P.J. Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. Thesis. Cambridge, MA: Harvard University; 1974: 453.
4. Rumelhart D.E., Hinton G.E., Williams R.J. Learning internal representations by error propagation. Parallel Distributed Processing. Cambridge, MA: MIT Press; 1986; (1):318–362.
5. Galushkin A.I. Sintez mnogosloynykh sistem raspoznavaniya obrazov [Synthesis of multilayer pattern recognition systems]. Moscow: Energiya; 1974: 368. (In Russ.)
6. Bartsev S.I., Ohonin V.A. Adaptivnye seti obrabotki informatsii [Adaptive networks for information processing]. Preprint No. 59B. Krasnoyarsk: Institut Fiziki SO AN SSSR; 1986: 20. (In Russ.)
7. Bartsev S.I., Gilev S.E., Okhonin V.A. Printsip dvoystvennosti v organizatsii adaptivnykh setey obrabotki informatsii. Dinamika khimicheskikh i biologicheskikh system [Duality principle in the organization of the adaptive information processing networks. Dynamics of chemical and biological
systems]. Novosibirsk: Nauka; 1989: 269. (In Russ.)
8. Bishop C.M. Pattern recognition and machine learning. Springer; 2006: 758.
9. Haykin S. Neural networks and learning machines. 3rd ed. Pearson Education; 2009: 906.
10. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences. 1982; 79(8):2554–2558.
11. Kosko B. Competitive adaptive bi-directional associative memories. Proceedings of the IEEE First International Conference on Neural Networks. San Diego; 1987; (2):759–766.
12. Elman J.L. Finding structure in time. Cognitive Science. 1990: 179–211.
13. Jordan M.I. Serial order: a parallel distributed processing approach. Institute for Cognitive Science Report 8604. San Diego: University of California; 1986: 40.
14. Denisov V.I., Timofeev V.S. Estimating the parameters of regression dependences using the Gramm–Charlier approximation. Optoelectronics, Instrumentation and Data Processing. 2008; (44):487–493. DOI:10.3103/S8756699008060010.
15. Rousseeuw P.J. Tutorial to robust statistics. Journal of Chemometrics. 1991; 5(1):1–20. DOI:10.1002/cem.1180050103.
16. Rousseeuw P.J., Leroy A.M. Robust regression and outlier detection. N.Y.: John Wiley & Sons; 1987: 334.
17. Huber J.P. Robust statistics. 2nd ed. N.J.: Wiley; 2009: 370.
18. Timofeev V.S., Sivak M.A. A robust neural network with simple architecture. Journal of Applied and Industrial Mathematics. 2021; 15(4):670–678. DOI:10.1134/S1990478921040104.
19. Sivak M.A., Timofeev V.S. Adjusting robust neural networks for solving the classification problem. Proceedings of Tomsk State University of Control Systems and Radioelectronics. 2021; 24(3):26–32. (In Russ.)
20. Fujimoto S., Meger D., Precup D. An equivalence between loss functions and non uniform sampling in experience replay. 2020. Available at: https://inlnk.ru/poZ0mB (accessed on April 30, 2024).
21. Barron J.T. A general and adaptive robust loss function. 2017. Available at: https://arxiv.org/abs/1701.03077 (accessed on April 30, 2024).
22. Andreou P., Charalambous C., Martzoukos S. Robust artificial neural networks for pricing of european options. Computational Economics. 2006; 2(27):329–351.
23. tf.keras.losses.Huber. Available at: https://www.tensorflow.org/api_docs/python/tf/keras/losses/Huber (accessed on April 30, 2024).
24. Flexa C., Gomes W., Viademonte S. An exploratory study on machine learning frameworks. 2019. Available at: https://www.researchgate.net/publication/338417674_An_exploratory_study_on_machine_learning_frameworks (accessed on April 30, 2024).
25. Fedorov V.V. Theory of optimal experiment design. N.Y.: Academic Press; 1972: 292.
26. Bishop C.M. Neural networks for pattern recognition. N.Y.: Oxford University Press; 1995: 502.
27. Gmurman V.E. Teoriya veroyatnosti i matematicheskaya statistika [Probability and statistical theory]. Moscow: Vysshee Obrazovanie; 2005: 479. (In Russ.)
28. Denisov V.I.Matematicheskoe obespechenie sistemy EVM-eksperimentator (regressionnyy i dispersionnyy analisy [Software for Experimenter system (regression and variance analysis)]. Moscow: Nauka; 1977: 252. (In Russ.)
29. Grigoriev Yu.D. Q-optimal experimental designs and close to them experimental designs for polynomial regression on the interval. Industrial Laboratory. Diagnostics of Materials. 2020; 86(5):65–72. (In Russ.)
30. Sebastiani F. Text categorization. Text mining and its applications, CRM and knowledge management. UK: Southampton; 2005. Available at: https://courses.ischool.berkeley.edu/i256/f06/papers/sebastiani02.pdf (accessed on December 20, 2024).
31. Huang L. Normalization in task-specific applications. Normalization techniques in deep learning.Berlin, Heidelberg: Springer; 2022: 87–100.
Bibliography link:
Sivak M.A., Timofeev V.S. Effect of experimental design on feed-forward neural network accuracy // Computational technologies. 2025. V. 30. ¹ 1. P. 132-146
|
