ARTificial intelligence raters. Neural networks for rating pictorial expression
Main Article Content
Abstract
Previous studies on classification of fine art show that features of paintings can be captured and categorized using machine learning approaches. This progress can also benefit art psychology by facilitating data collection on artworks without the need to recruit experts as raters. In this study a machine learning approach is used to predict the ratings of RizbA, a Rating instrument for two-dimensional pictorial works. Based on a pre-trained model, the algorithm was fine-tuned via transfer learning on 886 pictorial works by contemporary professional artists and non-professionals. As quality criterion, artificial intelligence raters (ART) are compared with generic raters (GR) created from the real human expert raters, using error rate and mean squared error (MSE). ART ratings have been found to have the same error range as randomly chosen human ratings. Therefore, they can be seen as equivalent to real human expert raters for almost all items in RizbA. Further training with more data will close the gap to the human raters on all items.
Downloads
References
Ali, M. (2014). Towards a decolonial computing. Ambiguous Technologies: Philosophical Issues, Practical Solutions, Human Nature, International Society of Ethics and Information Technology, 28-35.
Amirshahi, S. A., Hayn-Leichsenring, G. U., Denzler, J., & Redies, C. (2014). Jenaesthetics subjective dataset: analyzing paintings by subjective scores. European Conference on Computer Vision.
Gengenbach, T., & Schoch, K. (2021, August 23). ARTificial Intelligence Raters: Neural Networks for Rating Pictorial Expression. Open Science Framework. https://doi.org/10.17605/OSF.IO/QUDHX
Arnheim, R. (2000). Kunst und Sehen: Eine Psychologie des schöpferischen Auges [Art and seeing: A psychology of the creative eye] (3 ed.). Walter de Gruyter.
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., & Moody, A. R. (2019). Sample-size determination methodologies for machine learning in medical imaging research: a systematic review. Canadian Association of Radiologists Journal, 70(4), 344-353. https://doi.org/10.1016/j.carj.2019.06.002
Banerji, S., & Sinha, A. (2016). Painting classification using a pre-trained convolutional neural network. International conference on computer vision, graphics, and image processing.
Bauer, H. (1996). Form, Struktur, Stil: Die formanalytischen und formgeschichtlichen Methoden [Form, structure, style: The methods of form analysis and form history]. In H. Belting, H. Dilly, W. Kemp, W. Sauerländer, & M. Warnke (Eds.), Kunstgeschichte: Eine Einführung [Art history: An introduction] (Vol. 6, pp. 157-173). Reimer.
Betensky, M. G. (1991). Kunsttherapie und künstlerische Äußerung aus phänomenologischer Sicht [Art therapy and artistic expression from a phenomenological perspective]. In J. A. Rubin (Ed.), Richtungen und Ansätze der Kunsttherapie: Theorie und Praxis [Approaches to art therapy: Theory and technique] (pp. 176-183). Gerardi.
Brachmann, A., Barth, E., & Redies, C. (2017). Using CNN features to better understand what makes visual artworks special. Frontiers in Psychology, 8, 830. https://doi.org/10.3389/fpsyg.2017.00830
Castro, L., Perez, R., Santos, A., & Carballal, A. (2014). Authorship and aesthetics experiments: comparison of results between human and computational systems. International Conference on Evolutionary and Biologically Inspired Music and Art.
Cattaneo, M. (1994). Addressing culture and values in the training of art therapists. Art Therapy, 11(3), 184-186. https://doi.org/10.1080/07421656.1994.10759081
Cetinic, E., Lipic, T., & Grgic, S. (2018). Fine-tuning convolutional neural networks for fine art classification. Expert Systems with Applications, 114, 107-118. https://doi.org/10.1016/j.eswa.2018.07.026
Cetinic, E., Lipic, T., & Grgic, S. (2020). Learning the principles of art history with convolutional neural networks. Pattern Recognition Letters, 129, 56-62. https://doi.org/10.1016/j.patrec.2019.11.008
Chollet, F. (2018). Deep Learning mit Python und Keras: Das Praxis-Handbuch vom Entwickler der Keras-Bibliothek [Deep Learning with Python and Keras: The Practice Manual from the Developer of the Keras Library]. Manning.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press. https://doi.org/10.4258/hir.2016.22.4.351
Grant, C., & Price, D. (2020). Decolonizing art history. Art History, 43(1), 8-66. https://doi.org/10.1111/1467-8365.12490
Hua, K.-L., Ho, T.-T., Jangtjik, K.-A., Chen, Y.-J., & Yeh, M.-C. (2020). Artist-based painting classification using Markov random fields with convolution neural network. Multimedia Tools and Applications, 79(17), 12635-12658. https://doi.org/10.1007/s11042-019-08547-4
ImageNet. (2012). Large Scale Visual Recognition Challenge (ILSVRC). http://image-net.org/challenges/LSVRC/2012/index
ImageNet. (2016). Large Scale Visual Recognition Challenge (ILSVRC). http://image-net.org/challenges/LSVRC/2016
Kandinsky, W. (1955). Punkt und Linie zu Fläche: Beitrag zur Analyse der malerischen Elemente [Point and Line to Plane: Contribution to the analysis of pictorial elements]. Benteli.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 1097-1105.
Madhu, P., Kosti, R., Mührenberg, L., Bell, P., Maier, A., & Christlein, V. (2019). Recognizing characters in art history using deep learning. Proceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents (SUMAC ’19).
Mersch, D. (2019). Ästhetisches Denken: Kunst als Theoria [Aesthetic thinking: Art as theoria]. In D. Mersch, S. Sasse, & S. Zanetti (Eds.), Ästhetische Theorie [Aesthetic theory] (pp. 241-259). Diaphanes.
Meyer, G. (2011). Sprache der Bilder: Kunst verstehen: Form, Farbe, Komposition [Language of the pictures: Understanding art: Shape, color, composition]. Seemann.
Mosquera, G. (1992). The Marco Polo syndrome: Some problems around art and Eurocentrism. Third Text, 6(21), 35-41. https://doi.org/10.1080/09528829208576382
Nishikawa-Pacher, A., Heck, T., & Schoch, K. (2021). Open editors: A dataset of scholarly journals’ editorial board positions. SocArXiv. https://doi.org/https://doi.org/10.31235/osf.io/jvzq7
Panofsky, E. (2006). Ikonographie und Ikonologie: Bildinterpretation nach dem Dreistufenmodell [Iconography and iconology: Image interpretation according to the three-stage model]. DuMont.
prometheus. (2021). prometheus: Das verteilte digitale Bildarchiv für Forschung & Lehre [prometheus: the distributed digital image archive for research & education] https://www.prometheus-bildarchiv.de
Python. (2021). Python. https://www.python.org
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: An astounding baseline for recognition. Proceedings of the IEEE conference on computer vision and pattern recognition workshops.
Rodriguez, C. S., Lech, M., & Pirogova, E. (2018). Classification of style in fine-art paintings using transfer learning and weighted image patches. 2018 12th International Conference on Signal Processing and Communication Systems (ICSPCS).
Santos, I., Castro, L., Rodriguez-Fernandez, N., Torrente-Patino, A., & Carballal, A. (2021). Artificial neural networks and deep learning in the visual arts: A review. Neural Computing and Applications, 33(1), 121-157. https://doi-org/10.1007/s00521-020-05565-4
Sarkar, D., Bali, R., & Ghosh, T. (2018). Hands-On Transfer Learning with Python: Implement advanced deep learning and neural network models using TensorFlow and Keras. Packt Publishing Ltd.
Schoch, K. (2020, April 24). Ratinginstrument für zweidimensionale bildnerische Arbeiten (RizbA): Fragebogen mit Erläuterungen in deutscher Sprache [Rating instrument for two-dimensional pictorial works (RizbA): Questionnaire with explanatory notes in German language]. Zenodo. https://doi.org/10.5281/zenodo.3765221
Schoch, K., Gruber, H., & Ostermann, T. (2017). Measuring art: Methodical development of a quantitative rating instrument measuring pictorial expression (RizbA). The Arts in Psychotherapy, 55, 73-79. https://doi.org/10.1016/j.aip.2017.04.014
Schoch, K., & Ostermann, T. (2020). Giving the art greater weight in art psychology: RizbA, a quantitative questionnaire for formal picture analysis. Creativity. Theories – Research – Applications, 7(2), 373-410. https://doi.org/10.2478/ctra-2020-0019
Schoch, K., & Ostermann, T. (2021, July 13). Empirics vs. art theory: Exploring a factor structure of pictorial expression based on contemporary artworks. SocArXiv. https://doi.org/https://osf.io/preprints/socarxiv/xyrf5
Schoch, K., & Ostermann, T. (2022). Psychometrics of art: Validation of RizbA, a quantitative rating instrument for pictorial expression. Creativity research journal. https://doi.org/10.1080/10400419.2021.1987734
Simon, H., & Verstegen, U. (2004). prometheus: Das verteilte digitale Bildarchiv für Forschung und Lehre. Neuartige Werkzeuge zur Bereitstellung von verteiltem Content für Wissenschaft und Forschung [prometheus: Digital image database for research and education. New tools for providing shared content for science and research]. Historical Social Research/Historische Sozialforschung, 247-257.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ArXiv. https://doi.org/arXiv:1409.1556
Streb, J. H. (1984). Thoughts on phenomenology, education, and art. Studies in Art Education, 25(3), 159-166. https://doi.org/10.1080/00393541.1984.11650370
Stuhler-Bauer, A., & Elbing, U. (2003). Die phänomenologische Bilderfassung: Ein kunsttherapeutisches Instrument [The phenomenological picture survey: An art therapeutic instrument]. Musik-, Tanz- und Kunsttherapie, 14, 32-46. https://doi.org/10.1026//0933-6885.14.1.32
Stuhler-Bauer, A., & Elbing, U. (2004). Kritik standardisierter empirischer Bilderfassung aus kunsttherapeutisch-phänomenologischer Sicht: Die Nürtinger Beurteilungsskala und die Diagnostic Assessment of Psychiatric Art, deutsche Version [Review on standardized empirical picture analysis from an art therapeutic phenomenological view: The Nürtingen Rating Scale and the Diagnostic Assessment of Psychiatric Art, German version]. Musik-, Tanz- und Kunsttherapie, 15(1), 5-15. https://doi.org/10.1026/0933-6885.15.1.5
Tan, W. R., Chan, C. S., Aguirre, H. E., & Tanaka, K. (2016). Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification. 2016 IEEE international conference on image processing (ICIP).
Uzelac, M. (1998). Art and phenomenology in Edmund Husserl. Axiomathes, 9(1), 7-26.
Vollmar, K. (2008). Die faszinierende Welt der Farben: ein Glossar von A-Z [The fascinating world of colors: A glossary of A-Z]. ars momentum.
WikiArt. (2020). WikiArt API. https://www.wikiart.org/en/App/GetApi
WikiArt. (2021). WikiArt: Visual art encyclopedia. https://www.wikiart.org
Zhao, L., Shang, M., Gao, F., Li, R., Huang, F., & Yu, J. (2020). Representation learning of image composition for aesthetic prediction. Computer Vision and Image Understanding, 199, 103024. https://doi.org/10.1016/j.cviu.2020.103024