Translate this page:
Please select your language to translate the article


You can just close the window to don't translate
Library
Your profile

Back to contents

Software systems and computational methods
Reference:

An information model of the data structure and an experimental technique for improving the human-computer graphical interface

Borevich Ekaterina Vladislavovna

ORCID: 0000-0001-6263-3901

Assistant of the Higher School of Design and Architecture, Peter the Great St. Petersburg Polytechnic University

195251, Russia, Sankt-Peterburg oblast', g. Saint Petersburg, ul. Politekhnicheskaya, 29

plasma5210@mail.ru
Other publications by this author
 

 
Yanchus Viktor Edmundasovich

ORCID: 0000-0001-7220-0819

PhD in Technical Science

Associate Professor, Department of Higher School of Design and Architecture, Peter the Great St. Petersburg Polytechnic University

195251, Russia, Sankt-Peterburg oblast', g. Saint Petersburg, ul. Politekhnicheskaya, 29, aud. V 2.22(2)

victorimop@mail.ru

DOI:

10.7256/2454-0714.2022.1.37730

Received:

22-03-2022


Published:

03-04-2022


Abstract: In the article, the authors describe the developed and tested methodology for conducting a computational experiment to study the effect of color solutions on the visual perception of a video frame by the viewer. The developed technique is designed to assess the subjective emotional reaction of the viewer that occurs at the final stage of the human visual system when perceiving visual information. The article describes methods of preparation of stimulus material, methods of conducting an experiment using a developed software module located on a network resource. The authors have developed a methodology for conducting an experiment with online testing, the information structure of the database, a questionnaire for collecting information, the form of testing by the subjects, algorithms for statistical processing of the results of the experiment. This study is based on the results of a series of experiments conducted using a software and hardware complex for fixing oculomotor activity - an eye tracker, which allows obtaining objective parametric data of a template for viewing stimulus material. The technique described in the article is an addition to the experimental study using the eytracking technology. The results of the experiment should be used in the development of textbooks on color correction, as well as in the design of control interface elements. The methodology is programmatically implemented and tested in the local network of Peter the Great St. Petersburg Polytechnic University. It is planned to finalize the Internet survey module and conduct a global experiment with subsequent statistical processing of the results.


Keywords:

Visual perception, Visual system, Software module, Incentive material, Color scheme, Post-processing, Computational experiment, Information data model, Ranking method, Analysis of variance

This article is automatically translated.

Introduction

The methods of quantum physics describe light as electromagnetic radiation perceived by the human eye. As a consequence, the concept of color does not exist without an observer (human). The perception of color by a person goes beyond physics and goes into the field of physiology and psychology, as it is associated with the activity of the human brain.

There are alternative scientific approaches to the concept of the essence of color. Physicalists consider color to be a physical property, as real as any other [1]. An alternative direction is relational. Colors are relational or "situational" properties [2]. Adverbialism is a form of relativism that attributes color to perceptual processes rather than objects or the perceiving subject. The multifunctionality of color vision fits into this theory [3].

The functions of color are deeper than just "decoration". Thanks to color vision, a person distinguishes the shape of an object faster, determines the distance of the object from the observer faster [4]. The perception of color depends on the conditions of observation. The perception of color is influenced by nearby colors, lighting conditions and the shape of objects[5].

In studies of the physiology of oculomotor activity, various aspects are considered:the presence of a repulsive effect that really attracts attention [6]; effective tracking of only one color at a time by an observer [7]; investigation of the effect of screen brightness on the fatigue of the viewer when solving problems [8]; research on the work of information perception in the field of para-central vision. Within the framework of such studies, an experiment revealing the difference between male and female peripheral color vision is interesting [9]. It has been experimentally proven that there is a deficit of attention in places inaccessible to the eye. However, visual attention is not limited to the oculomotor range [10].

When perceiving a real environment, a person solves problems of identifying surrounding objects by color tone and contrast with other objects. Visual perception is a function of the meaning that we associate – through our own behavior or reasonable assumptions – with the object that we see [11,12]. Zheleznyakov V.N. speaks about the variables of visual perception of a movie frame [13]. To create conditions for immersion in cinema, in addition to artistic tasks, it is necessary to reproduce the real conditions of perception of the surrounding world. This problem is solved by recreating in the frame of the film a perception gradient corresponding to the conditions of perception of the real environment. Currently, the main work with the perceptual gradient in the frame is carried out at the post-processing stage using specialized programs for digital color correction of the video image (color correction). There are color correction algorithms based on mathematics, on the algebra of quaternions [14]. Also, for color correction, methods of recoloring this image or video are used by comparing this image with the reference image [15,16]

The aesthetic appeal of the frame consists of compositional construction and emotional tension arising from its perception. Semantic information of images is a good clue for predicting emotions [17].

The question of the correct change in the conditions of perception of the film frame in order to give the latter a stronger emotional load, according to the director's plan, remains unresolved and requires experimental research.

1. The purpose and objectives of the study

The purpose of the study is to assess the degree of influence of the factor of the color solution of the frame on its perception by the viewer. In accordance with this goal, research tasks were formulated, a number of experiments were prepared and conducted [18, 19]. The experiments were carried out using a device that records eye motor activity (eye tracker) [20], which made it possible to give an objective assessment of the influence of the color solution factor on the parameters of the stimulus study template. material.

Based on the Yuriev color perception model [21] and the results of experiments, a model of the human visual system for image recognition was proposed (Fig.1).

Fig. 1. The work of the human visual system on the perception of visual information

Perception of visual information occurs in three stages:

1.      The first stage consists in mechanical scanning of the image with the eye.

2.      The second stage is the recognition of the image seen.

3. The third stage is the formation of the viewer's emotional response. A person, based on his experience, forms his own impression of the image he saw.

At the first two stages, the dependence of the viewer's perception of the frame on the color solution was evaluated using an eytracker. A statistically significant effect of the color solution was revealed at the stage of pattern recognition.

At the final stage of image perception, the viewer's emotional impression depends not only on the observed object, but also on the person's life experience (education, erudition), as well as on the mood and well-being of the observer. In order to neutralize the influence of indirect factors of perception of the stimulus material, depending on the observer, on the emotional evaluation of the stimulus, it is necessary to expand the circle of subjects. In such experimental conditions, the eytracker turned out to be ineffective for two reasons:

• the need to interview a large number of subjects;

• the need to use a weighting factor of the attractiveness of the frame, which is determined by the subject himself (ranking method)[22].

The purpose of this experiment is to reveal the influence of the color scheme of the frame on the emotional reaction of the observer.

In accordance with this goal , the following tasks were formulated:

• development of an experimental installation;

• development of a software module located on a network resource and allowing to interview subjects without involving them in the laboratory;

• processing of the received data.

The first model of the developed software module turned out to be ineffective. Interest in passing disappeared after completing the task of the experiment by the subjects already on the fifth stimulus. Then the subject worked out a certain system of responses and acted in a template. To maintain interest throughout the experiment, an improved experimental model was developed, implemented through an appropriate software module [23].

2.      Methods of preparation of stimulus material

Frames from films and animated films were selected to compose the visual series, stimulus material. The content of the frames met a number of requirements, such as: the presence of a semantic load and the presence of two centers of interest (Fig.2). For additional stimuli, frames from films or animated films with one center of interest or with an implicit center of interest (types of nature) were used. To set up the experiment, four different color solutions of the film frame were selected in accordance with the theory of color and color contrasts according to I. Itten and a control black-and-white image (Fig. 3) [24].

As a result, two arrays of targeted stimuli were created: an array of photorealistic stimuli and an array of animated stimuli. Each array of targeted stimuli contains 25 stimuli in five color solutions (150 stimuli in total). In addition, two arrays of additional stimuli (photorealism and animation) were created, for stimuli with one center of interest, 50 stimuli in four color solutions (200 stimuli), and for stimuli with an indeterminate center of interest out of 25 stimuli, three color solutions (75 stimuli).

Fig. 2. Selection of a stimulus with two centers of interest. Zone No. 1 is the object that is the center of interest. Zone No. 2 is the object in the center of attention. Zone No. 3 – background

 

Fig. 3. Example of stimulus material (animation frame and video frame in five

color solutions)

The frames were transformed by processing in a graphic editor. The method of preparing stimuli is similar to previous experiments and is described in detail in previous articles: Experimental study of digital color correction models and their influence on the visual fixation of video frames; Investigation of the meaning of the color solution in the process of matching a movie frame [18, 19].

3. Information data model

Figure 4 shows a block diagram of the information model of the computational experiment data. The structure of the information model is based on the methodology of a computational experiment and performs the tasks of storing arrays of stimulus material, personal data of subjects, recording test results, pre-processing the results of a computational experiment.

 

Fig. 4. Block diagram of the information data model

Database tables with a list of attributes for each table:

•         Questionnaire: Personal identifier (PID), full name of the observer, gender (male-female), age (number), region, education (secondary, incomplete higher education, higher bachelor's degree, higher master's degree, academic degree), type of education (technical, humanitarian) artistic training (yes, no);

•         Results: PID, test date, Test execution time (single stimulus test), FK_anketa, FK_stimul, bww (black and white) (rank 1-5), tre (triad) (rank 1-5), dop (optional) (rank 1 to 5), onw (monochrome warm) (rank from 1 to 5), onc (monochrome cold) (rank from 1 to 5), err - the number of attempts to identify one stimulus in 5 color solutions;

•         Stimuli: PID, feature (two centers of interest, one, nature-background, i.e. none), color scheme (bww, tre, dop, onw, onc), media (photo, animation, graphics, text), container, file name;

Color scheme (reference): PID, Color scheme (bww, tre, dop, onw, onc);

Reference education: PID, education (secondary technical, secondary humanitarian, incomplete higher, technical (bachelor's degree), incomplete higher humanitarian (bachelor's degree), higher technical (Master's degree), higher humanitarian (Master's degree), technical degree, humanitarian degree);

Region Directory: PID, region name, part of the world.

 

4.      Methodology of the experiment

When setting up an experiment, the key point is to develop a task for the subjects. On the one hand, the task should not be too difficult to exclude a long preliminary training of the subject before passing the experiment. On the other hand, it is not too simple, so that during the experiment the subject does not lose interest in the task being performed.

To implement the ranking of a targeted series of stimuli, the task for the subject in the experiment was improved. Targeted stimuli (5 stimuli) are placed in a 4*4 matrix in random order (Fig. 5). The remaining fields are filled with additional stimuli selected from the corresponding arrays in random order. The screen size is considered 1600*1200 pixels. Therefore, the size of the stimuli in the matrix will be 400*300 pixels. With a 5*5 stimulus matrix, the size of the stimulus will be too small, which will make it difficult for the subjects to identify the targeted stimuli (Fig. 3). Figure 5 shows an example of the working field of the experiment, the targeted stimuli and additional ones are located in a grid. The figure also shows information about the number of centers of interest for each of the displayed stimuli in the lower left corner.

Fig. 5. Example of a field for ranking target stimuli by the test. The number of centers of interest is indicated in the lower left corner of each stimulus

When a stimulus is activated from an additional database, an error is recorded, which is entered into the experimental results table. Thus, each stimulus is assigned a weighting factor from 1 to 5 and the number of ranking errors of one row of targeted stimuli is fixed.

5.      Statistical processing of experimental results

Table 1 shows the data obtained after passing the test.

Table 1. Example of data after passing the test.

PID

Date

FK_anketa

 

FK_stimul

bww

tre

dop

onw

onc

err

xxx

Dd.mm.yy

Ivanov

1_foto

3

2

1

4

5

0

 

 

Ivanov

-------

 

 

 

 

 

 

 

 

Ivanov

25_foto

5

2

3

1

4

3

 

 

-------

-------

 

 

 

 

 

 

xxx

Dd.mm.yy

Petrov

1_foto

3

2

1

4

5

0

 

 

Petrov

-------

 

 

 

 

 

 

 

 

Petrov

25_foto

5

2

3

1

4

3

 

 

-------

-------

 

 

 

 

 

 

 

Mathematical processing of experimental data was carried out using the method of statistical processing – analysis of variance (ANOVA) [25]. The stages of processing the data obtained as a result of the experiment are reflected in the block diagram (Fig. 6).

Fig. 6. Algorithm of statistical processing of experimental data

At the first stage, the procedure of averaging the weighting coefficient over all homogeneous stimuli (photorealism-animation) is performed for each subject and for each color solution:

                                                                        (1)

where X(i,s,c) is a function for calculating the average values of the weighting coefficient in the corresponding color scheme and in the corresponding frame stylization for one observer; i is the observer's PID; s is stimulus stylization (photorealism or animation);c defines variable color solutions (bww, tre, dop, onw, onc); N with the identifier isck– the value of the weight coefficient for one observer for one stimulus; n– the number of stimuli in each color scheme and stylization of the frame. Since the observer in the experiment has to solve the problem on 25 main photorealistic stimuli and 25 animation stimuli, averaging occurs over 25 stimuli. The type of data is presented in Table 2. The "Mean_err" column indicates the average number of errors when passing the test for one set of targeted stimuli.

Table 2. Example of data after averaging operation.

PID

Date

FK_anketa

 

Steam

Mean_bww

 

Mean_tre

 

Mean_dop

Mean_onw

Mean_onc

Mean_err

xxx

Dd.mm.yy

Ivanov

photo

3,6

2,4

2,8

3,2

3

1,3

xxx

Dd.mm.yy

Petrov

anim

3,7

2.3

2,8

3,4

2,8

1,7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The second stage is a comparison of samples by factors:

• color scheme;

• frame stylization factor (photorealism or animation);

• gender attribute;

• factor of the type of education (technical or humanitarian);

• the factor of artistic training (presence or absence).

The comparison of samples is carried out using single-factor analysis of variance. Risk of making a mistake of the first kind (rejection of the true null hypothesis) corresponds to the p-value. If the p-value is less than the significance level, the null hypothesis is rejected.

The significance level is the acceptable probability of statistical error, in which the differences are considered significant, but in fact they are random. If the analyzed sample has more than 100 values, then it is advisable to take the threshold of rejection of the null hypothesis equal to 0.01. So, if the p-value is less than 0.01, it would be possible to make a decision about the presence of a connection (difference) [26,27].

6.      Conclusions

A method of conducting a computational experiment to study the influence of the color scheme factor on the perception of a video frame by the viewer at the final stage of image perception is proposed. The developed methodology allows to cover a large number of subjects. This factor will make it possible to level out the subjective component in the assessment of the stimulus material by the subjects and make the results of the experiment more reliable.

The developed experimental technique allows us to assess the degree of influence of the color solution on the subjective assessment of the frame by the viewer. The proposed system of arrays with data allows using the developed software module located on a network resource for conducting an experiment. The questionnaire element of the subjects allows us to assess the influence of factors of gender, age and education on the perception of staff. The task of preserving the interest of the subject throughout the experiment is solved by the experimental model using additional frames.

The results of the experiment should be used in the development of textbooks on color correction, as well as in the design of control interface elements.

The methodology is programmatically implemented and tested in the local network of Peter the Great St. Petersburg Polytechnic University. It is planned to finalize the Internet survey module and conduct a global experiment with subsequent statistical processing of the results.

References
1. Allen K.: Colour physicalism, naive realism, and the argument from structure. Minds and Machines 25, 2015. pp. 193-212
2. Cohen, J.: Color relationalism and color phenomenology. Perceiving the World, 2010. pp. 13
3. Chirimuuta M., Kingdom F.: The uses of colour vision: ornamental, practical, and theoretical. Minds and Machines 25(2), 2015. pp. 213-229
4. Chirimuuta M.: Editorial for Minds and Machines special issue on philosophy ofcolour. Minds and Machines 25, 2015. pp. 123-132
5. Kang W., Qin S., Zhang Q.: Computer-aided color aesthetic evaluation methodbased on the combination of form and color. Mathematical Problems in Engineering,2015. pp. 1-8
6. Baumeler D., Nako R., Born S., Eimer M.: Attentional repulsion effects producedby feature-guided shifts of attention. Journal of Vision 20(3),2020.pp. 1-12
7. Liu T., Jigo M.: Limits in feature-based attention to multiple colors. Atten PerceptPsychophys 79, 2017. pp. 2327-2337
8. Benedetto S., Carbone A., Drai-Zerbib V., Pedrotti M., Baccino T.: Effects ofluminance and illuminance on visual fatigue and arousal during digital reading.Computers in Human Behavior 41, 2014. pp. 112-119
9. Murray I., Parry N., McKeefry D., Panorgias A.: Sex-related differences in peripheralhuman color vision: A color matching study. Journal of Vision 12(1), 2012. pp. 1-10
10. Hanning N., Szinte M., Deubel H.: Visual attention is not limited to the oculomotorrange. In: Proceedings of the National Academy of Sciences.Proceedings of the National Academy of Sciences of the United States of America,United States of America. 2019. pp. 9665-9670
11. LesterP. L. Visual Communication: Images with Messages, 6th ed. CengageLearning, Wadsworth series in mass communication and journalism, 2012.466 pages
12. Le Meur O.: Visual attention modelling and applications. Towards perceptualbasedediting methods. Image Processing. University of Rennes 2014. 148 pages
13. Zheleznyakov, V.: Color and contrast. Technology and creative choice. Russian State University of Cinematography (2001)
14. Pie S., Hsiao Y.: Simple effective image and video color correction using quaterniondistance metric. In: IEEE International Conference on Image Processing, ICIP, United States of America 2015. pp. 2920-2924
15. Reinhard E., Ashikhmin M., Gooch B., Shirley P.: Color Transfer between Images.IEEE Computer Graphics and Applications 21, 2001. pp. 34-41
16. Faridul H., Pouli T., Chamaret C., Stauder J., Reinhard E., Kuzovkin D.,Tremeau A.: Colour Mapping: A Review of Recent Methods, Extensions and Applications.Computer Graphics Forum 35, 2016. pp. 59-88
17. Kim H., Kim Y., Kim S., Lee I.: Building emotional machines: recognizing imageemotions through deep neural networks. IEEE Transactions on Multimedia 20(11),2018. pp. 2980-2992
18. Mescheryakov S. V., Yanchus V. E., Borevich E. V.: Experimental research of digital colorcorrection models and their impact on visual fixation of video frames. Humanitiesand Science University Journal 27, 2017. pp.15-24
19. Yanchus, V., Borevich, E.: Investigation of the value of the color scheme in the process of harmonizing the film frame. St. Petersburg State Polytechnic University Journal 4, 53-68 (2016)
20. Orlov, P., Laptev, V., Ivanov, V.: On the issue of using eye-tracking systems. St. Petersburg State Polytechnic University Journal 5(205), 84-94 (2014)
21. Yuriev, F.: Color imagery of information. Kiev (2007)
22. Ratanova, T.: Psychophysical scaling and objective physiological reactions in adults and children. Voronezh (2002)
23. MescheryakovS.V., YanchusV.E., BorevichE.V.: Statistical Model of Computing Experimenton Digital Color Correction. In: Vishnevskiy V., Samouylov K., KozyrevD. (eds.) Distributed Computer and Communication Networks. DCCN 2019, LNCS, vol. 11965, Springer, Cham, 2019pp. 140-151. https://doi.org/10.1007/978-3-030-36614-8 11
24. Itten, I.: The Art of Color. Dmitry Aronov (2014)
25. Scheffe, R.: The Analysis of Variance. Moscow (1999)
26. Glanz, S.: Biomedical Statistics. Moscow (1999)
27. Kabakov, R.: R in Action. Analysis and Visualization of Data in the Program R. Moscow (2014)

Peer Review

Peer reviewers' evaluations remain confidential and are not disclosed to the public. Only external reviews, authorized for publication by the article's author(s), are made public. Typically, these final reviews are conducted after the manuscript's revision. Adhering to our double-blind review policy, the reviewer's identity is kept confidential.
The list of publisher reviewers can be found here.

The reviewed article examines scientific approaches to the description of light radiation, describes the functions of light in the context of the transmission of semantic information through images and the emotional load of a movie frame. The subject of the study is an experimental technique for improving the human-computer graphical interface. The purpose of the article is to assess the degree of influence of the factor of the color solution of the frame on its perception by the viewer. The research methodology includes the use of well-known color perception models, solving problems related to the development of an experimental setup and a software module that allows interviewing subjects without involving them in the laboratory, as well as processing the data obtained. The author rightly associates the relevance of the article with the fact that the question of the correct change in the conditions of perception of the film frame in order to give the latter a stronger emotional load, according to the director's plan, remains unresolved and requires experimental research. The novelty of the results of the reviewed study, according to the reviewer, lies in the method proposed by the authors of the article for conducting a computational experiment to study the influence of the color scheme factor on the viewer's perception of a video frame at the final stage of image perception. The proposed system of arrays with data allows you to use the developed software module located on a network resource to conduct an experiment. The following sections are structurally highlighted in the article: Introduction, Purpose and objectives of the study, Methodology for preparing stimulus material, Information data model, Experimental methodology, Conclusions, Bibliography. When presenting the material, the authors have maintained the scientific style of its presentation. The article examines the work of the human visual system on the perception of visual information, highlights the stages of perception of visual information, describes the technique for creating arrays of targeted stimuli (an array of photorealistic stimuli and an array of animated stimuli), their parameters; an example of stimulus material (an animated frame and a video frame in five colors) is given; a flowchart of an information model of computational data is presented. the results of the experiment and the results of statistical processing of the experimental results, an algorithm for statistical processing of experimental data is presented, which provides for averaging the weighting coefficient for all homogeneous stimuli (photorealism-animation) for each subject and for each color solution, as well as comparing samples by five factors. Based on the results of the study, conclusions were drawn reflecting the scientific novelty and practical significance of the results obtained as a result of the work done. The methodology developed by the authors makes it possible to cover a large number of subjects, which will make it possible to level out the subjective component in the assessment of the stimulus material by the subjects and make the experimental results more reliable. The bibliographic list of the article includes 27 sources – publications of domestic and foreign scientists. There are address references to the sources listed in the list of references in the text. The reviewed material corresponds to the subject of the journal "Software Systems and Computational Methods", reflects the results of the original research work, which allows us to recommend it for publication.