Determining the authorship of the "Notes of the Decembrist I.I. Gorbachevsky" by machine learning methods

Latonov Vasilii Vasilyevich ORCID: 0000-0002-7810-8033 PhD in Physics and Mathematics Head of the Data Research Department; Sberbank PJSC 41 Ostrovityanova str., room 1, 172, Moscow, 117342, Russia WLatonov@gmail.com Other publications by this author

Latonova Anastasiia Vyacheslavovna PhD in History independent researcher 41 Ostrovityanova str., room 1, 172, Moscow, 117342, Russia iskrenne_vasha_aa@mail.ru Other publications by this author

Introduction

The "Notes" of I.I. Gorbachevsky are one of the most valuable sources on the history of the Decembrist movement, created by its participants themselves. They highlight the formation and development of such a Decembrist organization as the Society of United Slavs, which later joined the Southern Society of Decembrists. Written in exile in Siberia, these notes represent not only a source of factual material, but also an original concept of the secret society's development, and a retrospective "inside look" at the mistakes made by the conspirators. A common thread through the entire concept of "Notes" is the exposure of the blunders of Southern society, the views and actions of the participants of which are opposed to the views and actions of the "united Slavs" themselves (not without some idealization of the latter).

The "Notes" in its printed version occupy not much more than 100 pages and are divided into three sections. The first of them tells the reader the end of the history of the Society of the United Slavs, when it merges with the Southern Society of the Decembrists. The second one highlights the uprising of the Chernigov infantry regiment, which ex-"Slavs" played a significant role in the events. The third section tells about the further fate of the rebels – about their trial, exile, and the unsuccessful rebellion of convicts organized by I.I. Sukhinov. Thus, as evidence of the author's undoubted historical self-awareness, according to the facts contained in them, "Gorbachevsky's Notes" go far beyond the limits of his biography, whoever he may be.

The last caveat is not accidental. Contrary to the well-established name of the "Notes" in the literature, we cannot unequivocally assert that their author from among the Decembrists was I.I. Gorbachevsky himself. The fact is that the first publication of this source, in the journal "Russian Archive" in 1882, issue 2, was presented under the heading "Notes of an Unknown Person from the Society of the United Slavs." The publisher (P.I. Bartenev) printed them based on an anonymous manuscript received from Siberia, accompanied by a note: "It seems that these "Notes" were compiled by former lieutenant of the 8th Artillery Brigade Ivan Ivanovich Gorbachevsky; but it is impossible to guarantee this."

An overview of the existing versions of the authorship of the "Notes"

The anonymity of the first publication gave rise to doubts about Gorbachevsky's authorship, which were formulated in detail by one of the founders of Soviet Decembrist studies, M.V. Nechkina. Having demonstrated in the course of her analysis that the "Notes" could similarly belong to the pen of Gorbachevsky's comrades (whose name originally arose as an assumption of the publisher who found the manuscript), she came to the conclusion that the author of the source was most likely another member of the Society of the United Slavs, P.I. Borisov ^{[1, pp. 136-143]}.

Gorbachevsky's biographer, G.P. Shatrova, also did not consider him the author of the "Notes" (in any case, the only author). In the monograph "The Decembrists and Siberia" (1962)^[2], she positioned herself as the creator of P.I. Borisov's "Notes", appealing to the opinion of M.V. Nechkina; in the monograph on Gorbachev personally, she came to the conclusion that the "Notes" were compiled on the basis of collective memories with subsequent literary processing by one author, and I.I. Gorbachevsky took an active part in this common work. The main argument against Gorbachevsky's exclusive authorship, G.P. Shatrova, found a significant discrepancy between the position of the "Notes" and Gorbachevsky's views expressed in letters from the 1850s to the 1860s on the settlement. ^{[3, P. 75]}. Another researcher, N.P. Matkhanova, wrote about the "Notes" as a result of a jointly developed concept as a result of an exchange of opinions within the team. In this regard, we read: "in the casemate community, individual memory was transformed into a social, identifying group, and the development of a "collective history" – that common version of the past, a conditional scheme., a general idea that consisted of images of events" ^{[4, p. 160]}.

At the same time, this point of view on the authorship of Gorbachevsky's "Notes" (and M.V. Nechkina's initial arguments themselves) was criticized by a number of other historians. B.E. Syroechkovsky, L.A. Sokolsky, I.V. Porokhov found inconsistencies between the "Notes" and other texts of Gorbachevsky (investigative testimony and letters) to be insignificant and They are explicable, and Gorbachevsky's authorship is absolutely indisputable. (For detailed argumentation, see: ^[5]; also see ^[6]) As a result of textual analysis, Zlobin E.V. later came to similar conclusions ^[7]. Already in the post-Soviet period, I.I. Gorbachevsky was officially listed as the author of the "Notes", for example, in the Great Russian Encyclopedia ^[8]. At the same time, the opposite point of view continues to prevail in the historiographical field, according to which Gorbachev is not the author of the "Notes" of the same name (see: ^[9]).

Thus, it can be argued that the issue of the authorship of Gorbachevsky's "Notes" has not been fully resolved to date. However, new opportunities are now emerging for its research due to the development of information technology. Agreeing with our predecessors that it is hardly possible to consider the question of the authorship of the "Notes" without comparing them with other texts by Gorbachevsky, let's try to analyze all these texts using stylometry ^[10]. Unlike the traditional approach, in which only a comparison of sources that overlap in any way is adequate, in our case we can include any texts that are guaranteed to belong to the pen of the persons we are interested in. So, if B.E. Syroechkovsky, L.A. Sokolsky and I.V. Porokh considered only Gorbachevsky's letter to M.A. Bestuzhev on June 12, 1861, containing a lot of material about the Decembrist movement as a whole and about individual Decembrists worthy of comparison with the "Notes", then we can now draw to comparison even those letters of Gorbachevsky in which there are no judgments on the Decembrist theme. The same applies to the materials of other members of the Society of the United Slavs, who may have participated in the creation of the "Notes", and above all, to the materials of P.I. Borisov.

There is also a version that the "Gorbachevsky Notes" were written by a team of authors. However, there is no reliable information about which authors and in what proportion could have participated in the creation of the "Notes". Consequently, in this discourse, the correct task of stylometry cannot be formulated and arbitrarily meaningful conclusions cannot be drawn. Therefore, in the framework of this work, we will focus on verifying the versions of the authorship of the "Notes" in the person of I.I. Gorbachevsky and P.I. Borisov.

Review of works on stylometry and problem statement

Stylometry is a discipline that deals with measuring stylistic characteristics in order to organize and systematize texts ^[10]. These characteristics can be calculated for any sufficiently large author's text, and for each author's style these characteristics will be unique. Thus, stylometry can be used to determine the authorship of a text if there are samples of texts by possible authors large enough to calculate characteristics.

The task of determining authorship itself has existed for many centuries, but it was first formalized, apparently, in the work of N.A. Morozov ^[11]. In this work, a method was proposed for identifying authorship through graphs of the frequency of use of words. The article ^[12] by A.A. Markov should also be mentioned among the first works on the mathematical study of the authors' stylistics. In this article, A.A. Markov applied a statistical analysis that he had previously described in another paper ^[13], where the probability of a letter being a vowel was estimated depending on the chain of the two previous letters.

In ^[14], the idea of chains of two-letter combinations from ^[13] was applied and it was shown that this approach makes it possible to determine the true author with a probability of 84% when considering 80 possible variants. A generalization of this method was proposed in ^[15], where not only two-letter combinations, but also single grammatical classes of words, as well as pairs of words, were considered as units of analysis. This work is of particular interest due to the fact that, according to the authors' conclusions, the definition of authorship of a text by two-letter combinations is more accurate than by single words and pairs of words. A similar result was obtained in ^[16], where it was shown that three-letter combinations make it possible to establish authorship more accurately than words.

In ^{[17, 18]}, to establish authorship, the probabilities of the occurrence of different n-letter combinations for n > 2, called n-grams, were considered. In some works, n-grams are also called sequences of n words, which are also suitable for determining authorship, for example in the article ^[19]. The authors found out that for English, 6-grams give the best result of recognizing the author, while, for example, for Greek, the best result is achieved when using 3-grams.

Another well–known approach to attribution of authorship is the Burroughs Delta ^[20], a method published in 2002. In his work, John Burroughs introduced a metric called Delta, used to measure the distance between texts ^[21]. The delta is calculated based on the entire dictionary of words used in all texts, between which the distance is calculated. This metric takes into account the frequency of use of each word in a single text and the frequency of use of the word in the entire set of texts. The Burroughs delta is widely used in linguistic research ^{[22, 23]}, including to determine the measure of the authors' style in collaboration ^[24].

Along with this, there are a number of works in Russian historiography devoted to defining the features of the author's style. The result of the works of L.V. Milov, L.I. Borodkin and other historians in the 1970s and 1980s was research ^[25-27], where network analysis of grammatical structures was used for the tasks of style analysis and attribution. In many ways, the object of these studies were medieval Russian texts.

Machine learning (ML) methods are often used in modern works to determine authorship. So, in ^[28], the authors applied a number of classical methods in the task of attribution of authorship using the example of literary texts. Among others, the k nearest neighbors (KNN) method ^[29] and the support vector machine (SVC) method ^[30] were tested. The authors pay special attention to the issue of text preprocessing: they consider learning both from the source text and from a text with deleted rarely occurring words.

Deep learning is also used in the task of attribution of authorship, for example, the authors ^[31] used a convolutional neural network (CNN) and compared its accuracy with other approaches, in particular, multilayer linear perceptron (MLP) ^[32], as well as with the KNN and SVC already mentioned above. In ^[33], the authors solved the problem of determining the authorship of Russian-language texts, building the learning process on texts from classical literature and short publications on social networks. Along with classical ML methods, the authors used neural networks that include architectures such as LSTM ^[34] and BERT ^[35].

In this paper, the problem of determining the authorship of Gorbachevsky's Notes is solved. The publication of the "Notes" of 1963 in Literary Monuments was used as the material for processing, for which there is a digital copy on the Internet, which makes its use more convenient. This version differs from the original 1882 publication in places by a different paragraph division, which is not essential for our task, as well as by the presence of brief annotations before the chapters (these annotations were deleted by us during text processing). I.I. Gorbachevsky and P.I. Borisov themselves were considered as candidates for the authorship of the "Notes", and samples of texts written by these two Decembrists were used for the solution. To study Gorbachev's style, his letters were used as a source, primarily storing information about the relations between the exiled Decembrists and their living conditions in the settlement. At the same time, these letters contain enough of Gorbachev's judgments about the Decembrist conspiracy. 81 of his letters for the period 1839-1868 are available for analysis. Letters were also used to study Borisov's style, as in the case of Gorbachev. There are relatively few letters from Borisov at our disposal, 20. All of them were written in 1838-1847 during his life in the settlement and are very close to the letters in their subject matter.Gorbachev's.

Problem solving

To solve this problem, we used classical ML methods: SVC and logistic regression (LR) ^[36]. The latter was chosen insofar as it is focused on solving the problem of binary classification, and in our formulation there are only two possible authors. It is worth noting that these methods cannot be applied by themselves to text data, so each sentence is encoded into a vector of numeric features.

Two methods were used for encoding. The first of them, Count Vectorizer (CV), refers to classical ML approaches. Each sentence was preprocessed before being encoded using CV:

1. Replacing the upper case with the lower case;

2. Removing punctuation marks, round and square brackets, and quotation marks;

3. Removal of official words: prepositions, conjunctions, particles, interjections.

The second chosen encoding method is the BERT model, pre–trained on Russian-language texts (the model available at the link was used: https://huggingface.co/papers/2408.12503 ). This approach allows you to achieve greater accuracy in learning than when coding using classical methods, since, having been pre-trained, it already contains information about Russian-language texts and how to encode them effectively. Thus, prediction models were trained.:

1. Count Vectorizer + SVC;

2. Count Vectorizer + LR;

3. BERT + SVC;

4. BERT + LR;

The diagram of models 1-2 is shown in Figure 1, models 3-4 – in Figure 2.

Figure 1.

Figure 2.

Each model accepts one sentence as input and classifies it as belonging to the authorship of Gorbachevsky or Borisov. Text preprocessing and model training were implemented in Python. The implementation of the SVC, LR, and Count Vectorizer models were taken from the sklearn library, and the pandas and numpy libraries were used for preprocessing.

There are a total of 411 sentences in Borisov's letters, while there are 2,620 in Gorbachev's letters. An equal number of sentences from both authors are needed for training, so 411 sentences had to be selected from Gorbachevsky's letters. The sample was determined by the random number generator of the numpy library, and one hundred different samples were taken using the seed parameter, varying in the range from 0 to 99. For each of the samples of Gorbachevsky's sentences, each of the four selected models was trained. During the training, 80% of the sentences were allocated to the training sample, and 20% to the test sample. The best hyperparameters were selected for each model using the Grid Search method.

Results

We have received four blocks of 100 models each. After training, each model was used to predict the author of each sentence from the "Notes" individually. The proportions of sentences attributed to Gorbachev by models of different blocks are shown in Figures 3-6. Also on these graphs, the average shares of proposals attributed to Gorbachev are marked with a crimson dotted line. The average is taken for all models within the same block. A solid crimson line marks the maximum and minimum fractions. For example, as can be seen from Figure 4, the Count Vectorizer + LR model, trained on one of a hundred samples, classified more than 80% of the sentences of the "Notes" as belonging to Gorbachevsky's authorship. At the same time, the minimum share of this model exceeds 57%. The Count Vectorizer + SVC and BERT + SVC models have a maximum share of just over 79%. The minimum shares of these models exceed 51% and 48%, respectively. Table 1 shows examples of error matrices of four models based on test samples – one model from each block. It can be seen that the accuracy of the models using BERT is higher than the rest. For all models in all blocks, the accuracy in the training and test samples differed by 1-3%. It is also seen that models that do not use BERT have higher accuracy on Gorbachevsky's sentences, and models with BERT have approximately the same accuracy on both authors. Table 2 shows the average sample shares for all models, as well as the average accuracy values for training within each block.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Table 1. Examples of model error matrices based on test samples.

Table 2. Accuracy of models and prediction of authorship of sentences from "Notes".

Most trained models classify about 70% of the sentences of the "Notes" as written by Gorbachev. More accurate models (using BERT) attribute slightly fewer sentences of "Notes" to Gorbachev, but this is still more than 64% on average. The variation in classification results, which is visible in Figures 3-6, is explained by the fact that one hundred different samples were used to train one hundred models in each of the blocks. Nevertheless, this spread does not affect the interpretation of the result, since the graphs show a significant advantage in favor of Gorbachev in each of the four blocks.

Conclusions

The paper considers the problem of determining the authorship of the "Gorbachevsky Notes", with the assumption that the author could be either I.I. Gorbachevsky himself or P.I. Borisov. The machine learning methods used in recent years have proven to be the most accurate in attribution tasks, and in our training work they showed accuracy of more than 80% (BERT + SVC and BERT + LR – about 90%). Almost all trained models classified about 70% of the sentences of the "Notes" as written by Gorbachevsky. Thus, it can be concluded that the "Gorbachevsky Notes" should be attributed as belonging to the pen of the real I.I. Gorbachevsky, and not P.I. Borisov, if the dichotomy of these two authors is considered as a choice.