Formal Interpretation of RL Agent Strategies in Cybersecurity Tasks Based on Temporal and Epistemic Logic

Lyutikova Larisa Adol'fovna ORCID: 0000-0003-4941-7854 PhD in Physics and Mathematics Head of the Department of Neuroinformatics and Machine Learning; Institute of Applied Mathematics and Automation of the Kabardino-Balkarian Scientific Center of the Russian Academy of Sciences 89a Shortanova str., Nalchik, 360000, Russia, Republic of Kabardino-Balkaria lylarisa@yandex.ru Other publications by this author     Hashirova Tat'yana Yur'evna Doctor of Technical Science Professor; Institute of Electronics, Robotics and Artificial Intelligence; Kabardino-Balkarian State University named after H.M. Berbekov Nalchik, St. 44 Shortanova, sq.5 khashirova@mail.ru

Arvanova Saniyat Muhamedovna Lecturer; Department of Computer Security; Kabardino-Balkarian State University named after H.M. Berbekov Nalchik, St. 44 Shortanova, sq.15 arvansa@mail.ru

Introduction

Modern cybersecurity systems face a constant increase in the complexity of network architecture and an increase in the variety of attacking influences. In these conditions, the ability of intelligent systems not only to effectively detect threats, but also to explain the decisions taken is of particular importance. The concept of explicable artificial intelligence (Explicable AI, XAI) is becoming a key direction in the development of trusted AI systems ^[1].

Traditional intrusion detection methods (IDS/IPS) based on signatures or statistical models have limited ability to adapt to new types of attacks and do not provide interpretation solutions. In contrast, Reinforcement Learning (RL) provides the ability to autonomously form an optimal strategy for agent behavior based on interaction with the environment. However, the internal mechanisms of RL agents are often opaque and difficult to interpret.

The problem of explainability of RL agent solutions is particularly acute in the field of cybersecurity, where a high degree of trust in the system's performance is required. To eliminate the "black box" effect, it is proposed in this paper to use formal logical models that allow describing and verifying agent behavior in terms of time dependencies and knowledge ^[2].

Linear Temporal Logic (LTL) allows you to capture cause-and–effect relationships between events, while Epistemic Logic (EL) formalizes agents' knowledge of the state of the system and the level of threats. The integration of these logical tools with reinforcement learning makes it possible to build explicable, verifiable, and reproducible strategies for responding to anomalies ^[3].

The purpose of this work is to develop and experimentally verify an RL agent model capable of making decisions in a network environment interpreted in terms of temporal and epistemic logic. The article presents the problem statement, mathematical formalization, description of the method of extracting logical rules and the results of a computational experiment confirming the effectiveness of the proposed approach^[4].

The subject of research in this paper is the logical and mathematical interpretation of the strategies of an intelligent agent trained by reinforcement learning, operating under conditions of partial observability and stochastic dynamics of the network environment.

The novelty of the work is the creation of a formally grounded explicable RL algorithm that ensures trust, reproducibility and transparency of solutions in cyber defense systems.

1. Setting the task

Let's assume that the environment is a computer network with a state reflecting traffic metrics (number of connections, ports, delays, error rate). The agent chooses an action (skip, check, block), receiving a reward. The dynamics of the environment is described by a stochastic process:

where is the noise of external influences.

The task of agent training is formulated as optimization of the return function.:

1

where is the strategy, and is the discount coefficient.

The agent must recognize abnormal patterns in network traffic (packet frequency, ports, IP addresses, delays, deviations). To determine the optimal response: "Pass" — to skip traffic, or "Inspect"— to check, or "Block" — to block the connection. Maximize the return function (1), where the reward takes into account the balance between safety and cost. To ensure the explainability of solutions, it is possible to describe actions in the form of logical formulas.

2. Mathematical model and formalization of the problem

Consider a cybersystem as a stochastic environment where:

— a set of environmental conditions describing network parameters (traffic speed, number of connections, number of errors, threat indicators);

— multiple agent actions (skip, check, block);

when performing an action in the state;

— reward function;

— discount rate.

At each time step, the agent observes the current state , selects a strategy action , receives a reward, and moves to a new state according to the distribution.

The agent's goal is to maximize the expected discounted return:

1

The value function is defined as the expected reward when choosing an action in a state and then following the policy.:

2,

The iterative update for the approximation function (the Q-learning method) has the form:

3,

where is the learning rate.

The state of the environment is described by a vector of network features:

where are the normalized values of network parameters: packet frequency, percentage of TCP reconnections, average response time, number of connections per port, etc.

The agent 's action is selected from a set where:

— skip traffic (pass);

— check (inspect);

— block (block).

The reward function has the form:

4

This function reflects a balance between security (timely blocking) and saving resources (avoiding unnecessary checks) ^[5].

After training, the strategy is approximated by a neural network with parameters. To ensure explainability, logical rules are extracted.

3. Logical grounds for interpreting agent behavior

Temporal Logic (LTL)

Linear Temporal Logic (LTL) is used to formally describe sequential processes where the state of the system changes over time. She uses statements about the future and the past of events, allowing her to formalize the requirements of safety and liveliness.

The main LTL operators necessary for this field are presented in Table e1.

Table 1. The main LTL operators

Example of the formula:which reads as: “always, if an anomaly occurs, then a blockage will occur in the future” ^[6].

The behavior of an agent in reinforcement learning is a sequence, which is naturally described in terms of temporal logic. This allows you to set and check the system properties.:

Safety This expression states that a state is prohibited where an attack has been detected, but the agent has not taken defensive action.

Achievability of a goal (liveness) asserts that it is always possible that a threat will be eliminated in the future.

The sequence of actions: which reads as "Always, if the check action is performed, then the next step will either block or restore the norm."

The introduced formulas of temporal logic (safety, liveness, and order) make it possible to move from empirical reinforcement learning based on statistical optimization of the reward function to formally verifiable agent behavior.

Within the framework of this model, each trajectory of an agent's interaction with the environment is considered as a sequence of states and actions over which a logical check can be performed for the fulfillment of a security property, which ensures that during the functioning of the system there are no states in which an attack remains unanswered. A livability property that ensures that the network can be restored to normal operation after a threat. And the property of the sequence of actions that captures the causal relationships between the phases of response (detection - verification — blocking/normalization) ^[7].

Verification of these formulas by means of formal verification (for example, using the NuSMV or SPIN checker model) makes it possible to verify that the agent's strategy not only maximizes remuneration, but also meets the logical criteria of correctness, stability and explainability.

Thus, stochastic reinforcement learning and formal logic methods are integrated, ensuring the explainability and reproducibility of intelligent security solutions.

Epistemic logic (logic of knowledge)

Epistemic Logic (EL) describes the knowledge and beliefs of agents. The main operator means: “the agent knows that.” Collective operators are introduced for groups of agents.:

Agents in the cybersystem work in conditions of incomplete information: everyone observes only a part of the signs of traffic. Therefore, a formal description of knowledge and awareness is required.

Individual knowledge looks like

collective knowledge:

This model is important for distributed agents operating on different network nodes: some detect a threat, others confirm or respond by forming collective knowledge and coordinating actions ^[8].

LTL and EL combination

The combined use of temporal and epistemic logic makes it possible to describe both the dynamics of behavior and the level of knowledge of agents. For example: – “always, if an agent knows about an anomaly, he initiates a block in the future.”

Justification of applicability to the data in Table 2.

Table 2. Rationale for applying LTL and EL logic to data

Input data structure

Each observation of the environment at a given time is described by a feature vector:

where are the normalized network metrics. The agent's input data is presented in Table 3.

Table 3. Agent input data

Agent's output data

The agent chooses actions:

– Pass: skip the stream;

– Inspect: perform an inspection;

– Block: Block the connection.

The result is an optimal policy and function. Based on them, logical rules are extracted, for example:

4. Computational experiment

The step-by-step implementation of the method is as follows:

1. Download the cybersecurity environment;

2. we train the DQN agent;

3. we collect trajectories and build graphs (reward, distribution of signs of norms/anomalies);

4. Extract and print the rules;

5. We show which anomalies the agent actually "saw".

An example of the data in Fig. 1.

Figure 1. Sample data.

The block diagram of the algorithm is shown in Fig. 2.

Figure 2. Algorithm diagram

Results

The graph of agent behavior in the reinforcement learning process is shown in Fig. 3.

Figure 3. The process of training an agent by episode.

X-axis (horizontal) this is the training episode number. Each episode is a separate series of agent interactions with the environment (for example, 50 steps). At the beginning of the training, the agent acts randomly, and by the end it becomes more meaningful.

Episodes 0-50 give chaotic low values, which indicates that the agent is just learning. Then (100-200 episodes) — the average reward begins to grow, although fluctuations remain. After ~250 episodes, stabilization is at a high level: the average reward value is above 60-70 (or any upper bound for your environment).

The main thing is the average line level. If the curve "as a whole" goes up, it means that the agent learns the strategy, even if each specific episode is "jagged" ^[9].

Intensity histograms (the X₁ sign) are a very revealing visual element that actually shows how an agent “sees” the difference between normal and abnormal traffic ^[10].

The histograms show the distribution of one feature, x1, which indicates the intensity of network traffic (number of packets per second) (Fig. 4).

Figure 4. Network traffic intensity

The left histogram is "X₁ (intensity) is the NORM".

Shows how the intensity values are distributed during normal network operation when there is no anomaly (anom = 0).

The right histogram is "x₁ (intensity) – ANOMALY". The same thing, but only for abnormal states (anom = 1).

Next comes the formation of logical rules, a fragment of which is shown in Fig. 5.

Figure 5 Fragment of logical rules

Based on the data under consideration, the agent achieves an anomaly detection accuracy of 94-96% with an average reaction delay of less than 0.3 s. The logical interpretation shows that the agent's strategy follows the rules of the form:

^[11]

The formulas obtained are successfully tested for the feasibility of the specifications and .

Conclusion

The paper proposes a method for the logical and mathematical interpretation of strategies obtained during reinforcement learning in relation to cybersecurity tasks. An RL agent model has been developed that interacts with the network environment and makes decisions about checking or blocking traffic based on optimizing the reward function.

The main scientific result is the integration of stochastic reinforcement learning methods with formal means of logical verification. The use of temporal logic (LTL) made it possible to formalize and verify the properties of safety, liveness, and causality of actions. Epistemic logic (EL) has provided an opportunity to describe the level of knowledge and awareness of agents in a distributed system.

The results of the computational experiment showed that the agent is able to achieve high accuracy in detecting threats with a short reaction time, and the resulting logical formulas are successfully tested for the feasibility of specifications. This confirms that the logical interpretation of strategies increases transparency and trust in intelligent system solutions.

In the future, it is planned to develop the proposed approach for multi-agent systems, including the exchange of knowledge between network nodes, as well as the implementation of automatic verification of logical properties in real time. Additionally, research on embedding logical constraints directly into the learning process (reward shaping) is of interest, which will allow combining optimality and explainability at the learning stage, and not just after the fact.

The article is published in the version approved by the reviewers (after receiving a positive review recommending the manuscript for publication) with corrections made by the author (after receiving the editor’s comments, if any).
Read all reviews on this article