Honeypot engagement is one concrete application of RL; securing live network traffic is another, and it is worth its own worked example because the state space is naturally continuous rather than a simple command count. Consider a defender deciding, every few seconds, whether to tighten or relax a rate limit on an endpoint under suspected denial-of-service pressure: too strict, and legitimate users are blocked; too loose, and the attack succeeds.
rate_limit_qlearning.py
import numpy as np
# State: current traffic rate, discretised into 10 bins from low to extreme
# Actions: 0 = relax limit, 1 = hold limit, 2 = tighten limit
n_states, n_actions = 10, 3
Q = np.zeros((n_states, n_actions))
alpha, gamma, epsilon = 0.1, 0.9, 0.3
def traffic_reward(rate_bin: int, action: int) -> float:
legit_user_cost = 0.3 if action == 2 else 0.0 # tightening blocks some legitimate traffic
if rate_bin >= 7: # genuinely high traffic: tightening pays off
attack_cost = 0.0 if action == 2 else (rate_bin - 6) * 0.4
else: # normal traffic: tightening is pure cost
attack_cost = 0.0 if action != 2 else 0.2
return 1.0 - legit_user_cost - attack_cost
for episode in range(500):
rate_bin = np.random.randint(n_states)
for _step in range(20):
action = (np.random.randint(n_actions) if np.random.random() < epsilon
else int(np.argmax(Q[rate_bin])))
r = traffic_reward(rate_bin, action)
next_bin = int(np.clip(rate_bin + np.random.choice([-1, 0, 1]), 0, n_states - 1))
Q[rate_bin, action] += alpha * (r + gamma * np.max(Q[next_bin]) - Q[rate_bin, action])
rate_bin = next_bin
epsilon = max(0.02, epsilon * 0.99)
ML concept: From Tables to Functions: Why Large State Spaces Need Deep Q-Networks
The agents in this unit use a table for $Q(s, a)$, which works because every state space here was deliberately kept small (ten discretised bins, ten session-length steps). Real network telemetry has a far larger, often continuous state space, packet rate, source diversity, protocol mix, and a table would need an impractical number of entries. A Deep Q-Network (DQN) replaces the table with a neural network that takes the state as input and outputs an estimated $Q(s, a)$ for every action, learning to generalise across similar states it has never exactly seen before, exactly as a classifier in Unit 2 generalises across feature vectors. The update rule is conceptually the same as table-based Q-learning; only the representation of $Q$ changes.
The same state-action-reward framing extends further, beyond both honeypots and rate-limiting:
- Moving target defence: periodically changing a system's exposed configuration (ports, service banners, network addresses) to deny an attacker the stable reconnaissance picture they need, where RL decides when to rotate based on observed probing activity.
- Patch and maintenance scheduling: balancing the security benefit of patching quickly against the operational cost of downtime, framed as a sequential decision problem with a long-term reward.
Case study: Moving Target Defence as an Active Research Area
Moving target defence has been an explicit, funded research priority for over a decade, including dedicated programmes historically supported by DARPA, precisely because static configurations give a patient attacker unlimited time to map a target. Framing when and how much to rotate a defence as a reinforcement learning problem, rather than a fixed schedule, lets the defence adapt its tempo to actually observed probing activity instead of rotating on a timer an attacker can simply wait out.
Try it yourself: Tune the Rate-Limit Reward Trade-Off
In the rate-limiting agent above, change legit_user_cost from 0.3 to 0.6, making the agent far more reluctant to tighten limits. Retrain and inspect the learned policy (np.argmax(Q, axis=1)): at what traffic-rate bin does the agent now choose to tighten, compared to before? This is the same reward-sensitivity lesson from the honeypot topic, applied to a second, independent problem.
Caution: RL Is Data-Hungry and Easy to Reward-Hack
The toy honeypot agent trained in 300 simulated episodes because the reward function was hand-written and the simulation was instantaneous. A real deployment needs either a very large number of real sessions or a realistic simulator, and badly designed reward functions are a famous source of RL agents that technically maximise reward while doing something useless or actively harmful, for example, an agent that learns to terminate every session instantly because termination's small negative reward is still better than a badly calibrated engagement reward. Reward design deserves as much scrutiny as the algorithm itself.
Try it yourself: Change the Reward, Watch the Policy Change
In the honeypot agent's code, change the detection_risk coefficient from 0.04 to 0.12 and retrain. Plot the new reward curve and compare how quickly the agent learns to terminate sessions earlier. You are directly observing how sensitive a policy is to the reward function's exact shape, the central caution above.