Unit 4 Quick Reference, Self-Study and Lab

For a web-application honeypot: the state could be how many distinct URL paths an attacker has probed so far; the actions could be serving a real-looking response, injecting an artificial delay, or redirecting to a dead end; the reward could be a positive score for each new technique observed minus a penalty proportional to how suspicious the attacker's behaviour suggests they have become; the policy is the learned rule mapping the probe count to the best action to take next.

An agent that always exploits its current best-known action can get permanently stuck on a merely adequate policy, because it never tries the alternatives that might actually be better. Continued exploration, even at a small rate, lets the agent keep testing whether an action it currently undervalues might in fact be superior, which a purely greedy policy would never discover.

A static honeypot responds to every connection the same scripted way: identical response timing, identical fake file listings, identical behaviour after an unusual command. A patient attacker or automated reconnaissance tool can notice this pattern over repeated interactions. An adaptive honeypot instead decides session by session how to respond, based on what has kept past attackers engaged longest, so there is no single fixed script for an attacker to fingerprint.

If the detection-risk penalty for continued engagement were set too high relative to the small negative reward for terminating, the agent could learn that immediately terminating every session is the safest way to maximise reward, since termination's small fixed cost beats the accumulating risk of staying engaged. The agent would technically be maximising its reward function while gathering zero intelligence on any attacker, which is the opposite of what the honeypot exists to do.

S is the set of ten discretised traffic-rate bins, from low to extreme. A is the set of three actions: relax, hold, or tighten the rate limit. P is the transition probability that traffic moves from one rate bin to a neighbouring bin after an action, here modelled as a small random step. R is the traffic_reward function, which penalises blocking legitimate traffic when tightening is unnecessary and penalises failing to tighten when the rate bin is genuinely high. Gamma is the discount factor, set to 0.9, controlling how much the agent values future reward relative to the immediate reward of its next action.

A table-based Q-learning agent needs one entry for every state-action pair, which only works because this unit deliberately kept every state space small, ten bins, ten steps. Real network telemetry has a far larger, often continuous state space built from packet rate, source diversity, and protocol mix, where an exhaustive table would need an impractically large number of entries. A Deep Q-Network replaces the table with a neural network that generalises across similar states it has never exactly seen before, the same way a classifier in Unit 2 generalises across feature vectors, making large or continuous state spaces tractable.

The honeypot agent's state is a simple discrete count of commands run so far, capped at nine, while the rate-limiting agent's state is a continuous traffic rate discretised into ten bins, reflecting that network traffic genuinely varies continuously rather than in discrete steps. Their actions differ in kind: the honeypot chooses how to respond to a single attacker session, while the rate-limiter chooses a defensive posture affecting all current traffic. Their reward functions reflect different trade-offs: the honeypot balances intelligence gathered against detection risk, while the rate-limiter balances blocking legitimate users against letting an attack through, each shaped by what that specific security problem actually needs to optimise for.