The dog analogy is enough to build intuition; a security engineering team building a real agent needs the formal structure underneath it. Nearly every reinforcement learning problem, including every example in this unit, is framed as a Markov Decision Process (MDP), defined by a five-element tuple $(S, A, P, R, \gamma)$.
| Symbol | Meaning |
|---|---|
| $S$ | The set of all possible states |
| $A$ | The set of all possible actions |
| $P$ | The transition probability: given state $s$ and action $a$, how likely is each next state $s'$ |
| $R$ | The reward function: what reward follows from taking action $a$ in state $s$ |
| $\gamma$ | The discount factor (between 0 and 1): how much the agent values future reward relative to immediate reward |
Every RL algorithm in this unit solves one of these.
ML concept: The Markov Property
"Markov" means the future depends only on the current state, not on the full history of how the agent arrived there. For the honeypot example later in this unit, this means the agent's next decision depends only on the session's current state (how many commands have run so far), not on the exact sequence of prior sessions. This assumption is what makes the problem tractable; when it is badly violated (the agent genuinely needs long memory of past sessions), more advanced architectures with explicit memory are needed, beyond this unit's scope.
Try it yourself: Name the Five Elements
For a spam filter that decides, email by email, whether to deliver, quarantine, or discard, and adjusts its policy based on user complaints about missed spam and false quarantines, sketch what $S$, $A$, $R$ and $\gamma$ would concretely be. Does this problem genuinely satisfy the Markov property, or does the right decision plausibly depend on more than just the current email?