The standard classification of TASK ENVIRONMENTS. Learn all six with what each implies for the agent. 1. FULLY OBSERVABLE vs PARTIALLY OBSERVABLE FULLY OBSERVABLE: the sensors give access to the COMPLETE state of the environment at each point in time — everything relevant to the choice of action. PARTIALLY OBSERVABLE: because of noisy or inaccurate sensors, or because parts of the state are simply missing from the sensor data. (UNOBSERVABLE is the extreme case: no sensors at all. Such an agent can still act, and sometimes succeed — a fixed cleaning sequence with no sensing is a real design.) IMPLICATION: fully observable → a simple reflex agent may suffice. Partially observable → the agent MUST maintain INTERNAL STATE, so a model-based structure is mandatory. EXAMPLES: chess is fully observable; poker is partially observable (hidden cards); driving is partially observable (you cannot see round the corner). 2. SINGLE-AGENT vs MULTI-AGENT The test is not whether other entities exist but whether their behaviour is best described as MAXIMISING A PERFORMANCE MEASURE THAT DEPENDS ON OUR AGENT'S ACTIONS. A taxi treating other cars as moving obstacles is a single-agent view; treating them as drivers who will react to being cut up is a multi-agent view. COMPETITIVE multi-agent: chess — one agent's gain is the other's loss. COOPERATIVE multi-agent: avoiding collisions, where both drivers want the same thing. IMPLICATION: multi-agent environments introduce RANDOMISED BEHAVIOUR as rational — being unpredictable has value when an opponent can exploit predictability, which is why randomisation appears here for a second, quite different reason than in the reflex-agent loop case. 3. DETERMINISTIC vs STOCHASTIC DETERMINISTIC: the next state is COMPLETELY DETERMINED by the current state and the agent's action. STOCHASTIC: it is not, and the uncertainty is quantified with PROBABILITIES. (NON-DETERMINISTIC is a distinct term: outcomes are listed but not given probabilities, so the agent must plan for all of them rather than for the likely one.) IMPORTANT SUBTLETY: an environment that is deterministic but PARTIALLY OBSERVABLE may APPEAR stochastic to the agent — and for design purposes, appearance is what matters. Taxi driving is stochastic because tyres blow and engines fail unpredictably. IMPLICATION: stochastic → the agent must handle uncertainty, which points at utility and expected values rather than goals. 4. EPISODIC vs SEQUENTIAL EPISODIC: the agent's experience is divided into atomic episodes; in each the agent receives a percept and performs one action, and CRUCIALLY THE ACTION IN ONE EPISODE DOES NOT AFFECT LATER EPISODES. SEQUENTIAL: the current decision COULD AFFECT ALL FUTURE DECISIONS. EXAMPLES: classifying defective parts on a belt is episodic — each part is judged independently. Chess and driving are sequential — short-term actions have long-term consequences. IMPLICATION: episodic environments need no lookahead, so the agent can be far simpler. Sequential ones require planning or learning about delayed consequences. 5. STATIC vs DYNAMIC STATIC: the environment does not change while the agent is deliberating. DYNAMIC: it does, so the agent that fails to act is effectively deciding to do nothing, and time itself matters. SEMIDYNAMIC: the environment does not change with the passage of time, but the agent's PERFORMANCE SCORE does — chess with a clock is the standard example. IMPLICATION: dynamic → the agent must respond within a deadline, which is the reactivity requirement and the reason for layered architectures. 6. DISCRETE vs CONTINUOUS Applies separately to the STATE, to TIME, and to the PERCEPTS and ACTIONS. Chess has a finite number of discrete states and discrete actions. Taxi driving is continuous-state and continuous-time: speed and location sweep through a range of values, and steering is continuous. IMPLICATION: continuous spaces cannot be searched by enumeration, so they require discretisation or methods built for continuous domains. AND A SEVENTH, OFTEN LISTED SEPARATELY: KNOWN vs UNKNOWN This refers to the AGENT'S (or designer's) KNOWLEDGE OF THE LAWS OF PHYSICS of the environment — not to the environment's observability. THE DISTINCTION IS COMMONLY CONFUSED AND WORTH GETTING RIGHT: a KNOWN environment can be PARTIALLY OBSERVABLE (solitaire — you know the rules, you cannot see the face-down cards), and an UNKNOWN environment can be FULLY OBSERVABLE (a new video game where everything is on screen but you do not yet know what the buttons do). IMPLICATION: unknown → the agent must LEARN the transition model, which is exactly what reinforcement learning does. THE HARDEST CASE IS THEREFORE: partially observable, multi-agent, stochastic, sequential, dynamic, continuous and unknown — which is a fair description of driving a taxi, and of most real problems.

THE STANDARD TABLE — worth memorising, because exam questions ask for exactly this: TASK ENVIRONMENT Obsv Agents Determ. Episodic Static Discrete ──────────────────────────────────────────────────────────────────────── Crossword puzzle full single determ. sequen. static discrete Chess with a clock full multi determ. sequen. semi discrete Poker partial multi stoch. sequen. static discrete Backgammon full multi stoch. sequen. static discrete Taxi driving partial multi stoch. sequen. dynamic continuous Medical diagnosis partial single stoch. sequen. dynamic continuous Image analysis full single determ. episodic semi continuous Part-picking robot partial single stoch. episodic dynamic continuous Refinery control partial single stoch. sequen. dynamic continuous English tutor partial multi stoch. sequen. dynamic discrete ──────────────────────────────────────────────────────────────────────── READ THE INTERESTING ROWS RATHER THAN MEMORISING BLINDLY: · CHESS is deterministic but BACKGAMMON is stochastic, purely because backgammon has dice. Both are fully observable multi-agent games, and that one difference changes the algorithm entirely — minimax works for chess, and backgammon needs expectiminimax with probability-weighted chance nodes. · IMAGE ANALYSIS is the only EPISODIC row among the non-robotic tasks, and that is why it is comparatively easy: each image is judged alone, with no consequences for the next. The enormous progress in image classification is partly a consequence of the problem's episodic structure. · THE PART-PICKING ROBOT is EPISODIC yet DYNAMIC and STOCHASTIC — each part is independent, but the belt keeps moving and the grip may slip. Episodic does not mean easy. · THE ENGLISH TUTOR is MULTI-AGENT, which surprises people. The student is an agent whose behaviour depends on what the tutor does — including gaming the system if the tutor's measure is gameable, which is the warning from the PEAS topic. · MEDICAL DIAGNOSIS is DYNAMIC because the patient's condition changes while the system deliberates, which is why diagnosis has a deadline in a way that crossword solving does not. WORKED CLASSIFICATION — the study-recommendation agent, and what each answer forces: Fully or partially observable? PARTIAL — the agent sees ticks, reading depth and notes, never comprehension. → FORCES internal state; a simple reflex agent is ruled out. Single or multi-agent? SINGLE, with a caveat: the student is arguably an agent, and if the recommendations are gameable the student will game them. Modelling the student as a passive environment is a SIMPLIFICATION we should record as such. → allows single-agent methods, but the measure must be chosen as if the student were adversarial. Deterministic or stochastic? STOCHASTIC — whether the student studies, and how fast they forget, are probabilistic. → FORCES expected-value reasoning; a plain goal is not enough, so utilities are required. Episodic or sequential? SEQUENTIAL — today's recommendation changes what is worth recommending tomorrow, and the exam is a single terminal event. → FORCES lookahead: recommending the highest-value topic today may be worse than building toward coverage. Static or dynamic? DYNAMIC — the exam date approaches whether the agent acts or not, and retention decays with time. → FORCES time into the state; a recommendation is only valid for a moment. Discrete or continuous? DISCRETE actions (which topic to suggest, from 519) with CONTINUOUS state (retention probabilities, time remaining). → a mixed case, which is normal. Known or unknown? PARTIALLY KNOWN — the syllabus and exam weights are known; the individual student's forgetting curve is not. → FORCES learning for that one component. THE CLASSIFICATION DECIDED THE DESIGN: partially observable + stochastic + sequential + dynamic + partly unknown means a UTILITY-BASED, MODEL-BASED, LEARNING agent — which is the most expensive type on the list, and now justified item by item rather than assumed.

EACH PROPERTY RULES A DESIGN IN OR OUT. That is the whole reason for the exercise, and it is the answer to "why not just start coding?" PROPERTY WHAT IT FORCES ───────────────────────────────────────────────────────────── partially observable internal state; belief over states rather than a known state stochastic probabilities and expected values; utility rather than goals alone sequential lookahead, planning, or learning about delayed reward dynamic a response deadline; layered architecture; anytime algorithms that can return a partial answer continuous discretisation, or methods designed for continuous spaces multi-agent modelling other agents; game-theoretic reasoning; randomisation has value unknown learning the transition model ───────────────────────────────────────────────────────────── THE COST OF GETTING IT WRONG, in both directions: ASSUMING THE ENVIRONMENT IS SIMPLER THAN IT IS · assuming full observability → an agent with no memory, which behaves erratically for reasons nobody can trace · assuming determinism → a plan that works in testing and fails in the field, because testing did not include the unlikely branch · assuming episodic → a system that optimises each decision alone and produces a poor sequence. This is a real and subtle failure: every individual recommendation is defensible and the overall study plan is bad. · assuming static → a system whose answer is correct and arrives too late ASSUMING IT IS MORE COMPLEX THAN IT IS · full probabilistic planning where a rule sufficed, so the system is expensive, slow and hard to explain for no gain · this error is less dangerous but far more common in student projects, and it is the same over-engineering the design-patterns topic warned about THE PRACTICAL DISCIPLINE: CLASSIFY THE ENVIRONMENT, WRITE THE SIX ANSWERS DOWN, AND RECORD ANY SIMPLIFICATION AS A SIMPLIFICATION. "We are treating the student as a passive environment although they are arguably an agent" is a sentence that saves an argument later — it is the assumption made explicit, which is exactly what the requirements unit means by documenting a domain assumption. A NOTE ON HOW ENVIRONMENTS ARE STUDIED: an ENVIRONMENT CLASS is a set of related task environments, and an ENVIRONMENT GENERATOR produces instances from it for evaluating agents. The distinction matters because an agent tuned to one instance may be worthless on the class — which is precisely the overfitting problem, arriving before machine learning is even introduced. AN AGENT SHOULD BE EVALUATED ON THE CLASS, NOT ON THE INSTANCES IT WAS DESIGNED AGAINST, and that is the same held-out-test-set discipline the history topic credited with ending the second AI winter.