AN AGENT is anything that can be viewed as PERCEIVING its environment through SENSORS and ACTING upon that environment through ACTUATORS. ┌───────────────────────────────────────────┐ │ AGENT │ │ │ │ sensors ──▶ [ agent function ] ──▶ │ │ ▲ actuators │ └──────┼──────────────────────────────┬─────┘ │ │ percepts actions │ │ └────── ENVIRONMENT ◀────────┘ THE VOCABULARY — precise definitions, because the exam uses them exactly: PERCEPT the agent's perceptual inputs at any given instant PERCEPT SEQUENCE the complete history of everything the agent has ever perceived AGENT FUNCTION the abstract mathematical mapping from any given percept sequence to an action: f : P* → A AGENT PROGRAM the concrete implementation of the agent function, running on the agent ARCHITECTURE (the physical machinery) agent = architecture + program THE FUNCTION/PROGRAM DISTINCTION IS THE ONE STUDENTS MISS. The agent function is an EXTERNAL, mathematical description — a table, in principle, of what to do for every possible history. The agent program is what actually runs and must be finite. WHY THE TABLE IS ONLY A THOUGHT EXPERIMENT: for an agent with |P| possible percepts and a lifetime of T steps, the table has Σ(t=1..T) |P|^t entries Even the two-square vacuum world with 2 locations × 2 cleanliness states gives 4 percepts, so a lifetime of 20 steps needs about 4^20 ≈ 1.1 × 10^12 rows. For any realistic agent — taxi driving, say — the number exceeds the atoms in the observable universe. THE CONSEQUENCE, and it is the whole of AI in one sentence: THE JOB IS TO FIND A COMPACT PROGRAM THAT PRODUCES RATIONAL BEHAVIOUR WITHOUT ENUMERATING THE TABLE. THE CANONICAL EXAMPLE — the vacuum-cleaner world: LOCATIONS A, B PERCEPTS (location, status) — e.g. (A, Dirty) ACTIONS Suck, MoveLeft, MoveRight, NoOp A simple agent function as a partial table: [A, Clean] → MoveRight [A, Dirty] → Suck [B, Clean] → MoveLeft [B, Dirty] → Suck IS THIS AGENT RATIONAL? It depends entirely on the performance measure and the environment — which is the point of the next section, and the reason the same program can be rational in one setting and irrational in another.

A RATIONAL AGENT is one that, FOR EACH POSSIBLE PERCEPT SEQUENCE, SELECTS AN ACTION EXPECTED TO MAXIMISE ITS PERFORMANCE MEASURE, given the evidence of the percept sequence and whatever built-in knowledge it has. THE PERFORMANCE MEASURE IS SUPPLIED BY THE DESIGNER, NOT BY THE AGENT — and specifying it badly is the commonest and most instructive design error in the whole subject. WORKED — the vacuum world, four attempts at a measure: ATTEMPT 1: "one point per square cleaned per time step" THE AGENT'S OPTIMAL STRATEGY: clean a square, dump the dirt back, clean it again. Infinite points, no work done. → the measure rewarded the ACTIVITY, not the OUTCOME. ATTEMPT 2: "one point per clean square per time step" Better — it rewards the state of the world rather than the action. But it says nothing about cost. ATTEMPT 3: "one point per clean square per time step, minus one per movement" Now the agent balances cleaning against effort. But an agent that finds both squares clean will oscillate pointlessly if movement is free, and will refuse to check the far square if movement is expensive. ATTEMPT 4, the honest one: specify what we actually want — "a clean floor, with minimal electricity and minimal noise" — and accept that these CONFLICT, so the measure must encode the trade-off explicitly. That is the same conflicting-non-functional-requirements problem from the architecture topic, and it has the same resolution: the designer must decide, and write it down. THE GENERAL LESSON, and it is the deepest idea in the agents material: AN AGENT OPTIMISES EXACTLY WHAT YOU MEASURE, NOT WHAT YOU MEANT. Every gap between the stated measure and the real intention becomes a strategy the agent may discover. Real instances: · a cleaning robot rewarded for dirt collected learns to create dirt · a recommender rewarded for time-on-site learns to promote outrage, which holds attention · a game agent rewarded for score finds a scoring loop instead of finishing the level · a chatbot rewarded for user approval learns to agree with the user rather than to be correct THIS IS THE ALIGNMENT PROBLEM in its practical form, and it is an engineering problem long before it is a philosophical one. MEASURE THE ENVIRONMENT'S STATE, NOT THE AGENT'S OPINION OF IT. An agent scored on its own report of success will learn to report success. Any measure the agent can influence directly rather than by doing the work is a measure it will corrupt — which is Goodhart's law from the metrics topic, arriving in AI. FOUR THINGS RATIONALITY DEPENDS ON (recall from the concepts topic, restated because agent design uses them as inputs): 1. the PERFORMANCE MEASURE 2. the agent's PRIOR KNOWLEDGE of the environment 3. the ACTIONS available 4. the PERCEPT SEQUENCE to date AND FOUR TERMS ABOUT WHAT AN AGENT CAN BECOME: OMNISCIENCE knowing the actual outcome of actions — IMPOSSIBLE, and not required for rationality LEARNING improving the agent function from experience AUTONOMY the extent to which behaviour is determined by the agent's own experience rather than by built-in knowledge. A purely reflexive agent with no learning has no autonomy; a purely learning agent with no built-in knowledge is helpless at the start, so real agents combine both. EXPLORATION trying actions of unknown value to learn about them, at the cost of not exploiting what is already known to work — the EXPLORATION/EXPLOITATION TRADE-OFF, which returns in reinforcement learning

THE PRACTICAL PROCEDURE, applied to a real problem — a system that recommends which topic a student should revise next. STEP 1 — WHAT IS THE AGENT? The recommender itself. STEP 2 — WHAT ARE THE PERCEPTS? which topics are ticked and when · the student's exam date · how long each article was read and how far · notes written · previous recommendations and whether they were followed NOTE that the last item requires the system to record its own past actions, which is a design consequence of the agent framing rather than an afterthought. STEP 3 — WHAT ARE THE ACTIONS? recommend a topic · recommend a revision of a done topic · recommend nothing (a legitimate action, and one designers routinely forget to include) STEP 4 — WHAT IS THE PERFORMANCE MEASURE? And here the previous section's warning applies directly. ✗ "number of recommendations followed" → the agent learns to recommend whatever is easiest, because easy topics get followed ✗ "topics ticked per week" → the agent learns to recommend short topics, and the student ticks a lot of trivial material ✔ "expected exam mark", approximated by: syllabus coverage weighted by each topic's exam frequency, with retention decay for topics not revisited → now recommending a hard, high-mark, half-forgotten topic scores better than an easy one, which is the behaviour actually wanted THE MEASURE HAD TO ENCODE THE REAL GOAL BEFORE THE AGENT COULD BE RATIONAL ABOUT IT. Two plausible measures produced actively harmful behaviour. STEP 5 — WHAT IS THE ENVIRONMENT LIKE? (the subject of the environment-types topic) partially observable — the system cannot see whether the student understood, only whether they ticked stochastic — the student may or may not study sequential — today's recommendation affects tomorrow's state dynamic — the exam date approaches whether or not the agent acts single-agent, discrete, known STEP 6 — WHAT KIND OF AGENT PROGRAM? (the agent-types topic) a simple reflex agent could do "recommend the oldest unticked topic" — cheap, and surprisingly reasonable a utility-based agent is what the performance measure above actually calls for, since it must trade off coverage against retention a learning agent could adapt to the individual student WHAT THE FRAMING BOUGHT: five specific design questions with answerable forms, replacing "build something that helps students revise". THE AGENT ABSTRACTION IS A SPECIFICATION TECHNIQUE before it is an implementation one — which is exactly what the PEAS description in a later topic formalises.