MACHINE VISION (computer vision) is the task of acquiring, processing and understanding images in order to produce information or decisions. THE FUNDAMENTAL DIFFICULTY: IMAGE FORMATION IS A PROJECTION FROM THREE DIMENSIONS TO TWO, AND PROJECTION LOSES INFORMATION. Vision is therefore an INVERSE PROBLEM — recovering what was lost — and inverse problems are ill-posed: MANY DIFFERENT SCENES PRODUCE THE SAME IMAGE. a small object nearby and a large one far away project identically a shadow and a dark surface are the same pixels a photograph of a chair and a chair are the same image THE SOURCES OF VARIATION that a recogniser must survive, and each is a reason the naive approach fails: VIEWPOINT the same object from a different angle has completely different pixels SCALE near and far ILLUMINATION the same surface under different light has different values; edges appear and vanish DEFORMATION a cat is not a rigid object OCCLUSION most of the object may be hidden BACKGROUND clutter that resembles the target INTRA-CLASS all chairs are chairs, and no two chairs VARIATION share a pixel pattern THE CONSEQUENCE: TEMPLATE MATCHING CANNOT WORK. Comparing an image against a stored picture of a cat fails as soon as the cat moves, turns or is lit differently — and that was the first thirty years of the field learning this the hard way. WHY IT IS AN AI PROBLEM, not an image-processing one: the same reason NLP is. RECOVERING THE SCENE REQUIRES ASSUMPTIONS ABOUT WHAT SCENES ARE LIKELY, and those assumptions are world knowledge. Humans use them constantly and unconsciously — which is the Moravec paradox again, since a task performed effortlessly by every two-year-old took the field five decades.

THE STAGES OF A VISION SYSTEM. Worth knowing even though modern systems learn several of them, because the stages remain how you describe what a system does. 1. IMAGE ACQUISITION A sensor produces a 2D array of intensity or colour values. A colour image is three such arrays (red, green, blue), so a 224 × 224 colour image is 224 × 224 × 3 = 150,528 numbers. 2. PRE-PROCESSING Improve the image for later stages: · NOISE REDUCTION by smoothing (a Gaussian filter) · CONTRAST ENHANCEMENT, histogram equalisation · GEOMETRIC CORRECTION for lens distortion · resizing and normalisation 3. FEATURE EXTRACTION — the classical heart of the pipeline EDGE DETECTION finds sharp intensity changes, which usually correspond to object boundaries. Implemented by CONVOLUTION with a small kernel: SOBEL horizontal gradient SOBEL vertical −1 0 +1 −1 −2 −1 −2 0 +2 0 0 0 −1 0 +1 +1 +2 +1 NOTE THAT THE WEIGHTS SUM TO ZERO. That is deliberate: a region of uniform intensity produces a response of zero, so the filter responds only to CHANGE. Every edge detector has this property. Also: CORNER detection (Harris), BLOB detection, TEXTURE measures, and hand-designed descriptors such as SIFT and HOG that were the state of the art before 2012. 4. SEGMENTATION Divide the image into meaningful regions — separating objects from background and from each other. Approaches: thresholding, region growing, edge-based, clustering, and now learned segmentation. 5. REPRESENTATION AND DESCRIPTION Describe each region: shape, size, moments, colour histograms, relationships to other regions. 6. RECOGNITION AND INTERPRETATION Assign labels, and build a description of the scene. CONVOLUTION IS THE KEY OPERATION, and the arithmetic is examinable. For an n × n input, an f × f filter, padding p and stride s, the output size is ⌊(n + 2p − f)/s⌋ + 1 WORKED: 32×32 input, 3×3 filter, no padding, stride 1 → 30×30 32×32 input, 3×3 filter, padding 1, stride 1 → 32×32 (padding 1 with a 3×3 filter preserves the size — the standard choice, called "same" padding) 224×224 input, 7×7 filter, padding 3, stride 2 → 112×112 (stride 2 halves the resolution) WHY CONVOLUTION RATHER THAN A FULLY CONNECTED LAYER — the parameter argument, and it is decisive: a fully connected layer from a 224×224×3 image to 1,000 units needs 224 × 224 × 3 × 1,000 = 150,528,000 weights one convolutional layer with 64 filters of size 3×3×3 needs (3 × 3 × 3 + 1) × 64 = 1,792 weights A FACTOR OF ABOUT 84,000. And the saving is not merely storage: the convolutional layer embodies two assumptions that are TRUE of images and that the dense layer has to learn from scratch — LOCALITY: nearby pixels are related, distant ones less so TRANSLATION EQUIVARIANCE: a feature detector useful in one part of the image is useful everywhere, so the weights are SHARED THOSE TWO ASSUMPTIONS ARE THE WHOLE REASON CONVOLUTIONAL NETWORKS WORK. They are prior knowledge about images, built into the architecture rather than learned.

THE STANDARD TASKS, in increasing difficulty — a hierarchy worth knowing because the words are used precisely: IMAGE CLASSIFICATION one label for the whole image "this is a cat" OBJECT LOCALISATION one label plus a bounding box OBJECT DETECTION many objects, each with a class and a box SEMANTIC SEGMENTATION a class label for EVERY PIXEL, but two cats are one "cat" region INSTANCE SEGMENTATION every pixel labelled AND separate objects distinguished POSE ESTIMATION the positions of an object's or a body's parts DEPTH ESTIMATION distance per pixel — recovering the dimension the projection lost OPTICAL FLOW per-pixel motion between frames 3D RECONSTRUCTION building a scene model from several views ACTIVITY RECOGNITION what is happening over time IMAGE CAPTIONING a sentence describing the image — vision and NLG combined THE APPROACHES, and the historical break: CLASSICAL (pre-2012): hand-designed features (SIFT, HOG) fed into a classifier (support vector machine). Every feature was invented by a researcher, and progress came from better features. DEEP LEARNING (2012–): CONVOLUTIONAL NEURAL NETWORKS learn the features from data. The layers form a hierarchy — early layers detect edges and colour blobs middle layers detect textures and parts late layers detect object-like configurations NOBODY DESIGNED THAT HIERARCHY; it emerges from training, and it resembles what is known of the visual cortex. THE 2012 RESULT: AlexNet won the ImageNet competition with a top-5 error of about 16% against roughly 26% for the best hand-engineered system. A gap that large in one year ended the debate, and within a few years error rates fell below the human benchmark of about 5% on that dataset. THE THREE ENABLERS ARRIVED TOGETHER, which is why it happened then and not earlier: LARGE LABELLED DATA — ImageNet, over a million labelled images GPU COMPUTATION — convolution is embarrassingly parallel ALGORITHMIC IMPROVEMENTS — ReLU activations, dropout, better initialisation NONE ALONE WOULD HAVE SUFFICED, which is a useful corrective to accounts that credit only the algorithms. LIMITATIONS THAT REMAIN, and they matter for deployment: · ADVERSARIAL EXAMPLES: imperceptible pixel changes flip a confident classification, which shows the learned function is not the one we assumed · DISTRIBUTION SHIFT: a model trained on daytime photographs fails at dusk · DATA HUNGER, and the LABELLING cost · no genuine 3D or physical understanding — a model may label a floating chair as a chair without noticing it is impossible · BIAS: face recognition accuracy has been shown to vary substantially across skin tone and gender in audited systems, which is a training-data problem with real consequences · NO EXPLANATION: saliency maps indicate where the model looked, not why THE HONEST SUMMARY: PERCEPTION IS NOW THE STRONGEST PART OF AI AND THE LEAST UNDERSTOOD. Systems exceed human accuracy on narrow benchmarks while failing on inputs a human would find trivial, which means benchmark performance and reliability are different claims — the same distinction the AI applications topic drew between accuracy and verifiability.