Visual data

Here we consider what might be called visual data: images, videos etc. The study of visual data and vision more generally is a large and multidisciplinary field, with a long history in computer science, signal processing, and machine learning, in addition to experimental psychology, neuroscience, and biology. For good background, see for example Szeliski’s Computer Vision: Algorithms and Applications and Torralba, Isola and Freeman’s Foundations of Computer Vision.

Similar to our discussion of engineering vs machine learning modelling in ENGSCI 205, one important division of approaches to visual data is between those that focus on explicit models of perception and those that focus on data-driven prediction. The former is historically associated with computer vision and artificial intelligence, while the latter is historically associated with signal processing and machine learning, though the division has never been absolute and their boundaries continue to blur. In practice, many modern approaches to visual data combine elements of both. We will primarily focus on the machine learning, data-driven approach, but relate this back to physics-based ideas of perception and modelling where appropriate, particularly in the context of camera models and the geometry of images and videos.

Our motivating example will be object tracking in video data, e.g. sports players in game footage or animals in drone footage for ecological monitoring. We will use tracking-by-detection: apply an image-based object detector to each frame, and then associate detections across time. This modular approach remains widely used and makes the separate sources of error relatively easy to inspect.

Modern alternatives, such as transformer-based trackers, often aim to jointly learn detection and temporal association. These blur the boundary between detection and association, but must still answer the key question of which observations at different times belong to the same object. Also, as mentioned above, frontier AI systems like GPT-5 Astra can now probably do many of the vision tasks we will look at. However, it is still currently more cost-effective to use systems like those we will look at in this course for many vision tasks, potentially trained with the help of a system like GPT-5 Astra.