1 Overview

One of the aims of this course is to equip you with machine learning tools appropriate to a range of types of data, often called data modalities, that you might encounter in engineering practice. In machine learning (which can be thought of as a subset of artificial intelligence/AI), it is common to distinguish between tabular, time-series, image, video and text data. These categories are not mutually exclusive, however. For example, time series, text and video can all be thought of as sequence-type data, and we can, e.g., represent an image as a row of a tabular design matrix by treating its pixel values as features. However, the distinction can be useful as different types of data have different relationships and structure that we may want our models to use or preserve.

In learning how to address different data modalities, one obvious goal is to learn a collection of modality-specific tools; another is drawing out common ways of thinking and exposing you to how to think like a machine learning practioner. Because most of you are also engineers (or mathematicians) rather than, say, computer scientists, we also want to integrate the machine learning way of thinking into our usual engineering thinking. Furthermore, splitting by data modality is only one way to approach our goals; many recent AI models are multi-modal and the common element is in the computational method or mathematics rather than data type.

This half of the course will cover the data modalities of image, video, and text data. We lump images and video under visual data and can also refer to text data as language data. In addition, we will look at agentic AI systems that have arisen out of language models in particular, but, as indicated, are increasing multi-modal across language, images, video, and so on, data. Ultimately a true artificially intelligent agent would be expected to be able to see things via a vision system and communicate via language (as well as, e.g., hear things), and, importantly, make decisions and take actions in the world.

Aspects of this thinking will have been covered Engineering Machine Learning section of ENGSCI 205 (as well as the first half of the course), which many of you will have seen. As times we will draw on some of that material and so I have made the 205 course material available on Canvas.

1.1 A note on the present landscape of AI

There is currently something like a revolution occurring in and because of AI. This is largely due to the development of large language models (LLMs) and their associated agentic AI systems. We will learn a little about these later in the course. One point worth making: this is a big deal!

Things are rapidly changing and it is hard to know what will still be relevant even in a few months from now. One short to medium term trend, however, is using these general agentic systems to train cost-effective specialised models for particular tasks. For example, frontier AI systems like GPT-5 Astra can now probably do many of the vision tasks we will look at. However, it is still currently more cost-effective to use systems like those we will look at in this course for many vision tasks. An incresingly popular approach is to instead use a system like GPT-5 Astra to help label and train cheaper models. That is, the main change is many of the steps we might have done manually in the past are now being done by these agentic systems. This is a big change, but the underlying models being trained are still often similar. However, this may not be the case in the near future.

For some interesting context on modern machine learning and AI I recommend:

The Bitter Lesson by Richard Sutton, a computer scientist and one of the founders of reinforcement learning. A now famous essay that observes that simple, general search and learning algorithms that scale with computation have consistently outperformed specialised, human-designed solutions in machine learning and AI in the long run, and that this is likely to continue to be the case.
The Scaling Era: An Oral History of AI, 2019-2025 by Dwarkesh Patel with Gavin Leech. Based on a series of interviews with leading AI researchers and company founders, including Anthropic CEO Dario Amodei, DeepMind cofounder Demis Hassabis, OpenAI cofounder Ilya Sutskever, and many others. I recommend the audio book for actual interview excerpts. Basic point is LLMs and agentic AI systems can be scaled up with more data and computation, a la the Bitter Lesson, and that this may be enough to achieve ‘artificial general intelligence’.
AI 2027 by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean. To see how some are thinking about AI. Quote: “We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. We wrote a scenario that represents our best guess about what that might look like.” Whether you agree or not, this is the kind of thing many in Silicon Valley and the frontier labs are thinking.

# Overview One of the aims of this course is to equip you with machine learning tools appropriate to a range of types of data, often called **data modalities**, that you might encounter in engineering practice. In machine learning (which can be thought of as a subset of artificial intelligence/AI), it is common to distinguish between **tabular**, **time-series**, **image**, **video** and **text** data. These categories are not mutually exclusive, however. For example, time series, text and video can all be thought of as sequence-type data, and we can, e.g., represent an image as a row of a tabular design matrix by treating its pixel values as features. However, the distinction can be useful as different types of data have different relationships and structure that we may want our models to use or preserve. In learning how to address different data modalities, one obvious goal is to learn a collection of modality-specific tools; another is drawing out common ways of thinking and exposing you to **how to think like a machine learning practioner**. Because most of you are also engineers (or mathematicians) rather than, say, computer scientists, we also want to integrate the machine learning way of thinking into our usual engineering thinking. Furthermore, splitting by data modality is only one way to approach our goals; many recent AI models are **multi-modal** and the common element is in the computational method or mathematics rather than data type. This half of the course will cover the data modalities of **image**, **video**, and **text** data. We lump images and video under **visual** data and can also refer to text data as **language** data. In addition, we will look at **agentic** AI systems that have arisen out of language models in particular, but, as indicated, are increasing multi-modal across language, images, video, and so on, data. Ultimately a true artificially intelligent **agent** would be expected to be able to see things via a vision system and communicate via language (as well as, e.g., hear things), and, importantly, make **decisions** and take **actions** in the world. Aspects of this thinking will have been covered Engineering Machine Learning section of ENGSCI 205 (as well as the first half of the course), which many of you will have seen. As times we will draw on some of that material and so I have made the 205 course material available on Canvas. ## A note on the present landscape of AI There is currently something like a revolution occurring in and because of AI. This is largely due to the development of **large language models** (LLMs) and their associated **agentic** AI systems. We will learn a little about these later in the course. One point worth making: **this is a big deal!** Things are rapidly changing and it is hard to know what will still be relevant even in a few months from now. One short to medium term trend, however, is using these general agentic systems to train cost-effective specialised models for particular tasks. For example, frontier AI systems like GPT-5 Astra can now probably do many of the vision tasks we will look at. However, it is still currently more cost-effective to use systems like those we will look at in this course for many vision tasks. An incresingly popular approach is to instead use a system like GPT-5 Astra to help label and train cheaper models. That is, the main change is many of the steps we might have done manually in the past are now being done by these agentic systems. This is a big change, but the underlying models being trained are still often similar. However, this may not be the case in the near future. For some interesting context on modern machine learning and AI I recommend: - [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) by Richard Sutton, a computer scientist and one of the founders of reinforcement learning. A now famous essay that observes that simple, general search and learning algorithms that scale with computation have consistently outperformed specialised, human-designed solutions in machine learning and AI in the long run, and that this is likely to continue to be the case. - [The Scaling Era: An Oral History of AI, 2019-2025](https://press.stripe.com/scaling) by Dwarkesh Patel with Gavin Leech. Based on a series of interviews with leading AI researchers and company founders, including Anthropic CEO Dario Amodei, DeepMind cofounder Demis Hassabis, OpenAI cofounder Ilya Sutskever, and many others. I recommend the audio book for actual interview excerpts. Basic point is LLMs and agentic AI systems can be scaled up with more data and computation, a la the Bitter Lesson, and that this may be enough to achieve 'artificial general intelligence'. - [AI 2027](https://ai-2027.com/) by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean. To see how some are thinking about AI. Quote: "We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. We wrote a scenario that represents our best guess about what that might look like." Whether you agree or not, this is the kind of thing many in Silicon Valley and the frontier labs are thinking.