2 Application: From amateur basketball video footage to player trajectories

In this section on visual data, we use a motivating example focused on recovering player trajectories from fixed-camera amateur basketball video. This will connect ideas from camera representations, image analysis, object detection, and tracking across frames. The goal is to demonstrate how to use computer vision techniques in a practical application. In fact, this example is inspired by some consulting work I and some students did for a company specialising in automated video footage for amateur basketball. Again, this is a case where modern agentic AI tools are rapidly changing what is possible and what the most effective approach is, but here we will focus on a more traditional approach to the problem. Currently, the best approach is probably to (use a modern agent to!) re-use and/or train a specialised model such as what we will do in the next section, but this may not be the case in the near future.

2.1 The data and problem

Here we will use data from TrackID3x3 (Yamada et al. 2025), a public dataset for multi-object tracking in 3×3 basketball. It contains data subsets from three different settings: Indoor, Outdoor and Drone. Each has a different camera viewpoint and different conditions. We will first focus on the Indoor dataset, which uses a fixed camera. The indoor videos are of a controlled coaching, ‘role sharing’ exercise, rather than an actual game. We begin with a particular indoor clip, basket_S1T4_pre, which means ‘Session 1, Trial 4, recorded before the coaching intervention’.

Here, our task as machine learning engineers is to determine the identities and locations of the six active on-court players in the video, tracking them over time. In addition to ‘image coordinates’ we also want to determine the physical court coordinates.

The TrackID3x3 dataset is publicly available under a CC BY 4.0 licence. Although you can hence download this yourself from the original source, for this course we provide a bespoke course extract. This course extract contains a silent copy of the selected video, minimal annotations and court information required for the exercises, and selected precomputed detector and tracker outputs. As per licensing requirements, the README in the dataset folder provides the full dataset citation, source link, licence and a record of changes made in preparing the extract.

2.2 Notes on approach

After initial exploration of the data, we could go straight to image-based object detection and tracking. However, we will first consider a model of the camera and the court. This represents a perception-oriented, physics-based approach to the problem, and will help us understand the geometry of the scene and the relationship between image coordinates and physical coordinates on the court. This is a big topic, and we will only scratch the surface of physics-based approaches to visual data here. We will then move on to our main, data-driven approach using object detection and tracking algorithms, while maintaining a mapping between image coordinates and court coordinates. This is also the approach taken by the authors of the TrackID3x3 dataset when providing baseline tracking results for the dataset.

As mentioned in the overview, we will focus on a modular approach to tracking, called tracking-by-detection, which involves first detecting objects in each frame and then associating those detections across frames to form tracks.

Yamada, Kazuhiro, Li Yin, Qingrui Hu, Ning Ding, Shunsuke Iwashita, Jun Ichikawa, Kiwamu Kotani, Calvin Yeung, and Keisuke Fujii. 2025. “TrackID3x3: A Dataset and Algorithm for Multi-Player Tracking with Identification and Pose Estimation in 3x3 Basketball Full-Court Videos.” In Proceedings of the 8th International ACM Workshop on Multimedia Content Analysis in Sports, 163–73. ACM. https://doi.org/10.1145/3728423.3759400.

# Application: From amateur basketball video footage to player trajectories In this section on visual data, we use a motivating example focused on recovering player trajectories from fixed-camera amateur basketball video. This will connect ideas from camera representations, image analysis, object detection, and tracking across frames. The goal is to demonstrate how to use computer vision techniques in a practical application. In fact, this example is inspired by some consulting work I and some students did for a company specialising in automated video footage for amateur basketball. Again, this is a case where modern agentic AI tools are rapidly changing what is possible and what the most effective approach is, but here we will focus on a more traditional approach to the problem. Currently, the best approach is probably to (use a modern agent to!) re-use and/or train a specialised model such as what we will do in the next section, but this may not be the case in the near future. ## The data and problem Here we will use data from **TrackID3x3** [@trackid3x3], a public dataset for multi-object tracking in 3×3 basketball. It contains data subsets from three different settings: *Indoor*, *Outdoor* and *Drone*. Each has a different camera viewpoint and different conditions. We will first focus on the *Indoor* dataset, which uses a fixed camera. The indoor videos are of a controlled coaching, 'role sharing' exercise, rather than an actual game. We begin with a particular indoor clip, `basket_S1T4_pre`, which means 'Session 1, Trial 4, recorded before the coaching intervention'. Here, our task as machine learning engineers is to **determine the identities and locations of the six active on-court players in the video, tracking them over time**. In addition to 'image coordinates' we also want to determine the physical **court coordinates**. The TrackID3x3 dataset is publicly available under a CC BY 4.0 licence. Although you can hence download this yourself from the original source, for this course we provide a bespoke **course extract**. This course extract contains a silent copy of the selected video, minimal annotations and court information required for the exercises, and selected precomputed detector and tracker outputs. As per licensing requirements, the README in the dataset folder provides the full dataset citation, source link, licence and a record of changes made in preparing the extract. ## Notes on approach After initial exploration of the data, we could go straight to image-based object detection and tracking. However, we will first consider a **model of the camera and the court**. This represents a **perception-oriented, physics-based** approach to the problem, and will help us understand the geometry of the scene and the relationship between image coordinates and physical coordinates on the court. This is a big topic, and we will only scratch the surface of physics-based approaches to visual data here. We will then move on to our main, **data-driven** approach using **object detection** and **tracking** algorithms, while maintaining a mapping between image coordinates and court coordinates. This is also the approach taken by the authors of the TrackID3x3 dataset when providing baseline tracking results for the dataset. As mentioned in the overview, we will focus on a modular approach to tracking, called **tracking-by-detection**, which involves first detecting objects in each frame and then associating those detections across frames to form tracks.