Programming Assignment: Multiple Object Tracking
The goal of this part of the programming assignment is for you to learn more about the
practical issues that arise when designing a tracking system. You are asked to track moving
objects in video sequences, i.e., identifying the same object from frame to frame:
• You may consider two frames at a time (or, more ambitiously, use multiple hypothesis
tracking (MHT) with more than two frames).
• You may use a greedy, suboptimal bipartite matching algorithm (or, more ambitiously,
implement an optimal data association algorithm).
• To estimate the state of each tracked object, you may use an alpha-beta filter (or, more
ambitiously, a Kalman filter).
If you decide to use the advanced options (MHT, optimal data association, Kalman filter), you
are encouraged to use a 3rd party library.
We provided two datasets for you.
1) The bat dataset shows bats in flight, where the bats appear bright against a dark sky. We
included both grayscale and false-color images from this thermal image sequence; you may use
whichever images you prefer.
2) The cell dataset shows mouse muscle stem cells moving in hydrogel microwells, the
brightness of the pixels within the cells are very similar to the values of the background.
Just in case you had any trouble segmenting images and distinguishing objects (multi-object
labeling), we are providing segmentation and/or detections for the bat dataset.
The segmentation of the bat dataset is provided in a set of label maps. There is one number per
pixel, delimited by commas. Pixels with the value 0 are background. The maps are 1024 by 1024.
The detections are given in a comma delimited file, one for each frame. There is one point per
line. Each point is given as the X coordinate followed by the Y coordinate, delimited by commas.
For the cell dataset, no segmentation is provided. It's your task to do both segmentation and
tracking. Note: 1) The filopodia, which cells have during migration, are long "feet" that are
difficult to outline automatically. 2) Some cells spit into daughter cells. Since accurate cell
segmentation is very challenging, to obtain full credit, your focus should be on the multiobject tracking task (and detecting the birth of a new cell), while your segmentation result
can be relatively coarse.
You do not need to use our segmentation/detection if you would like to see the results using
your own algorithms. You may use any of your code from the previous assignments and any
library functions you wish.
Display the results of your tracking algorithm on top of the original images. Use different colors
to show that you successfully maintain track identity. Draw lines to show the history of the flight
trajectories.
In your write-up, you should discuss the following items:
1. Show your tracking results on some portion of the sequence. In addition to showing your
tracking results on an easy portion of the data, identify a challenging situation where
your tracker succeeds, and a challenging situation where your tracker fails.
2. How do you decide to begin new tracks and terminate old tracks as the objects enter
and leave the field of view?
3. What happens with your algorithm when objects touch and occlude each other, and how
could you handle this so you do not break track?
4. What happens when there are spurious detections that do not connect with other
measurements in subsequent frames?
5. What are the advantages and drawbacks of different kinematic models: Do you need to
model the velocity of the objects, or is it sufficient to just consider the distances between
the objects in subsequent frames?