Training optical flow models

When training optical flow models on synthetic data, it is important to compare the distributions of flow magnitude between the real and synthetic datasets. For each valid pixel in the flow map, calculate the magnitude of the flow vector and look at the distribution of pixels in the dataset (because there are so many pixels, it’s often also useful to calculate the mean flow magnitude per frame and look at the distribution over the frames).

Flow magnitude is largely driven by scene composition: highway scenes have very high flow magnitude (at the edges of the frame and on oncoming vehicles) while residential scenes will have low flow magnitude. We have found that synthetic data is most beneficial in training if the flow magnitude distribution is similar to the distribution in the real data.

The flow magnitude of synthetic data can be altered in a number of ways:

• by changing the speed of the ego vehicle
• by cropping to focus on high or low magnitude flow during training
• by altering the frame rate (flow between frames is larger at 5Hz than at 10Hz)

Other tips for training optical flow:

• Pretraining on synthetic and finetuning on real data is generally more effective than joint training.
• If you are already pretraining on Flying Chairs and Flying Things, adding pretraining on PD data just before finetuning on real data worked best in our experiments.
• Learning rate tuning is very important in optical flow. After pretraining with PD data, we found that lower learning rates during finetuning work best.
• Even though real world optical flow data is usually sparse, we found that training on dense synthetic data works better than adding a sparse mask, even if that mask is informed by the real world data (e.g. the LiDAR scan pattern).