Optical flow on a rare landslide video
What and Why?
Debris flows are rapid flowing mixtures of sediment and water that occur in areas with steep topography worldwide (Major and Iverson, 1999). Often occurring after or during heavy rains, debris flows possess flow behavior intermediate to landslides and floods and have the capacity to carry boulders and trees. The damage caused by post-wildfire debris flows was recently on display in 2018 at Montecito, CA, killing 23 people and damaging 408 homes (Fig. 1; Kean et al., 2019). These flows are of fundamental interest to geoscience as the form of the upper
reaches of stream channels in steep areas are considered to be shaped primarily by debris
flows (Stock and Dietrich, 2006).
reaches of stream channels in steep areas are considered to be shaped primarily by debris
flows (Stock and Dietrich, 2006).
Figure 1. After of January 9th, 2018, Montecito debris flows, Santa Barbara, CA. Photo: Jim Logan
Despite the practical and scientific interest, debris flows are rarely systematically observed due to long return intervals and seemingly stochastic spatial occurrence (McCoy et al., 2010). To my knowledge, videos from the United State's Geological Survey's Chalk Cliffs, Colorado, monitoring site (Fig. 2; McCoy et al., 2010; Kean et al., 2015) account for the only publicly available, adequate-quality video data (i.e., HD resolution with stationary camera). Velocity is a basic property of debris flows and scales with other properties like depth of flow (Major and Iverson, 1999).
Figure 2. Downslope view of the USGS's upper monitoring station in the heavily instrumented Chalk Cliffs, Colorado, site. An HD camera is activated by rainfall.
State of the Art
(Large-scale) Particle Image Velocimetry (LS-PIV) has been applied to stream and debris flow
video data (Theule et al., 2018), but this method requires the
identification of tracer particles within the flow. Originally developed outside of computer vision by fluid mechanics workers to study turbulence, PIV is similar to correlation-based optical flow (Liu et al., 2015). Recently, Rapstine et al. (2020) used a proper optical flow algorithm to measure landslide velocity in a purpose-built flume. My thinking is that optical flow methods will continue to improve due its usage in deep learning tasks while LS-PIV/PIV will not in a relative sense. My goal was to use an established optical flow algorithm to extract velocity from a rare natural landslide video.
video data (Theule et al., 2018), but this method requires the
identification of tracer particles within the flow. Originally developed outside of computer vision by fluid mechanics workers to study turbulence, PIV is similar to correlation-based optical flow (Liu et al., 2015). Recently, Rapstine et al. (2020) used a proper optical flow algorithm to measure landslide velocity in a purpose-built flume. My thinking is that optical flow methods will continue to improve due its usage in deep learning tasks while LS-PIV/PIV will not in a relative sense. My goal was to use an established optical flow algorithm to extract velocity from a rare natural landslide video.
Challenges
I initially wanted to look at videos from the United States Geological Survey's debris flow flume (Logan et al., 2018) as well, but could not obtain any information about camera arrangement. Without some sense of the size of the field of view in the video, transformation of velocity magnitude in pixel units to real-world units is not possible. The feasibility of deep learning methods such as PWC-Net (Sun et al, 2018) were looked into, but the narrow use-cases of the codebases (i.e., two-frames or benchmark datasets only) made them too onerous (for a humble geoscientist) to implement.
Approach
I used a single video (1280x720; 24 fps) of a natural debris flow at the Chalk Cliffs site to explore the usage of optical flow. The flow occurred on July 4th, 2014, and has a published peak velocity measurement associated with it (Kean et al., 2015). I used OpenCV's Farneback (2003) dense optical flow implementation to obtain pixel displacement vectors in the x and y direction. This algorithm doesn't directly solve the canonical optical flow equation solved by Horn and Schunck (1981), but approximates it by polynomial expansion (Farneback, 2003). These vectors can be converted into velocity magnitude and angle by transforming into polar coordinates. I averaged the velocity magnitude and angle values over a grid of patches to denoise the data (Fig. 3).
Figure 3. First frame of the debris flow video showing an oblique view of the channel. The cyan-colored grid shows the scale of the patches used to average the velocity magnitude and angle data. The arrow down the center of the channel is ~5.5 m. The colored patches will be used during optimization (blue) and comparison (blue & orange).
Also, I optimized the algorithm by stepping through a bit of the parameter space of what appeared to be the most importance parameters: number of pyramids and Gaussian window size. The optimal parameter values (15 pyramids; 40x40 window) were used to calculate optical flow for the final run. Velocity magnitude in pixel units was roughly converted to meters per second using a published measurement of the centerline of the channel located in the camera's field of view and the frame rate of the video.
Results
Velocity magnitude through time was visualized by blending a grayscale version of the video and colorized magnitude normalized to a maximum value (5.0 m/s). Qualitatively, the video displays expected behavior such as slower velocities on the sides of the channel (Video 1). In the future, this could be an optimal way to show results for flows with dominant velocity angle. The velocity magnitude signal deteriorates at the end of the video. This could be due to higher-than-capturable velocities for the 24 fps video or a change in texture from great water content and/or smaller rock fragments.
OpenCV's implementation of the Farneback algorithm can make of coarse-to-fine pyramids with each pyramid reducing the size of a frame by 0.5. I ran it on the video with 0, 5, 10, and 15 pyramids with a constant Gaussian filter radius of 25 pixels. Velocity magnitude was extremely noisy within the blue patch of Fig. 1 when using no pyramids while 5, 10, and 15 pyramids plotted on top of each other (Fig. 4). Angle/initial angle (dimensionless) through time was used a measure of stability as the flow only goes one direction. No pyramids yields highly variable angles while 5, 10, and 15 pyramids all yield noticeable intervals of angular stability (Fig. 5). This is to be expected as the displacements in the video are large, and coarse-to-fine pyramid schemes reduce displacements at the coarse resolutions (Black and Anandan, 1996).
Figure 4. No pyramid levels equated to an unreliable velocity signal. Five, ten, and fifteen levels plotted on top of each other.
Figure 5. Angle/initial angle (dimensionless) shows that an interval of stability exists within the data and that 5 or more levels is needed to generate it.
The algorithm also can do its own Guassian filter to smooth results. I used a window of 5x5, 10x10, 20x20, and 40x40 with a constant 15 pyramid levels to see if there were noticeable differences in velocity magnitude (Fig. 6) and the angle/initial angle metric (Fig. 7). The largest window makes both measures less susceptible to high-frequency noise.
Figure 6. Larger Guassian window sizes displayed less variability in terms of velocity magnitude.
Figure 7. The largest Gaussian window size showed the most stability during the interval of stable angles.
The comparison with a published velocity magnitude of 3.2 m/s obtained by manual particle tracking (Kean et al., 2015) was satisfactory for data falling in the zone of stable angle ratios (Fig. 8). One must keep in mind that this video is from the wild with no forethought in terms computer vision methods (as far as I know), so the result is encouraging for future work. This method improves over manual tracking by providing a time series of velocity.
Figure 8. Velocity from optical flow compares well to a published, manually-derived estimate of velocity for this event.
Prospects
In the near term, more accurate optical-flow-derived velocity could be obtained at the Chalk Cliffs monitoring site using a camera with a higher frame rate and resolution. Higher resolution data could yield valuable insights on internal variability and turbulent structures present in debris flow (Takahashi, 2007). As optical flow an important computer vision task, it stands to reason that capabilities of algorithms will continue to improve (Tu et al., 2019). This trend is not immediately evident in the (LS)-PIV realm (Liu et al., 2015), so geomorphology and natural hazards researchers (the disciplines most interested in landslides) should adopt optical flow in the future. Furthermore, videos of natural landslides using vision systems specifically designed for optical flow could yield a benchmark dataset for purpose-built algorithms.
Code/Links
Optical flow code (Google Drive)
Video of surge 1 of 8 from the July 4th, 2014 debris flow at Chalk Cliffs, CO
Video of surge 1 of 8 from the July 4th, 2014 debris flow at Chalk Cliffs, CO
References
Major, J. J., and Iverson, R. M. (1999). Debris-flow deposition: Effects of pore-fluid pressure and friction concentrated at flow margins. Geological Society of America Bulletin, 111(10), 1424-1434.
Kean, J. W., Staley, D. M., Lancaster, J. T., Rengers, F. K., Swanson, B. J., Coe, J. A., and Lindsay, D. N. (2019). Inundation, flow dynamics, and damage in the 9 January 2018 Montecito debris-flow event, California, USA: Opportunities and challenges for post-wildfire risk assessment. Geosphere, 15(4), 1140-1163.
Stock, J. D., and Dietrich, W. E. (2006). Erosion of steepland valleys by debris flows. Geological Society of America Bulletin, 118(9-10), 1125-1148.
McCoy, S. W., Kean, J. W., Coe, J. A., Staley, D. M., Wasklewicz, T. A., and Tucker, G. E. (2010). Evolution of a natural debris flow: In situ measurements of flow dynamics, video imagery, and terrestrial laser scanning. Geology, 38(8), 735-738.
Kean, J. W., Coe, J. A., Coviello, V., Smith, J. B., McCoy, S. W., and Arattano, M. (2015). Estimating rates of debris flow entrainment from ground vibrations. Geophysical Research Letters, 42(15), 6365-6372.
Theule, J. I., Crema, S., Marchi, L., Cavalli, M., and Comiti, F. (2018). Exploiting LSPIV to assess debris-flow velocities in the field. Natural Hazards and Earth System Sciences, 18(1), 1-13.
Liu, T., Merat, A., Makhmalbaf, M. H. M., Fajardo, C., and Merati, P. (2015). Comparison between optical flow and cross-correlation methods for extraction of velocity fields from particle images. Experiments in Fluids, 56(8), 1-23.
Rapstine, T. D., Rengers, F. K., Allstadt, K. E., Iverson, R. M., Smith, J. B., Obryk, M. K., and Olsen, M. J. (2020). Reconstructing the velocity and deformation of a rapid landslide using multiview video. Journal of Geophysical Research: Earth Surface, 125(8), e2019JF005348.
Logan, M., Iverson, R.M., and Obryk, M.K., 2018, Video documentation of experiments at the USGS debris-flow flume 1992–2017 (ver 1.4, January 2018): U.S. Geological Survey Open-File Report 2007–1315, https://doi.org/10.3133/ofr20071315.
Sun, D., Yang, X., Liu, M. Y., and Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8934-8943).
Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363-370). Springer, Berlin, Heidelberg.
Horn, B. K., and Schunck, B. G. (1981). Determining optical flow. Artificial intelligence, 17(1-3), 185-203.
Black, M. J., and Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer vision and image understanding, 63(1), 75-104.
Takahashi, T. (2007). Debris flow: mechanics, prediction and countermeasures. Taylor & Francis.
Tu, Z., Xie, W., Zhang, D., Poppe, R., Veltkamp, R. C., Li, B., and Yuan, J. (2019). A survey of variational and CNN-based optical flow techniques. Signal Processing: Image Communication, 72, 9-24.
Kean, J. W., Staley, D. M., Lancaster, J. T., Rengers, F. K., Swanson, B. J., Coe, J. A., and Lindsay, D. N. (2019). Inundation, flow dynamics, and damage in the 9 January 2018 Montecito debris-flow event, California, USA: Opportunities and challenges for post-wildfire risk assessment. Geosphere, 15(4), 1140-1163.
Stock, J. D., and Dietrich, W. E. (2006). Erosion of steepland valleys by debris flows. Geological Society of America Bulletin, 118(9-10), 1125-1148.
McCoy, S. W., Kean, J. W., Coe, J. A., Staley, D. M., Wasklewicz, T. A., and Tucker, G. E. (2010). Evolution of a natural debris flow: In situ measurements of flow dynamics, video imagery, and terrestrial laser scanning. Geology, 38(8), 735-738.
Kean, J. W., Coe, J. A., Coviello, V., Smith, J. B., McCoy, S. W., and Arattano, M. (2015). Estimating rates of debris flow entrainment from ground vibrations. Geophysical Research Letters, 42(15), 6365-6372.
Theule, J. I., Crema, S., Marchi, L., Cavalli, M., and Comiti, F. (2018). Exploiting LSPIV to assess debris-flow velocities in the field. Natural Hazards and Earth System Sciences, 18(1), 1-13.
Liu, T., Merat, A., Makhmalbaf, M. H. M., Fajardo, C., and Merati, P. (2015). Comparison between optical flow and cross-correlation methods for extraction of velocity fields from particle images. Experiments in Fluids, 56(8), 1-23.
Rapstine, T. D., Rengers, F. K., Allstadt, K. E., Iverson, R. M., Smith, J. B., Obryk, M. K., and Olsen, M. J. (2020). Reconstructing the velocity and deformation of a rapid landslide using multiview video. Journal of Geophysical Research: Earth Surface, 125(8), e2019JF005348.
Logan, M., Iverson, R.M., and Obryk, M.K., 2018, Video documentation of experiments at the USGS debris-flow flume 1992–2017 (ver 1.4, January 2018): U.S. Geological Survey Open-File Report 2007–1315, https://doi.org/10.3133/ofr20071315.
Sun, D., Yang, X., Liu, M. Y., and Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8934-8943).
Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363-370). Springer, Berlin, Heidelberg.
Horn, B. K., and Schunck, B. G. (1981). Determining optical flow. Artificial intelligence, 17(1-3), 185-203.
Black, M. J., and Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer vision and image understanding, 63(1), 75-104.
Takahashi, T. (2007). Debris flow: mechanics, prediction and countermeasures. Taylor & Francis.
Tu, Z., Xie, W., Zhang, D., Poppe, R., Veltkamp, R. C., Li, B., and Yuan, J. (2019). A survey of variational and CNN-based optical flow techniques. Signal Processing: Image Communication, 72, 9-24.