A pinboard by
Mennatullah Siam

PhD, University of Alberta


Combining motion and appearance in convolutional neural networks

Motion cues is important in the analysis and reasoning of video sequences. An area that has not been researched enough in object detection and segmentation is handling sequences of images. Information gained from one image is restrictive than sequences of images. In order to comprehend if an object is static or dynamic its temporal motion has to be processed instead of one frame. Utilizing motion cues has great impact in autonomous driving application as one example. What would be the best method to incorporate motion with appearance? How to use that for the specific application of autonomous driving? These are the research questions that I'm addressing in my current research.


Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Abstract: Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep"' in that they can be compositional in spatial and temporal "layers". Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.

Pub.: 31 May '16, Pinned: 01 Jul '17

Convolutional Gated Recurrent Networks for Video Segmentation

Abstract: Semantic segmentation has recently witnessed major progress, where fully convolutional neural networks have shown to perform well. However, most of the previous work focused on improving single image segmentation. To our knowledge, no prior work has made use of temporal video information in a recurrent network. In this paper, we propose and implement a novel method for online semantic segmentation of video sequences that utilizes temporal data. The network combines a fully convolutional network and a gated recurrent unit that works on a sliding window over consecutive frames. The convolutional gated recurrent unit is used to preserve spatial information and reduce the parameters learned. Our method has the advantage that it can work in an online fashion instead of operating over the whole input batch of video frames. This architecture is tested for both binary and semantic video segmentation tasks. Experiments are conducted on the recent benchmarks in SegTrack V2, Davis, CityScapes, and Synthia. It is shown to have 5% improvement in Segtrack and 3% improvement in Davis in F-measure over a baseline plain fully convolutional network. It also proved to have 5.7% improvement on Synthia in mean IoU, and 3.5% improvement on CityScapes in mean category IoU over the baseline network. The performance of the RFCN network depends on its baseline fully convolutional network. Thus RFCN architecture can be seen as a method to improve its baseline segmentation network by exploiting spatiotemporal information in videos.

Pub.: 16 Nov '16, Pinned: 01 Jul '17