A Framework for Combined Recognition of Actions and Objects

Abstract

This paper proposes a novel approach to recognize actions and objects within the context of each other. Assuming that the different actions involve different objects in image sequences and there is one-to-one relation between object and action type, we present a Bayesian network based framework which combines motion patterns and object usage information to recognize actions/objects. More specifically, our approach recognizes high-level actions and the related objects without any body-part segmentation, hand tracking, and temporal segmentation methods. Additionally, we present a novel motion representation, based on 3D Haar-like features, which can be formed by depth, color, or both images. Our approach is also appropriate for object and action recognition where the involved object is partially or fully occluded. Finally, experiments show that our approach improves the accuracy of both action and object recognition significantly.