With increasing crime rates in today’s world, there is a corresponding awareness for the necessity of detecting abnormal activity. Automation of abnormal Human behavior analysis can play a significant role in security by decreasing the time taken to thwart unwanted events and picking them up during the suspicion stage itself. With advances in technology, surveillance systems can become more automated than manual. Human Behavior Analysis although crucial, is highly challenging. Tracking and recognizing objects and human motion from surveillance videos, followed by automatic summarization of its content has become a hot topic of research. Many researchers have contributed to the field of automated video surveillance through detection, classification and tracking algorithms. Earlier research work is insufficient for comprehensive analysis of human behavior. With the introduction of semantics, the context of a surveillance domain may be established. Such semantics may extend surveillance systems to perform event-based behavior analysis relevant to the domain. This paper presents a survey on research on human behavior analysis with a scope of analyzing the capabilities of the state-of-art methodologies with special focus on semantically enhanced analysis.