The era of digitization has paved the way for the emergence of multimedia technologies. The multimedia applications use a large video database. Applications such as video surveillance, traffic monitoring and live streaming of events need continuous monitoring of the video. Thus the video surveillance system for tracking multiple target objects from videos is one of the important research works. Video surveillance has many challenges such as inter-object occlusion, object confusion, different posing, environment with heavy clutter, the small size of objects, similar appearance among objects, and interaction among the multiple objects. The hybrid tracking model utilizes the spatial tracking model, based on the tangential weighed function and the visual tracking model based on the second derivative model.