Vision-based traffic surveillance has been one of the most promising fields for improvement and research. Still, many challenging problems remain unsolved, such as addressing vehicle occlusions and reducing false detection. In this work, a method for vehicle detection and tracking is proposed. The proposed model considers background subtraction concept for moving vehicle detection but unlike conventional approaches, here numerous algorithmic optimization approaches have been applied such as multi-directional filtering and fusion based background subtraction, thresholding, directional filtering and morphological operations for moving vehicle detection. In addition, blob analysis and adaptive bounding box is used for Detection and Tracking. The Performance of Proposed work is measured on Standard Dataset and results are encouraging.