Development of high-performance, low-cost Unmanned Aerial Vehicles paired with rapid progress in vision-based perception systems herald a new era of autonomous flight systems with mission-ready capabilities. However, modern autonomous systems are not yet fully capable of coping with dynamically changing environments, and one of the key features of a robust autonomous flight systems is a mid-air collision avoidance strategy. This work focuses on the design, implementation and examination of vision-based moving object detection and tracking system with decision-making capabilities for performing evasive maneuvers. In detail, the geometrical transformation of a 2-D image structure due to the movement of the camera is approximated as a 2-D Homography transform, and this approximation is utilized to detect independent moving objects in the scene, which are then tracked using a Kalman Filter-based tracking system. Risk assessment and decision-making for performing evasive maneuvers is performed with data from stereo camera-based depth estimation. Examination was performed with a quadrotor UAV equipped with this system, and the performance was comparable to state-of-the-art systems without any additional sensors.