;Abstract This paper proposes a deep learning algorithm by using “convolutional neural network algorithm” and “color space analysis” for real-time recognition, which combines background substraction, morphology, corner detection and regions of interest to achieve the flame tracking recognition of continuous image. In recent years, the rise of smart home monitoring systems and the technology based on real-time image have become more and more mature, so the preventive measure of fire is very different from traditional methods, such as smoke sensors and temperature sensors. Therefore, compared to the real-time image detection, the traditional methods are too slow to measure the prevention. As it happened recently, there are seven firefighters died on the fire disaster. If the surveillance image is used, there may not be this tragedy. As a consequence, the system discussed in this article can notify people by using real-time image to flame recognition before the flame begin to expand, so as to reduce the loss of personnel and property.