An Early Warning System for Curved Road Based on OV7670 Image Acquisition and STM32

Xiaoliang Wang , Wenhua Song Bowei ZhangBrandon Mausler and Frank Jiang

Abstract: Nowadays, the number of vehicles in China has increased significantly. The increase of the number of vehicles has also led to the increasingly complex traffic situation and the urgent safety measures in need. However, the existing early warning devices such as geomagnetic, ultrasonic and infrared detection have some shortcomings like difficult installation and maintenance. In addition, geomagnetic detection will damage the road surface, while ultrasonic and infrared detection will be greatly affected by the environment. Considering the shortcomings of the existing solutions, this paper puts forward a solution of early warning for vehicle turning meeting based on image acquisition and microcontrollers. This solution combines image acquisition and processing technology, which uses image sensor to perceive traffic condition and image data analysis algorithm to process perceived image, and then utilize LED display screen to issue an early warning.

Keywords: Curve traffic, turning meeting, early warning, image identification, vehicle detection.

1 Introduction

With the rapid development of Chinese economy, the number of vehicles keeps increasing. However, the conventional transportation infrastructures have been unable to meet the higher requirements of safety, stability and flexibility. Especially in some narrow and old curved road, there are great potential safety hazards.

In China, mountainous areas occupy 2/3 of the land area, and mountain roads also occupy more than half of the total mileage of highways [Cui (2018)]. In mountainous areas,curved roads are very common. Traffic accidents and casualties occur easily when turning meet, but at present 50% of curved road do not have early warning solutions, and 15% of intersections only have wide-angle convex mirror [Xu, Fang and Chen (2017)].These convex mirrors work as early warning devices are used to warn drivers.

By the investigation from some drivers [Qi (2016); Tao (2016)], it is found that 65% of drivers like to drive in the middle of the road in the mountain curved roads, and only 5%of drivers obey the traffic rules [Zhu (2014)]. However, the optical convex mirror solution has some shortcomings, such as insufficient early warning scope, image distortion, being easily obscured, night dysfunction. These shortcomings will make drivers unable to get real traffic information quickly and make correct judgments at the first time, which will lead to traffic accidents.

In recent years, with the development of Intelligent Transportation System (ITS), a large number of equipment for road safety have been put into use, such as high-speed guardrail and deceleration belt. However, these road equipments only reduce vehicle losses after the accident. Some vehicle equipments inside such as traffic recorder, airbag are mostly used to record the vehicle's travel process and reduce the damage after collision.Similarly, they do not provide early warning function or link with the road warning system [Wang (2016)]. In order to solve these problems, this paper proposes an early warning system in road for vehicle meeting in curved road which is linked with vehicles.To improve the conventional early warning system of turning road, vehicle edge detection and night frame difference algorithm are added to effectively improve the detection effect of the system in dark night and bad weather environment. At the same time, the system can communicate with the vehicular software in real time to forecast the turning traffic situation on curved roads and minimize the driver's misjudgment operation.The construction of this paper is as follows. The second chapter introduces the related work. Chapter 3 introduces the structure of the system, including image analysis and processing model and verification algorithm. The fourth chapter gives the experiment and analysis, and the fifth chapter is the conclusion and our future work.

2 Related work

Optical effect, traditional sensors and image sensors are commonly used in the design of existing early warning system for vehicle meeting in curved road.

2.1 Using optical convex mirror for early warning of vehicle meeting in curved road

At present, the main way of traditional early warning is to remind the driver of the road condition at the turning by using the close-range imaging function of convex mirror. The solution is easy to install and performs well under sufficient light. However, under weak ambient light intensity and severe weather conditions, the scheme cannot play its due role in early warning [Hu, Li and Liu (2013); Yan (2015)].

2.2 Using traditional sensors to carry out early warning of bend crossing

According to the shortcomings of optical early warning mode, many researchers adopt the combination of sensors and electronic display screen to carry out early warning of vehicle meeting in curved road. These conventional sensors include ring coil electromagnetic induction sensor, strain pressure sensor, geomagnetic vehicle sensor and so on. This method can play a very good warning role in rain and snow, haze, hail, thick fog weather, but this conventional sensor solution has some shortcomings [Wang, Gu and Yan (2018)]. That is to say, no matter whether they are ring coil electromagnetic induction sensor, strain pressure sensor or geomagnetic vehicle sensor, they are easy to cause damage to roadbed or road surface. At the same time, in the rugged mountain road,this kind of solution is difficult to construct and faces the risk of soil collapse. For example, Hu Yawei et al. put forward a ring coil vehicle detection system [Hu, Yan, Liu et al (2016)]. The system judges the vehicle information at the turning by the change of the vehicle electromagnetic field, but the road surface will be damaged in a certain range when the ring coil is installed. Subsequently, Ma Fanglan et al. put forward a vehicle detection system of geomagnetic sensor [Ma, Zhang, Xu et al. (2017)]. The system has high detection accuracy, but it still needs to damage road surface when it is installed.Recently, Yang Shuyuan et al. proposed an automatic vehicle monitoring system based on laser sensor [Yang, Wang and Xu (2018)]. The system avoids the operation of destroying the road like the above two systems and installs the detection device above the road. However, its construction and installation are difficult, and it is also vulnerable to bad weather.

2.3 Using image sensor to carry out early warning of curved traffic

Some researchers are aware of the shortcomings of conventional sensors, and propose a solution of vehicle detection using image sensors. In these studies, single-chip microcomputer and image sensors such as OV7670 are combined to detect vehicles [Huang,Yu, Ning et al. (2012); Tao and Wang (2014); Zeng, Zhang and Zhen (2014); Xie, Qin,Xiang et al. (2018)]. For example, Zhang Yinghui et al. used the improved frame difference method and background method to detect moving vehicles [Zhang and Liu (2017)]. The algorithm has strong robustness, but the system requires a high configuration of running environment, which is not suitable for field application. Then Gao Lei et al. proposed vehicle detection based on edge symmetry [Gao, Li, Zhu et al. (2008)]. This algorithm improves the accuracy of vehicle detection to a certain extent, and the cost of the platform is relatively low, but it still cannot run steadily in the wild environment.

At present, the existing solutions for vehicle meeting in curved roads have the disadvantages of difficult installation, poor display effect at night and bad weather, and insufficient strain. Therefore, this paper proposes a scheme combining real-time acquisition technology based on image sensor with vehicle Edge Detection and night frame difference algorithm for vehicle detection, so as to improve the system’s early warning ability at night. The system is placed on the side of the road whose equipment cost is low. At the same time, the installation method is convenient and the sensitivity coefficient is high. In addition, the system uses long life LED display screen as display carrier, and the power supply is two-way mode, namely solar cells and lithium batteries,so as to achieve stable operation in the wild environment.

3 Our scheme

3.1 Structure of early warning system for curved road

In this section, we first introduce the overall structure of the system, then introduce the image analysis and processing model of the system. Then, according to the image analysis and processing model, combined with vehicle Edge Detection and night frame difference algorithm, we analyze the image information acquired by the image sensor.The edge feature and symmetry feature of vehicle image make the algorithm achieve fast detection speed and strong adaptability to outdoor environment changes, so it is suitable for vehicle detection in mountain bend environment.

The structure of the new warning system for turning meeting is as follows:

As shown in Fig. 1, the system is divided into three parts: power supply, image acquisition and processing, and LED display. The power supply of the system is composed of solar charging board, rechargeable lithium battery and external standby charging terminal. The two-way power supply of the solar battery and lithium battery is processed by the voltage control circuit of the battery, so that the electric current is smoothly transmitted to STM32. Image acquisition and processing is composed of STM32 and OV7670. Firstly, OV7670 stores the captured image frame into STM32,carries on the image gray processing, and then complete the contrast analysis through the builtin algorithm on STM32. If the passing vehicle is identified after the algorithm analysis, the signal will be converted into text, and the LED display screen will be open by the warning lamp driving circuit. At the same time, the information will be sent to the vehicle software.

In the system, STM32 is a 32-bit microcontroller based on Cortex-O core of ARM. It is specially designed for low-power, low-cost and high-performance embedded applications.According to its different core architecture, STM32 can be divided into many different products, and its performance and power control are better [Li (2019); Zhou (2015)].

Figure 2: The OV7670 image sensor connecting MCU STM32

OV7670 is a 1/6 inch, 300,000 effective pixel image sensor made of complementary metal oxide semiconductor (CMOS). It is controlled by a two/three wire serial camera control bus and outputs 8-bit image data in parallel. The output of VGA image can reach up to 30 frames per second [Chen, Wang, Yang et al. (2017)].

Vehicle image identification for OV7670 is based on frame difference method. The process is described as follows:

In this formula, f1is the image frame at t1time, and f2is the image frame at t2time. By comparing the difference between the two image frames, the basic information of the vehicle is retained and the influence of environmental factors is weakened. At the same time, this method has less computational complexity and is beneficial to the realization of low-power platform.

Because the external outline of a vehicle is not easily affected by the outer environment,its feature quantity is large, the identification accuracy is not affected by the change of a single feature, and the external outline has the natural symmetry characteristic, so the system adopts the symmetry detection algorithm based on the external outlines of vehicles. At present, the symmetry detection algorithm usually determines the direction of the symmetry axis through some image features, and then determines the position of the symmetry axis by calculating the center of mass of the object, so as to achieve the purpose of determining the symmetry axis.

Figure 3: Simulation scenery

As shown in Fig. 3, our early warning system is designed to be located outside the road.The OV7670 image sensor is placed above the LED display screen to collect image data.The front and rear cameras of the camera module face respectively the lanes on both sides. They are installed under the power management module and the STM module of the single-chip microcomputer to detect the traffic status of the lanes on both sides.STM32 microcontroller is embedded in the shell of LED display screen, responsible for image data processing and control of each module. The solar charging board in the power management module is installed on the top of the camera module, which is convenient for receiving sunlight adequately. The other parts of the power management module and the MCU module are installed under the solar charging board. Through SCCB connection,the LED display device can be directly displayed, so as to warn drivers and pedestrians.

3.2 Image analysis and processing model and identification algorithm

In this system, image processing algorithm is the key. Firstly, the single frame image captured by OV7670 is processed by gray level. Then the processed single frame image is transmitted to STM32, which is processed by the optimized Sobel edge detection algorithm and frame difference method preset in STM32. By setting the system clock, the system uses the optimized Sobel edge detection algorithm to analyze and process the daytime image, and the optimized frame difference method to analyze and process the night image.

Figure 4: Image processing model

The optimized Sobel edge detection algorithm steps are as follows:

Firstly, the acquired image is preprocessed to get the corresponding image frame. Then,the Sobel operator (Fig. 5) is used to proceed mask calculation in the vertical and horizontal directions (Formula 1, Formula 2, Formula 3). At the same time, the edge gradient in the target image area is counted. Subsequently, the system determines the candidate area according to the characteristics of large fluctuation between the outline of vehicle area and road background, and the number of edge changes, and runs the preset openmv module on STM32. In vehicle detection, if the symmetrical axis direction of vehicle image has been determined as the vertical direction, it is necessary to further determine the position of symmetrical axis, measure the intensity of symmetry and analyze the symmetry of vertical gradient projection of vehicle image, so as to verify whether the candidate area is the real vehicle area.

Figure 5: Sobel operator

f(x,y)is the gray value of the image(a,b).

The essence of edge detection is to find the fast changing region of brightness in the image, that is, the region whose first derivative of brightness is larger in magnitude than the specified domain value. Because the vertical edge of the vehicle image has the strongest symmetry, in order to eliminate interference and reduce the computational complexity and time consuming, this algorithm only processes the vertical component of the edge.

The steps of the improved night frame difference algorithm are as follows: the optimized frame difference method is to subtract the previous frame with behind frame, and the moving object is retained in the difference-image. Because its principle is equivalent to the gradient calculation of time dimension, its difference-image has a certain degree of gradient effect.

Here,f1is the image frame at t1time, and f2is the image frame at t2time.

Firstly, the system temporarily stores the image which has been grayed, and then subtracts some slow-changing shadows and halos to eliminate the influence of shadows and halos. In order to reduce the computational burden of single-chip microcomputer, the algorithm only considers the area with light as the target area, considering the light characteristics of vehicles on the fly at night.

4 Experimental results and analysis

In our scheme, STM32F767 is used as the test platform, and OV7670 is used as the image sensor. We use 57.60 degrees horizontal view to photograph various traffic scenarios such as city highway, spiral mountain road and rural curved road for passing vehicle detection. At the same time, we consider the effect of image acquisition under different experimental scenarios, such as the comparison of sunny and rainy weather environment, and the comparison of daytime, evening and night illumination environment.

Figure 6: Comparison of background frame and foreground frame without passing vehicle

Figure 7: Gray difference between target area and background frame without passing vehicle

As can be seen from Fig. 6, since the position of the system device before and after installation does not change, the background frame after graying is almost the same as the foreground frame when no vehicle passes by. Then, the gray level of the image is analyzed for the scene without vehicle passing, and then the difference value is calculated. The result of Fig. 7 can be obtained. The horizontal and vertical coordinates of the image represent the gray level analysis value of the image. When the gray level difference of the horizontal coordinates is 250, the fluctuation is larger than before. Combining with the symmetry principle of the Sobel algorithm, it is determined that there is no vehicle passing.

Figure 8: Comparison of background and foreground frames when vehicles passing

Figure 9: Gray difference between target area and background frame when vehicles passing

From the comparison of Fig. 7 and Fig. 9, it can be seen that the gray value difference when vehicle passing is more obvious than the gray value difference without vehicle passing. In the experimental scenario, we fixed the position of the system, set two parallel target areas in the test platform, the target area will detect the details of the vehicle (the details of this paper are headlights), and run Sobel vehicle detection algorithm to process the captured image. From the processing results of Fig. 9, it can be seen that there are two peaks in the gray value difference of abscissa coordinates at 142±3 and 178±5. Compared with the single peak value when no vehicle passes by, the single peak value changes greatly, so it is determined that there is a vehicle passing. In addition, because the system has two parallel target areas, compared with the whole picture, the comparative computation is less, which can meet the real-time requirements of detecting passing vehicles.

Figure 10: Comparison of detection accuracy of three solutions within 48 hours

In order to verify the vehicle detection accuracy of the proposed scheme, we compared the traditional convex mirror, traditional sensors and our system in a same curved road within 48 hours. In the figure above, blue represents the accuracy of traditional convex mirror, orange represents the accuracy of traditional sensor, and green represents the accuracy of our solution.

As can be seen from Fig. 10, the identification effect of convex mirror scheme is close to 0 when the light is bad, and there will be the shortcoming of sunlight reflection when the light is too strong. Therefore, the identification effect of this scheme is very bad when the light is too strong or too weak. The traditional sensor scheme is better than convex mirror scheme, but slightly worse than our scheme. The reason lies in the error rate of traditional sensors. Traditional sensors cannot distinguish the number of vehicles passing through this section when the number of vehicles is large. Our scheme is to rely on image sensors to clearly distinguish the number of vehicles passing. Through experiments, we find that our scheme has a correct detection and alert rate of nearly 100% when the light intensity is high and a correct detection and reminder rate of 58.8% when the light intensity is weak. Tab. 1 shows the comparison results of three schemes

Table 1: Comparison of other solutions with image sensors

5 Conclusions

The system based on STM32 and OV7670 uses frame difference method and edge detection algorithm for image analysis. It can detect the passing vehicles in real time and warn drivers through LED display screen, so it greatly reduces the risk of turning meeting.Experiments show that the identification rate of our scheme is much higher than that of the other two traditional schemes in strong or weak light environment. In addition, the scheme has the characteristics of simple installation and strong real-time performance. However,the scheme in this paper is mainly aimed at the situation that a small number of vehicles are traveling at low speed on curved mountain roads, and there are still some limitations for the identification rate of continuous and large number of vehicles passing on urban roads.Future research will use deep learning, adaptive matching algorithm [Li, Qin, Xiang et al.(2018)] and border regression technology to reduce the impact of a large number of vehicles passing on the detection results, and effectively improve the detection accuracy.

Acknowledgments:This project is supported by the Cooperative Education Fund of China Ministry of Education (201702113002, 201801193119), Hunan Natural Science Foundation (2018JJ2138), Excellent Youth Project of Hunan Education Department(17B096), the H3C Fund of Hunan Internet of Things Federation (20180006) and Degree and Graduate Education Reform Project of Hunan Province (JG2018B096).