Study on video intelligent early warning and tracking system based on ARM①

Liu Yansong (刘岩松):Xu Yihong:Xing jianping

(*School of Mechanical and Electrical Engineering:Shandong Management University:Jinan 250357:P.R.China)(**School of Information Science and Engineering:Shandong University:Jinan 250014:P.R.China)



Study on video intelligent early warning and tracking system based on ARM①

Liu Yansong (刘岩松)*:Xu Yihong*:Xing jianping*To whom correspondence should be addressed.E-mail:sduxingjianping@163.comReceived on Sep.15，2015**

(*School of Mechanical and Electrical Engineering:Shandong Management University:Jinan 250357:P.R.China)(**School of Information Science and Engineering:Shandong University:Jinan 250014:P.R.China)

The protrusion of the planning of numerical intelligent early-warning and tracking system in this study which can ease triggerman’s work strength:lay the next generation intelligence supervision system foundation and expand effectively the video resources use etc.In the numerical intelligent early-warning matrix sub-system:the authors have designed a kind dual-core system which includes both ARM and DSP:and designed detailedly traffic dynamics affairs early-warning arithmetic which bases on that system.And then:this system will carry quickly on fixing the right position of license plate:correcting the inclination degree of license plate:and thinning it to get the number of this license and severity grade.Secondly:in the rotated dome camera sub-system:the authors have designed three-dimensional trajectory mathematical model which makes use of a fuzzy PID controller to achieve the high-speed track.At last:Simulation shows that the proposed control method has high profile tracking precision:accuracy and robustness of the disturbance.

embedded dual-core system:intelligent early warning:accident recognition:tracking and monitoring:simulation

0 Introduction

In recent years:with the enhancement of economic power in China:urban area is extending:law-and-order situation is complicated:public safety issues are constantly highlighted:urban crime is evident:and the means is updated and upgraded continuously:which has an urgent demand on accelerating development of digital video intelligent early warning and tracking surveillance system dominated by proactive prevention[1,2].In September 2009:Ministry of Public Security in China officially launched urban networking alarm and surveillance system construction (Project 3111) which developed alarm and surveillance system construction pilot projects all over the country to promote the construction of safe city.However:from the present urban networking alarm and surveillance system construction:the video surveillance system is constructed by simulating monitoring system or adding video capture card to PC:which is difficult to satisfy the requirements of real-time process:stability and reliability:accident recognition and automatic tracing.And it is confined by post-mordem forensics.Therefore:it is urgent to develop a new video surveillance system.Rapid development of large-scale integrated circuit and embedded software and hardware technology provides new thought of solving the problem:which means to develop digital video intelligent early warning and tracking surveillance system based on embedded structure.

The innovation of the study includes:Firstly:analysis of video image is made in video matrix (the chip uses AT91RM920T of ATMEL Corporation):that is:DSP chip is used to make high-seed analysis on large number of data in video pictures and automatically extract key information of video source:which can detect the problems and make timely early warning.After early warning:the license plate is automatically recognized and GPRS is used to send the recognized license plate and accident situation to the person on duty:and then tracing speed dome is used to track abnormal situations until early warning is removed.Secondly:a response algorithm model of early warning condition is designed.The model can make high-speed analysis on large number of compressed video data.The extracted key information is used to judge if it has need to make early warning and implement corresponding operations:for example:making early warning picture display on monitor and opening DVR for recording picture.A hardware structure is designed and fuzzy PID controller is used to control speed dome to track suspicious targets and make real-time surveillance until the early warning is removed.

1 Image early warning algorithm of dynamics visual event recognition based on dynamics

1.1 Introduction of visual event recognition based on dynamics

For dynamic visual method:all events to be recognized should be defined as verb firstly.It is known from the practice that only three verbs can describe all events.The verbs are:1) Support:an objectYfalls when it is not supported by objectX:which means that objectXsupports object Y.2) Connection:when two objects contact:it is called connection.3) Attachment:when two objects move as a community:it is called that an object attaches to the other object.

The events which are described by the above verbs and the combination with logic relationship are used to express a complicated event.The event of object X catching object Y and throwing it is taken as an example.

Defining throw(X:Y)

There are two sub-events (I:J)……where I means X catches object Y:and J expresses X throws object Y

Logics and (when sub-event I:connecting(X:Y)

when sub-event is I:attaching to(X:Y)

when sub-event is I:supporting (X:Y)

when sub-event is J:attaching to(X:Y)

when sub-event is J:connecting (X:Y)

Logic=(I ends)(Jbegins))

It can be seen that the contour features including place:direction:the shape and size of the outline of moving targets can be gained through outline process after achieving traffic information images.The contour features are assigned to joint model and hierarchical model:which can judge if the outline meets the features of traffic accidents and determines if it is necessary to make early warning.

1.2 Dynamic description of traffic incidents

After defining the principle of dynamic event recognition:there are how many traffic early warning incidents is the next problem to be solved.It is known from practical experience that common traffic accidents include rear-end collision:vehicle collision and vehicle breakdown.A brief description of these incidents is the following.

(1) Rear-end collision and vehicle collision

Dynamic description of rear-end collision and vehicle collision is consistent.They can be described by using the following verbs and logic (supposing the passive car isX:and the active car isY).

Defining rear-end collision(X:Y)

There are two sub-events(I:J)……where I represents thatYcollides withX:and J meansXandYmove together or reverse movement or other physical condition.

Logic and (when sub-event is I:attaching to(X,Y))……Rear-end collision happens(or collision)

when sub-event isI:attaching to(X,Y)……Taking two cars moving together as an example

when sub-event isJ:connecting(X,Y)……Two vehicles separate.

when sub-event is J:still (X)and still(Y)……Two vehicles stop

Logic=(I ends)(J begins))

(2)Vehicle breakdown

Defining vehicle breakdown(X)

There are two sub-events(I:J) ……I means vehicleXbreaks down:andJmeans to re-determinate if vehicleXbreaks down again 60 seconds later.(Logic and(when sub-event isI:still(X)……Possible breakdown incidents happen

(when sub-event is J:still (X))……Vehicle breakdown

Logic=(Iends)(Jbegins))

1.3 Image early warning algorithm

Based on the above description of traffic vehicles:an algorithm is designed.When DSP polls all images of input audio and video matrix:the vehicles and pedestrian of images are for outline operation and dynamic comparison of traffic incidents.If it conforms to the above instruction:it means traffic incidents happen.

2 Event notification content——license plate number location method

2.1 Binarization of license plate image

It is not difficult to discover grey-level features of license plate area.The characters and background pixels of license plate have even gray values which have no great difference[3,4].But there is great difference for average gray of two pixels.In addition:there are rich edges in rectangular area that license plate belongs to.There are regular intervals between strokes of each character:which is not common in rectangular area of license plate.In order to make full use of the features:the study firstly makes level difference accumulated operation on the original gray-level image to highlight the upright edge of image.And the selected method is

(1)

where f(x:y) is gray value of image:HD(x:y) is difference accumulated result:x and y express the value of horizontal and vertical coordinate:and d means offset.The achieved difference image can highlight edge features of images and expand the number of pixels:which provides basis for the extraction of license plate (see Fig.1).

Fig.1 Gray level of license plate

Adaptive threshold method is used to make binarization processing on the difference image achieved by accumulation:which can get edge image including region area of license plate.The method is where T is Bayes which means to use minimum error criterion as convergence condition to get the threshold of segmenting images.Lastly:the image which is processed by the proposed algorithm is shown in Fig.2.

2.1.1 量表的内部一致性信度 对量表中睡眠/精力维度、困扰/关注维度及总量表的内部一致性进行检验，结果见表1。量表各维度得分与总分做Spearman相关分析，睡眠/精力、困扰/关注维度与总量表的相关系数分别为0.843与0.715，均具有统计学意义(P<0.05)。

(2)

Fig.2 License plate binarization

2.2 Acquirement of license plate candidate area

Although license plate area has rich and concentrated edges:the edges may not be interconnected to form connected continuum:which is not conductive to vertical location of license plate.So the achieved binarization edge image needs to make horizontal mathematical morphological dilation:which makes a license plate area become a connected area and can use the license plate as a whole and non-division region for analysis.While making dilation operation:the interference of noise on edge image causes many pseudo lines.In order to effectively filter license plate area:these lines need to be filtered.Under the premise of not breaking the existing connected region:the work scans from vertical and horizontal direction:and filters the influence of the lines by restricting the length of all white lines.And the algorithm is:

Step 1 Scanning one column (line):and counting the number of continuous nonzero pixels:NUM。

Step 2 If NUMTcol1(Trow1):the pixel in the part is set to be zero or keeps invariant.

Step 3 NUM is cleared.The column (line) continues to be scanned and counted:and turning to Step 1 until the scanning on the column (line) is over:and turning to Step 4.

Step 4 Entering the next column (line) and turning to Step 1 until the scanning on image is over.

Tcol1(Trow1) is the length threshold which is preset according to priori knowledge[2-4].

After processing:several connected regions are achieved as the candidate license plate area.In order to make further process:the connected regions of the image need to be marked:and the leftmost:rightmost:uppermost and nethermost coordinates of each connected region is respectively marked as LeftiRightiTopiand Bottomi:and is defined as a candidate rectangular region.And the length and width of each rectangular region can be calculated as

Wi=Righti-Lefti:Hi=Topi-Bottomi

(3)

The image after processing is shown in Fig.3.

Fig.3 License candidate region

2.3 Vertical and horizontal location of license plate

Each candidate rectangular region image needs to be made vertical-direction difference operation.The vertical-direction projection VPi(x) of vertical difference image needs to be solved:

-f(x+Lefti:y+Bottomi)|

(4)

If Maxiis the maximum of element of VPi(x):as the width of license plate characters of images is variant in certain interval:and license plate characters have no evident regularity because of noise interference.Therefore:when horizontal scanning liney=Scan to scan VPi(x) from 0 to Maxiis used in the interval [0:Maxi] according to certain Intervali:the number of continuous elements in VPi(x) which are greater than the scanning value of the scanning line is recorded and characterized as the length of multiple scanning lines.Then the length of all lines in [0:Maxi] and the frequency is accumulated.Only the area in which the frequency of the lines with specific length is greater than a specific number is real license plate area.Through the processing:each non-license plate area can be eliminated:which completes screening and location of license plate area in vertical direction and can gain mean width of license characters of images.

Horizontal location process of license plate is the process of determining the left and right boundaries of license plate.In order to determine the left and right boundaries of license plate:it can use the proportional relationship between the length and width of license plate and concrete width of license plate of the achieved figure to speculate the concrete length of license plate:which uses the interval of the length of license plate as sliding window:and makes accumulated statistics on the projection in vertical direction of the license plate.When the accumulated value is the maximum:the left and right boundaries of the window can be as the left and right boundaries of the license plate in the image[5-7].The processing image is shown in Fig.4.

Fig.4 License plate location

3 Event notification content——license plate number recognition algorithm

After the license plate is located:the process of license plate recognition is started.The recognition principle of number:character and English letters of license plate is the same to that of printed characters.

3.1 Inclination correction of license plate images

As shooting angle is different:license plate inclination appears in license plate images:which makes it difficult to refine and recognise the license plate characters in the subsequent step.(Note:as tracking speed dome is used:it is less likely for the paper to obtain inclination of license plate.But the step is still operated in order to make correct recognition).From analysis on features of license plate images:it can be seen that license plate may be inclined:but the upper:down:left and right sides of license plate frame:and vertical edge of some specific characters still keep straight line and have great length value.So Hough conversion can be used to detect the inclined or non-inclined frames which are greater than specific length and similar edges to complete inclination correction of license plates.In view of timeliness requirement of the whole license plate recognition system and horizontal stripes interference of vehicles:Hough conversion which is used in the paper only can detect the vertical edge in the interval of specific length and the long edge of license plate characters strokes.As the achieved edge angle can determine the inclination angle of license plate:the inclination angle value is used to translate the pixels in each line of binary images:which makes vertical edge correct[8].

3.2 Refinement of license plate images

After obtaining binary images after inclination correction:the features of numbers:characters and English letters are extracted.In order to make feature extraction easy:the images should be refined.

Refinement is that binary images with width area are for operations and becomes a framework with only one pixel width.It is an important operation for image analysis and mode recognition.The framework image which receives refined process provides a simple expression for image processing and analysis of the subsequent image:which is conductive to making detailed analysis on images[9-11].Refinement algorithm has the following requirements.(1) Image skeleton must keep connectivity and topological structure of the original images:(2) Image skeleton should be the central line of the original image.(3) The result of refinement should be a line image with a pixel width.(4) Processing speed should be rapid.(5) Anti-noise performance is good.

The design uses the improved and refined algorithm proposed by Zhang and Wang in Ref.[12]:as follows.

The algorithm uses 4×4 formwork：

P10P11P12P13P9P2P3P14P8P1P4P15P7P6P5P16

Pixels of binary images are expressed by 0 and 1.1 represents prospect:and A (P1) means the number of 01 in P2:P3:P4:……P9 and P2 around point P1.Function BCP1) is used to mean the number of nonzero in the connected area from P1 to 8.When P1 satisfies the following requirements:it will be deleted from the original image:(1)P1=1；(2)2<=B(Pl)<=6；(3)A(Pl)=1；(4)P2×P4×P8=0 or P11=1；(5)P2×P4×P6=0 or P15=1.

3.3 Character recognition

The features of the number and letters of the refined license plate should be analyzed before recognition:which makes it differentiate from various numbers and letters.The principles of feature extraction are as follows.(1) Distinguishable.The features of different categories of objects have evident difference.(2) Reliability.The characteristic values of different objects are similar.(3) Independence.There is no relationship between the features which are used.(4) Small number.Complication of pattern recognition system increases with the dimension of the system.

Figure is taken as an example for analysis.According to the configuration of figure:four features are selected.(1) Horizontal line:XR=nR/widthR.nR is the number of continuous pixel points in horizontal direction:and widthR is the horizontal pixel of single letter image.When XR>0.5:there is a horizontal line.(2) Vertical line:YC=nC/widthC.nC is the number of continuous pixel points in vertical direction:and widthC is the vertical pixel of single letter image.When YC>0.5:there is a vertical line.(3) Number of horizontal lines.A line is drawn in some horizontal position of image:and it is the number of the image crossing with the horizontal line.(4) Number of vertical lines.A line is drawn in some vertical position of image:and it is the number of the image crossing with the vertical line.The following is the judgment process.Firstly:it can be divided into two categories according to the number of horizontal lines:A here is the number of horizontal lines (2、4、5、7)and B there is not the number of vertical lines (0、1、3、6、8、9).A can be differentiated by horizontal position and the standard if there is vertical line.B is differentiated by the standard if there is vertical line:the number of horizontal lines and vertical lines.

4 Image tracing model and simulation

4.1 Design and mathematical model of single-dimensional fuzzy PID controller

From the working principle of fuzzy PID controlling speed dome in Section 3:it can be seen that fuzzy parameter adaptive PID control system is the result of fuzzy rules adjusting PID parameters timely[13,14].Therefore:as long as fuzzy rule is designed:reliable speed dome can be designed.In view of the simplification and rapidity of fuzzy parameter adaptive PID control system:the structural form of fuzzy controller is used:which means that system error e and error change Δe are used as input statement variables.

For fuzzy control rules:if the linguistic variable of error e is E:the value of corresponding fuzzy subset A is PB:PM:PS:ZO:NS:NM and NB.If the linguistic variable of error change Δe is ΔE:the linguistic value of fuzzy subsetBis the same to that of fuzzy subset A.Their membership functions are triangular:and the width of each value is equal.If T-S(Takagi and Sugeno) fuzzy model is used:the following implication needs to be used:if e=A and Δe=B then Uf=g(e):in which A and B are fuzzy subsets:and g(e) is continuous function of e.According to the above model:fuzzy control table can be achieved:as shown in Table 1.

In Table 1:g1(e)=Kue:g2(e)=0.7Kue:g3(e)=0.35.According to fuzzy reasoning algorithm:a fuzzy query table can be get.

Because the output of fuzzy adaptive mechanism is Uf:and Uf=f(e,Δe)Kue……(5.1):in which f means nonlinear relationship:its value is determined by the fuzzy query table.

The output Ufof fuzzy adaptive controller means the input of fuzzy PID controller.Because the discrete expression of conventional PID control is u(k)=Kpe(k)+KiT∑e(j)+KdΔe(k)/T in which T is sampling period:e(k) and Δe(k) are input variables:the

Table 1 Fuzzy control table

unknown variables are parameters of controller:Kp:Ki:Kdand Ki=Kp/Ti:Kd=KpTdin which Tiis integral time constant and Tdis derivative time constant:the output of fuzzy adaptive PID is

(5)

In order to improve the performance of PID controller:besides adjusting the design rules of fuzzy controller:scaling factors can be set to be adaptive parameter correction according to error e and error change Δe of the system.The correction rules are:When error e and error change Δe are great:less Keand KΔeshould be taken and greater Kpand Kivalues should be taken:which not only reduces resolution ratio of error e and error change Δe:but also can accelerate the response speed.When the system is close to be static and error e and error change Δe are less:greater Keand KΔeshould be taken and less Kpand Kivalues should be taken:which not only improves resolution ratio of error e and error change Δe:but also can reduce the overshoot of system.According to the above principles:parameter correction module can easily reason out the multiplication n of parameter correction from the value of error e and error change Δe:and Ke=Ke·n:KΔe=KΔe·n:Kp=Kp/n and Ki=Ki/n is calculated as new parameters of fuzzy controller for controlling.And from the above condition:it can be realised that integral time constant is determined by derivative time constant

Ti=aTd

(6)

Eq.(5) is substituted into Eq.(6):the Eq.(7) can be got:

(7)

Form Eq.(7):it can be known that the gain of fuzzy adaptive PID controller is nonlinear function:and fuzzy adaptive PID controller has the function of dynamically adjusting proportional coefficient of PID controller.

4.2 Mathematical model of three-dimensional tracking path

Cameras based on speed dome can be applied to all-round surveillance and demands to make coordination control on multi-axis stepper motor to drive cameras:which makes it track motion objects.

Therefore:at least 3 stepper motors are needed to realize the traction ofxaxis:yaxis andzaxis:which realizes spatial location.In order to deduce the mathematical model of tracking path:it is supposed that the basal coordinate of any locus function f(x:y:z) is (x1:y1:z1):each coordinate changes with time parameters:and suppose T is geometric mapping operator:f(t1)=T{x1(t1):y1(t1):z1(t1)}.

The motion equation only includes registration parameter and is the same to common geometric function:which has corresponding geometric features such as the slope of a tangent:curvature and torsion.And it indicates important motion information which means dynamic relationship of motion displacement.The differential equation can express the relationship between motion parameters.Deriving the time t1of Eq.(8) can be obtained:

(8)

It can be seen from the formula that each item only relates to geometric mapping operator T:which means that motion speed of f(x:y:z) is equal to the multiplication of geometric parameter and time rate of change.Therefore:if locus function is known:motion parameter of any point can be solved.

4.3 Model simulation

MATLAB language is used to make simulation comparison test on control system which uses and doesn’t use fuzzy adaptive controller.In the experiment:the transfer function of controlled object (stepper motor) is

The unit step response results of the system are shown in Fig.4 ( the curve ① is the system output of fuzzy adaptive PID control:and curve ② is the system output of the conventional PID control).

Fig.4 Simulation of fuzzy PID and conventional PID control system

Simulation curves show that when fuzzy adaptive PID control is used:the dynamic response of the system is rapid without overshoot:and there is no state error:so it has good controlling effect.But when conventional PID control is used:the response speed is low:and there is overshoot and vibration which is between 0.5s and 0.75s.

5 Conclusions

Based on studying digital video intelligent early warning and tracking surveillance system at home and abroad:the paper studies the existing problems of digital video intelligent early warning and tracking surveillance system in our country:and proposes a digital video matrix structure based on ARM and DSP dual-core structure and speed dome based on fuzzy PID automatic tracing.

The innovation of the work includes:dual-core embedded structure is used to make digital video intelligent early warning and tracking surveillance system:which not only realizes the analysis and comparison of various images:but also improves the overall performance of the system.Using dynamic incidents fuzzy recognition and GPRS incidents not only greatly reduces false alarm rate of the system:but also makes the work of monitoring personnel reduce.Fuzzy PID automatic tracing is used in combination with simple fuzzy control and conventional PID control.According to the size of error e and error change Δe:fuzzy rules can be modulated automatically:and proportional factors Ke:KΔeand Kp:Kican be corrected adaptively:which makes natural frequency and damping ratio of the system transferring functions regulate automatically:which makes the system realize reducing positive fixed torque and increasing reverse braking torque:which can overcome nonlinearity of speed dome:instability of parameters and influence of various interferences on the system.For example:conventional PID control and simple fuzzy control have higher outline tracing accuracy:positioning accuracy and robustness on disturbance.

[1] Sang N.Principles and Application Development Techniques of Embedded System.Beijing:Beijing University of Aeronautics and Astronautics Press:2002.56-83

[2] Du C L.Architecture and Programming of ARM.Beijing:Tsinghua University Press:2002.62-90

[3] Zhang W X:Chen L:Xu G H.Principle and Development Application of DSP Chip.Beijing:Publishing House of Electronics Industry:2004.199-226

[4] Sun J G:Hu S M.Basic Course of Computer Graphics.Beijing:Tsinghua University Press:2005.189-201

[5] Chen C.Computer Image Processing Technique and Algorithm.Beijing:Tsinghua University Press:2003.26-53

[6] Rafael C G:Richard E W.Digital Image Processing.Beijing:Electronic Industry Press:2005.115-210

[7] David F R:Basic Algorithms of Computer Graphics.Beijing:China Machine Press:2002.16-22

[8] Bian Z Q:Zhang X G.Pattern Recognition.Beijing:Tsinghua University Press:2000.72-78

[9] Xie J Y:Jia Q.Microcomputer Controlling Technique.Beijing:National Defense Industry Press:2005.101-109

[10] Yu H S.Microcomputer Controlling Technique.Beijing:Tsinghua University Press:2002.55-60

[11] Zhu J.Principle and Application of Fuzzy Control.Beijing:China Machine Press:2005.67-72

[12] Zhang Y Y:Wand P S.New parallel thinning metho-dology.InternationalJournalofPatternRecognition&ArtificialIntelligence:2011:08(5):999-1011

[13] Liu M.Design of Embedded System Interface and Development of Linux Driver.Beijing:Beijing University of Aeronautics and Astronautics Press:2009.20-26

[14] Duan Y H:Yang S.Fuzzy PID control of stepping motor.ComputerSimulation:2006:23(2):290-293

Liu Yansong:born in 1982.He received the M.E.degree in Shandong University of Science and Technology in 2009.He also received the B.S.degree in College of Information Engineering from Jinan University in 2005.His research interests include wireless sensor networks and embedded system.

10.3772/j.issn.1006-6748.2016.03.005

①Supported by the National Basic Research Program of China (No.2011CB707000):Science and Technology Development Program of Shandong Province(No.J13LC51，2011XH17006):Independent Innovation Program of Ji’nan Colleges and Universities(No.201401213).