Multi Processing OpenCV Motion Detection Security Camera

Having an old webcam hanging about and a Pi Zero (in a customized matchbox case) available (a MagPi freebie) I decided to try my hand at Motion Detection. Looking at some of the stuff around, I was introduced to Pyimagesearch search a site which is excellent for anything to do with OpenCV. A lot of the ideas for my system came from here. Also very worthy of praise is Derek Simkowiak work on tracking gerbils.

But to go back a bit in time, I came from a slightly different angle than the above. My idea was based on how I thought a robotic vision should work. Rather than processing every frame for faces in order to do facial recognition or object detection I thought the following sequence of events should take place. First the robot should be triggered by movement. Then within the movement area which is a lot smaller than the whole frame you should be able to identify a figure. Then from that you could identify a face in the top part of the figure. From the recognition of a face you should be able to identify a particular person. In my mind there was also a lot of inefficiency – why would you after identifying a person keep repeating the same procedure frame after frame. Once something or someone was recognized you should be able to follow it.

With that in mind I started off by looking at the Lightweight Motion Detector which seemed a very simple cut to the bone motion detection that I could be the first triggering. But I soon realized that this just wasn’t sophisticated enough for a security camera looking out through the window. Mainly because you have natural movement of pixels due to natural atmospheric conditions (or fairies as my daughter liked to think). That is where I came across Adrian at Pyimage’s work using accumulated weight and then subtracting the current frame from the average. Then contours are calculated which basically makes a box around any movement and the size of this is calculated in order to ascertain whether it is a genuine movement.

Where I differed from Adrian was that I didn’t want to check every frame and I wanted to save complete videos of the motion. But straight away the issue is if you only detect motion after say four frames and you start filming at that point you then have lost the first part of the motion.

To get past this I came up with the idea to set up a queue of images so that a couple of seconds of film is held at all times. If there isn’t any motion the frame at the beginning of the queue is removed and forgotten. If there is motion the film can start to be saved from the queue. This gives a couple of seconds of film before the event which is a bit like going back in time. I also allowed a few seconds after motion has stopped before the filming finishes. The problem here was if the postman went past the window to post letters through the door the film would stop. But then he would go back past the window and the queue hadn’t had enough time to fill up again. So a lot of fiddling was required – first to allow motion to be detected more rapidly if motion had just stopped so filming didn’t have to stop and start in situations like that.

Another issue I had was too many things triggering the detection. Since the window looks out on to a road every car that passed I got tired of looking at the videos. For that reason I created masked areas. But simple masking an area was not really good enough because if someone walked off the road and down the path if the road didn’t count as a motion detect area then I would lose the first part of the movement. To get round that I created the concept of motion in a blue area, i.e. not that important, and green area, important, and several blue area plus a single green area was enough to count as true movement.

Though I didn’t do the motion calculations every frame it seemed to be that there still was a lot of inefficiency as why would you wait while you calculated motion before grabbing the next frame from the camera. In my mind there were three distinct processes involved here. The first being the camera grabbing process which should be consistently taking frames and pushing them on to the queue. This is of the utmost priority as you don’t want to lose frames per second as this would ruin the film. The second process is the motion detection. This has medium priority as it doesn’t need to process every frame in order to detect motion (that is unless you want to prove the existence of superman). And thirdly the process that takes frames off the queue and writes them to a file if required.

I saw an article on the Pyimagesearch about reducing latency using threads. However, I was a bit disappointed in that this didn’t work on the Pi Zero. To be fair Adrian pointed out that due to the number of cores the benefits of threading on the Pi Zero were minimal. My experience was worse – even without and processing on the image the fact of even having threads slowed the performance down.

But after a bit of research into multiprocessing I was able to rewrite the system with three processes running at the same time. Though this is a bit more tricky to program with using events to communicate between the processes. One of the things you can do is use the nice() facility which will set the priority of the process. Maybe this should be called polite() instead because the less the figure is the more demanding on the system it is. But works really quite well. In good light the film works at 27 to 30 frames per second. And mostly it is flicker free.

What about in bad light? – what about in the night? Well I have a motion detection floodlight outside the house anyway so if there is anything moving outside the light will come on. This means that there is just about enough light to film any movements outside. Though this is a bit splodgy and films around 4fps.

Though it really works well, the system does suffer from weather conditions. On a semi cloudy day with wind the light can vary and create movement of shadows. When the sun is low it reflects in bus windows or side of vans which also creates false positives. Heavy wind can make the leaves of a bush move intensely. Unfortunately I can’t mask this out as I need to capture movement in front of it. To counter this I first need to ensure that the bush is well trimmed and change the minimum size of the contours in strong winds some times of the year.

Sometimes I get what looks like dementors flying past, which I assume to be insects. And strangely, for two weeks an insect of some kind kept walking on the window in front of the camera. Not just for a short while – it would do it once or twice every hour!

Uploading the Film

Of course a security camera is not much use if you can’t check up at what’s happening at home when you are away. I got this idea again from Pyimage Search, where Adrian sends a still photo to his dropbox. Unfortunately the formats that OpenCV supports is very limited and in particular it can’t write mp4s. I tried various ways of converting from avi, but there didn’t seem to be any way to do it on the fly, though you could once the file had been saved. But the conversion uses a lot of resources and is time consuming. But luckily there was little reason to fret, dropbox kindly converts avi files on the fly when you click on them and streams the video back to yourself.

Having got this working I found I found there was a problem. I would do a small change to some setting or another – leave for work and find that if it wasn’t quite correct I would then have to wait until the evening before I could make any change. Same for the masked areas. So I went a bit further and used my dropbox folder to sync the configuration files so that they could be updated from a far. Also for the same purpose of debugging I uploaded the log files. If the config file changes then the motion detector program gets restarted to take on the new values. I also wrote a program that checks that the motion detect is working correctly and tries to restart the system if it doesn’t look healthy. As I wanted to check the logs from the whole system I also created a socket logger which also logs errors from the standard error. For example if OpenCV throws a wobbly. Finally there is a housekeeping prog which removes old film files stored locally. It doesn’t remove them from dropbox – that I still do manually.

Code is uploaded here at bitbucket.

Future Developments

There are many possible ways to take this project further. I am thinking possible of separating it into three separate services which communicate with each other. I could add another process which fires to detect a figure within the movement ROI (region of interest). Once movement is detected maybe object tracking would be preferable to continued motion detection - see camshift. I have also plans to get the system to only turn on at certain times and when everyone has left the house. This could be done by Bluetooth vicinity. More details of this and other projects can be found at http://danicymru.wordpress.com.