Smart video analytics on cameras already installed: how to tell a real event from a shadow
We built a video analytics system that works with the cameras and video recorders a company already has installed. Zones are marked out in the frame, and if something appears in a zone or disappears from it, an alert arrives with a short video clip. Here is where the main difficulty lies in a task like this, and why we chose these particular solutions.
Almost every company has video surveillance, but it is usually less useful than it could be. The cameras record round the clock, the archive grows, and it is only watched after an incident, when something has already happened. There is nobody to watch the screens in real time.
The task was formulated simply: use the cameras that are already in place, and get an alert only when something has genuinely changed in an important spot. There was no need to buy new equipment or install a server for it.
The task: zones, not “motion in the frame”
The system connects to IP cameras and video recorders over the standard RTSP protocol. Most modern devices support it, including the Dahua recorders on which the system was tested. In each camera's frame, the zones to be watched are marked out: the reception desk, a display window, a shelf in the warehouse. For each zone the system remembers how it should look, and reports if a new object has appeared there or one that was there has disappeared.
In words it sounds simple. In practice, an ordinary motion sensor is almost useless in such places: there are people at reception all the time, shadows move across a display window during the day, and the lights come on in the evening. If you react to every change in the picture, there will be so many alerts that people will stop reading them by the second day.
How the system works
The system runs as a Windows application on an ordinary office computer. There is a panel for viewing the cameras and a screen for setting up zones. For each camera the system:
- continuously reads the video stream and keeps the last few seconds of footage in memory;
- compares the marked zones with a reference and judges whether anything in them has changed in substance;
- recognises people in the frame, to know whether someone is nearby;
- when an event occurs, saves a short clip: a few seconds before the event and the event itself;
- sends an alert with this clip.
All background processes are supervised. If recording on any camera stops, the system notices and restarts it.
Why it is built this way
A shadow changes brightness, but not the pattern
The main problem is false alarms from shadows, glare and changes in lighting. If you simply compare how much the brightness in a zone has changed, the shadow of a passing person looks exactly the same as a box put down on a desk.
We relied on a simple observation: a shadow makes a surface darker but keeps its pattern, that is, the texture of the desk, the edges of objects, the design. An object that has appeared in the zone covers this pattern. So the system assesses not brightness but how well the structure of the image has been preserved. It calculates what share of the zone's area has lost its previous pattern.
The first version averaged this score over the whole zone, and that did not work. The average value for shadows on real frames ranged from 0.77 to 0.87, and for a real event it was 0.806, that is, within the same range. An object is usually small relative to the zone and gets lost in the average. We also tried compensating for the overall change in brightness, but that did not help separate out shadows either.
A different distinction worked. An object destroys the structure in a solid patch, while the edge of a shadow or a strip of light produces a thin line. The system removes thin lines with a special filter, keeps the patches and then calculates their share of the zone's area. The thresholds were selected using real frames from the sites:
- daytime shadows — from 0 to 6.5% of the zone's area;
- lights switched on at night — from 6 to 7.8%;
- a real event — from 12.2 to 13.4%.
The boundary is set at 8%. If the change is smaller, the system marks the zone as “possibly a shadow” and does not raise an alarm.
This check is not the only safeguard. If something has changed at night or in an empty room and there are no people in the frame, it is almost certainly the lighting. Lights being switched off en masse is recognised separately, by the change across the whole scene. The structure check covers the hardest case: shadows during the day, when there are always people in the frame and the rule “no person, no event” does not help.
A buffer in memory instead of continuous recording
The first version recorded video continuously, and in an empty warehouse this quickly filled the disk with hours of footage in which nothing happens. Now the last 60 seconds of each stream are kept in RAM, and only an episode with activity, together with a few seconds before it, goes to disk. In operation on four cameras this took about 53 MB of memory in total, roughly 16 MB per camera at 1280×720 resolution.
Recording starts when the system sees a person or noticeable movement. To keep the clip from breaking up, the threshold for continuing recording is lower than the threshold for starting it: a person sitting at a desk barely moves, and the recognition confidence for them fluctuates. Without this, recording cut off even though the person was still in the frame. Adjacent episodes are joined without repeats. Previously they overlapped by 10–19 seconds, and the same video was saved twice.
Clips are saved in a format that can be read even if the program crashes in the middle of recording. We checked this by forcibly stopping the process at the 96th second of recording: the file opened in full.
Keyframes instead of full decoding
Video from a camera is compressed. Full frames, or keyframes, arrive roughly once every two seconds, and in between only the changes are transmitted. To get every frame, the whole stream has to be decoded continuously, and that is the main load on the processor.
We measured the consumption on a Full HD camera with the H.265 codec. Simply reading the stream costs 0.01 of a processor core, reading the stream and decoding only the keyframes 0.06 of a core, and decoding everything 0.43 of a core. To watch objects in zones, one frame every two seconds is enough, so an economy mode is switched on for such cameras. It is not suitable for counting people crossing a line: a person passes faster than the next keyframe arrives.
The stream is still read in full, though. If it is read more slowly than the camera delivers it, the picture starts to lag. In one case the lag reached eight minutes, even though by every internal indicator the camera appeared to be working. Since then, how current a frame is has been checked against the clock that the camera itself stamps onto the image.
Measure first, then optimise
Preliminary estimates promised from 1.7 to 2.5 processor cores per camera. A measurement on the running system showed something else: about 55% of the time goes on comparing zones and only about 3% on reading video streams. So we optimised what actually loads the system and did not reduce the resolution. The structure check works with fine details of the image, and with a reduced picture it would be worse at telling a shadow from an object.
What was built
- A Windows application: a camera panel, zone setup, a separate screen for people in the frame, launch from the system tray and an installer.
- Connection to IP cameras and recorders over RTSP, including Dahua equipment.
- Zone monitoring with a structure check that filters out shadows, glare and changes in lighting.
- Recognition of people in the frame as an additional filter.
- Event-based recording with an in-memory buffer, and clips that withstand crashes.
- An economy stream-reading mode for cameras that watch objects.
- Continuous checking of background processes.
Practical value
The company gets value from cameras it has already bought and installed. Instead of an archive of many hours that nobody watches, it receives alerts about specific events: an item was left at or taken from reception, the contents of a shelf in a display window or warehouse have changed. Each alert comes with a short clip, so the situation can be understood in a minute.
The alerts are to the point, so people do not start ignoring them: the system is useful only as long as its messages are trusted.
No separate server is needed. The in-memory buffers for four cameras take up tens of megabytes, and the economy stream-reading mode uses a small fraction of a processor core per camera.
Outlook
The system's thresholds were selected using real frames and can be adjusted for a specific site. If very small or thin objects need to be noticed in a zone, the sensitivity of the check is increased. For places where every passage matters rather than the state of a shelf, the full video analysis mode remains available.