Overview

As a response to the coronavirus (COVID-19) pandemic, Neuralet has released an open-source application that helps people practice physical distancing rules in different places such as retail spaces, construction sites, factories, healthcare facilities, etc. This guide provides a technical overview of the approach to smart social distancing. Familiarity with deep learning computer vision methods and Jetson Nano is recommended to follow this guide.

Problem description

Social distancing (also physical distancing) is one approach to control and prevent the rate of infection of different contagious diseases e.g. coronavirus and COVID-19, which is currently causing one of the largest pandemics in human history. Current solutions to practice social distancing are either not practical or not viable in the long-term. Please visit this blog post to read more about why current approaches fail, and why we need a more robust approach.

Our solution

Our approach uses artificial intelligence and edge AI devices such as Jetson Nano to track people in different environments and measure adherence to social distancing guidelines, and can give notifications each time social distancing rules are violated. System design is illustrated in figure 1 and a high-level schema of the approach and the modules are depicted in figure 3.

Our solution can be modified to work in real-time by processing a USB or Camera Serial Input. The demo runs on a video that can be provided via the configuration file.

Setup

Neuralet GitHub repository

Neuralet is an open-source platform for edge deep learning models on GPU, TPU, and more. The source code of the Smart Social Distancing application is available on GitHub under the application directory. You can find the setup guide in the project Readme.

git clone https://github.com/neuralet/neuralet.git
cd neuralet/applications/smart-distancing/

Jetson Nano

You will need an Nvidia Jetson Nano device (figure 2) to run this tutorial. We are making this application available on other devices, please find more details about compatible devices here. Jetson Nano is NVIDIA’s small and high-performance edge product that enables you to run a wide range of machine learning and deep learning applications.
To run this application you require to set up your Jetson Nano using JetPack 4.3 and have Nvidia-docker runtime available.

Object Detection Model

The core of this application uses Convolutional Neural Networks to detect pedestrians. We are working on adding more models, but as of right now, we use the pre-trained SSD MobileNet V2 which is trained on the MS COCO dataset for the proof of concept. We extract the “person” class of this model as the pedestrian detector.
We are running TensorRT runtime to get the best performance on the Jetson Nano. The model can be swapped easily in libs/core.py. You can implement other models by following our implementation example on libs/detectors/jetson.

Codebase Architecture:

In this section, we discuss our codebase structure and explain the main idea of each module. We highly recommend you to study this part thoroughly if you want to contribute or customize the behavior of the software for your specific application. In case you want to use the software as it is, you can skip this section.
In a high-level study, we can divide our code into two primary modules, which we review as follows:

Computer Vision Engine

The computational core of this application lies in this module. The Distancing class in the libs/Core.py file takes a video frame as the input and returns the coordinates of each detected object as well as a matrix of distances measured between each pair. The coordinates will further be used by the update  method in the WebGUI class to draw bounding boxes around each person.

We will now discuss the primary building blocks of the Distancing class.

Data Pre-Processing

To prepare each frame to enter the object detection model, we applied some pre-processing such as re-sizing and RGB transformation to the frames.

resized_image=cv.resize(cv_image,tuple(self.image_size[:2]))

rgb_resized_image = cv.cvtColor(resized_image, cv.COLOR_BGR2RGB)

Model Inference

In the constructor, we have a detector attribute that specifies the object detection model. We used SSD-MobileNet-v2 as the default model for this application. The average inference time on Jetson Nano was 44.7 milliseconds (22 frames per second).

The object detection model is built based on the config file and should have an inference method that takes a proper image as input and returns a list of dictionaries. Each dictionary of this list contains the information of a detected object, i.e., bounding box coordinates and object id.

tmp_objects_list = self.detector.inference(rgb_resized_image)

Bounding boxes post-processing

Since we used a general-purpose object detector (trained on COCO with 80 different classes) that has not been trained for our specific task, the output of this model needs partial post-processing to be robust for our specific purpose.

We applied three post-processing filterings to the raw bounding boxes to eliminate large boxes, collapse duplicated boxes for a single object, and keep track of moving objects in different frames.

These post-processing filterings are applied in the following lines of code:

New_objects_list = self.ignore_large_boxes(objects_list)

new_objects_list = self.non_max_suppression_fast(new_objects_list,                                        float(self.config.get_section_dict("PostProcessor")[     "NMSThreshold"]))


tracked_boxes = self.tracker.update(new_objects_list)

Compute distances

After post-processing, we need to calculate the distances between every pair of persons detected in each frame.

We use Python’s scipy library to calculate the distances between each pair of bounding box centroids.

centroids = np.array([obj["centroid"] for obj in new_objects_list])
distances = dist.cdist(centroids, centroids)

It is clear that the distances matrix is symmetric and has zeros on the diagonal.

User Interface (UI) and Web Application

The UI section is responsible for drawing bounding boxes around the detected persons at each frame. It uses different colors to show the distance between each pair of people.

The WebGUI object implements a Flask application and serves as an interface for the user. WebGUI constructor takes config and engine_instance parameters as inputs and acts as the central application for the output view.

Processing the video begins with the WebGUI start method. Within this method, the engine instance calls process_video to process the video frame by frame. This method returns a list of dictionaries for each detected object, a matrix of distances, and the image itself with the desired output resolution. These values are then passed to the update method of the WebGUI class that draws bounding boxes with proper colors around each object, i.e., person. 

How to Use Our Application

Config file

The user can interact with the code by changing some parameter values in the config file. Some parameters might need modifications to meet the user’s specific purposes. To run our application on any video file of your choice, set the VideoPath parameter to point to the target file.

The ConfigEngine class is responsible for parsing the config file. Feel free to change or add other parameters to the config file if you want to develop your customized version of the software.

Docker container

After customizing the config file, you can run the web application and see the output in your browser by running the following commands. Please note that you should have Docker installed on your device.

cd neuralet/applications/smart-distancing/

# 1) Download TensorRT engine file built with JetPack 4.3:
./download_jetson_trt.sh


# 2) build Docker image:
docker build -f Dockerfile-jetson-nano -t "neuralet/social-distancing:jetson-nano" .


# 3) run Docker container:
docker run -it --runtime nvidia --privileged -p HOST_PORT:8000 -v /PATH_TO_CLONED_REPO_ROOT/:/repo neuralet/social-distancing:jetson-nano

This container executes the neuralet_distancing.py file that wraps up all of the modules we mentioned earlier and streams the output video on the HOST_PORT.

Smart Social Distancing roadmap

As the project is under substantial development, lots of ideas can help us improve application performance and add other exciting features. Smart Social Distancing roadmap can best explain our priorities in the future of this project:

1. Evaluate and benchmark different models.
2. Improve the distance calculation module by considering perspectives.
3. Provide a safety score to a given site and calculate useful statistics about a specific time interval; for example, how many people entered the place, and how many times social distancing rules were violated.
4. UI improvements: show statistical and historical data in the GUI.
5. Aid the program optimization by profiling different parts of the software, from the CV engine to UI overheads.
6. Provide the ability for the user to customize and re-train models with task-specific datasets.

Check out our other open issues on our GitHub repo.

Contribution instructions

Due to the tremendous social impact that this service can provide and the fact that this project is under intensive development, any contribution from the community is warmly welcomed by our team. In this section, we present some examples of how you can contribute to the project.

Getting feedback and sending issues

The easiest way to participate in improving this project is to report feedback. If you find any potential bugs or have feature suggestions, please check other issues in the repository before opening a new issue to avoid duplicate issues.

When creating a new bug report, please make sure to include all the necessary details and the code snippet that reproduces the bug to help our team and the community fully understand the problem.

Send PR on current issues:

If you find a bug or an enhancement interesting to work on, first, check the current issues to make sure no one else is working on the same problem. Then, tell others you are working on this issue by posting an announcement on the corresponding issue page. Before sending a pull request, make sure that you add a brief description of what your PR solves and refer to the issue you worked on.

Share your experience of using Smart Distancing with the community

Finally, if you have used this software in practice at your office, your warehouse, or any other place, you can share your experience with the community and let us know if you find this software useful in practicing social distancing.

Stay connected

Please visit our project at Neuralet website for more information and subscribe to our newsletter to receive the latest news and updates about our products. Don’t forget to check out our GitHub page for other models and applications. 

This is the beginning of Galliot’s work on the Social Distancing Detection product.
You can skip to the technical documentation of this work and read about its new codebase architecture.
Head out to our GitHub and roadmap.
Check out our guide to data labeling methodology for building your datasets.

1. Intro

To Shop or Not to Shop
Call it whatever you like; as the United Nations chief puts it, it is “the worst global crisis since World War II.” Coronavirus Disease 2019, or COVID-19, has caused chaos and fear all over the world. All non-essential businesses and services have been shut down in 42 states due to the COVID-19 pandemic, and many businesses are struggling to survive. This pandemic will reshape how we live our lives for many years to come.

Essential businesses such as grocery stores and pharmacies remain open, but they are hotspots for COVID-19. The workers are at a high risk of contracting the virus, and so are the shoppers. White House’s response coordinator Deborah Birx said in a statement that “This is the moment not to be going to the grocery store, not going to the pharmacy, but doing everything you can to keep your family and your friends safe.” People are afraid to go out shopping for their essential needs, but with few alternatives, many have no choice but to make the trip. So how do we protect shoppers and workers at the store?

6 Feet of Separation…

So we are experiencing a deadly pandemic, but we still need access to essential goods and services. We also know that, according to the Center for Disease Control (CDC), to stay safe during this pandemic, we need to practice social distancing rules and keep a minimum distance of 6 feet from people outside of our households. Essential businesses are allowed to be open so long that they follow social distancing rules. There is a problem; however, stores have limited space, and people might break social distancing rules without even realizing it. So how do we work with the limited options we have to make shopping for essential goods safe?

In response to social distancing rules, stores were asked to limit the number of people who can be inside at the same time. Shoppers are forced to keep their distance while waiting in line to get in or during checkout. Some stores are marking where people should be standing, spacing everybody apart, and some supermarkets are testing one-way aisles.

Besides the entrance and checkout lines, there is nowhere else in the store where these safety measures are enforced. The best we can hope for is that shoppers will honor the social distancing rules while they shop. So far, creating long lines and increasing wait time for getting groceries has been our best hope in minimizing the spread of COVID-19, but there are drawbacks to this approach.

Longer wait times mean more time spent outside the house, and more time spent in public means a higher risk of exposure to COVID-19. Longer wait times have also led to overbuying, which stresses the supply chain, results in waste, and causes shortages for others. Many stores have brought in more staff to help manage the social distancing etiquette. This is not only resource-intensive but puts the health of personnel at risk. More importantly, what is the guarantee that all shoppers will maintain a safe distance at all times? Social distancing might work in theory but might not be practical without the appropriate measures and tools. Aisles are sometimes too narrow, high-demand items might be placed next to each other, and shoppers might simply forget or misjudge the safe distance, what if two customers want a product from the same shelf? It will quickly become apparent that social distancing at stores is not as simple as limiting the store’s total capacity. Ad hoc solutions to social distancing in the absence of relevant data could lead to unexpected outcomes, such as high-traffic areas and a higher risk of exposure.

2. What is Smart Distancing?

Social distancing is crucial in our fight against COVID-19; it may remain an essential part of our lives for months and reshape how we interact with the outside world forever. The current way of social distancing at stores is not a viable long-term solution, and we need to come up with a better way to restore the shopping experience while ensuring the safety of everyone.

We can be smarter in socially distancing ourselves at all times and make shopping more efficient and safer during COVID-19 and beyond. With the help of Artificial Intelligence (AI), the same technology that is the backbone of self-driving Teslas and Netflix recommendations, combined with edge computing, the technology that is reshaping the Internet of Things (IoT), we can practice social distancing with minimal disruption to our daily lives. Imagine a seamless integration of social distancing to our shopping experience powered by big data and AI. Using available data stores can better implement social distancing. For example, stores can change aisle traffic in real-time, identify hotspots and re-distribute products to eliminate them, and vary the number of cashiers to reduce long wait times while eliminating the risk of exposure to shoppers and workers.

3. Distancing Smartly with Galliot

Our open-source platform for deep learning models on edge devices has come up with an innovative solution to help us distance ourselves smartly during these unprecedented times. It can measure social distancing rates in stores, shopping centers, workplaces, and elsewhere.

Introducing an open-source application that uses existing infrastructure, such as security cameras, to measure social distancing rates automatically. It will then alert business owners if social distancing rules are being broken at their workplace.

This project works with existing hardware infrastructure and connects to your store camera, and uses an edge device such as Jetson Nano or Google Coral Dev Board to monitor social distancing. It also generates and processes data to make useful suggestions. For example, this model can identify high-traffic areas and specify high-demand items that need to be re-distributed. It can also analyze shopper activities and output statistics for the degree of store crowdedness at different hours of the day. These data analytics and suggestions can help managers make better decisions.

4. Smart Distancing on the Edge of AI

In summary, social distancing is our new normal, and we need to be respectful of it for the health and safety of ourselves and others. We can, however, be smart about how we distance ourselves in public places. smart distancing will help us be safe and efficient every time we shop by doing the following:

– It allows us to free up personnel and resources needed for the manual implementation of social distancing

– Using data and AI, this solution will intelligently standardize social distancing. Without a human in the loop, this application assures store owners, workers, and shoppers that social distancing standards are met at the highest level.

– This model takes advantage of existing hardware infrastructure and state-of-the-art embedded edge devices, eliminating the need for IT Cloud infrastructure investment.

– This approach preserves data privacy and security by processing data on edge devices at the user’s end. Other solutions use traditional cloud architecture, sending the data to an external environment.

– It can recognize high-traffic areas by analyzing historical data and help managers redesign the store accordingly and distribute shoppers along different aisles.

– The data generated by this application can be further processed and shared with shoppers. It can also make suggestions for the safest time to shop. By choosing the least crowded time to shop, shoppers can reduce the risk of exposure to COVID-19 and have a better shopping experience. The generated data can result in data-driven policies at the State or Federal level that can help us during future pandemics.

We hope everyone is staying safe and healthy during this pandemic. We are living through an unprecedented time and hope that technology can help us be safer and live better lives.

We have open-sourced the initial version of this solution on our GitHub page. We appreciate your contribution and effort in this project. You can reach out to us by covid19project@galliot.us