Small parts detection at municipal waste sorting plant

Problem to solve

The waste comes from collection of recycling bins. At the (manual) sorting plant, workers each pick one type of recyclable item (plastic bottle or cardboard or..) from the conveyor belt , as fast as they can, to put into the output bin as sorted material. The problem is all the small recyclable items pass through and get away, because workers don't have time to get a hold on it; they have to wear clumsy gloves for protection.

Proposed solutions

This project builds a vision recognition system to identify the recyclable small items, and command a small low cost robot arm in real time to recover these items. This system is much more cheaper than a robotic plant solution. It is more likely adopted by municipal sorting plants as a helper, without modifying the existing workflow or installation. Judging by the quantity of dumped small parts on the sorting line, this solution can increase the recycling ratio by 10%. It is also a cost-effective initial step towards a complete AI driven, high-performance autonomous sorting plants.

AI aspect of the solution

A typical MobileNet(V1)-SSD Convolutional Neural Network is trained with photos of waste items in our municipal sorting plant. Loaded with the trained model, a NVIDIA Jetson Nano is fed the video view of the sorting belt loaded with recyclable items. Each time a item is identified, Jetson Nano signals the position of the item on the belt and its type to the robot arm. The robot arm then pick up the item. Beyond the time frame of this project, the robotic arm and its control and planning logic will be for future development.

All training, evaluation, coding are done on jetson nano board. Jetson nano is remotely accessed from a ubuntu PC. Only the annotation is done on ubuntu PC under Firefox browser, then the annotated training set is transferred back to jetson nano before starting the training. Though the annotation task can be performed on jetson nano too, it is slow and to be so for any GUI intensive CPU tasks.

Software packages installation on Jetson nano board

1) OS installation on Jetson nano board.

• Run Balena Etcher on Ubuntu to flash nv-jetson-nano-sd-card-image on a 64G SD Card
• Insert the sdcard into Jetson Nano and power on.
• Use local USB keyboard, connect Nano hdmi to TV, connect Ethernet to the box/router. Locally configure a admin user: login user=jetson, passwd=jetson, hostname=jetson, and note down the IP allocated to Jetson Nano by the router: 192.168.1.46
• Remotely access Jetson Nano from a PC for ease of operations (see next - details 2).

also write down the PC's IP address (a ubuntu PC in this text):

ip a
    wlp3s0    Link encap:Ethernet  HWaddr 28:b2:bd:ae:4c:fa
    inet addr:192.168.1.26  Bcast:192.168.1.255  Mask:255.255.255.0

In summary, both a unbuntu PC (named local PC in this text) and the Jetson nano board (named remote Jetson nano) are connected to the home wifi network, with the following IP addresses:

• local ubuntu PC: 192.168.1.26
• remote Jetson nano board: 192.168.1.46

2) Use VSCODE as remote IDE ("Visual Studio Code" - not "Visual Studio")

VSCODE is only installed on ubuntu PC (PC is local vs Jetson nano which is remote); VSCODE is not installed on Jetson nano.

This is the most efficient workflow I found to operate on Jetson nano remotely from PC. Hope it benefits other hacksters too.

• Vscode is free, install it from https://code.visualstudio.com/, available for all OS, We use Linux version.
• Press "Install the Remote - SSH extension" button on page https://code.visualstudio.com/remote-tutorials/ssh/getting-started
  to install the extension for vscode that allows the vscode running on local PC to operate on remote jetson nano.

• After installed the Remote SSH extension, with VSCODE launched, click on '><' icon on the extreme bottom left corner to start remote ssh login,

- choose 'Remote-SSH:Connect to Host..' from the drop-down list,

- enter 'jetson@192.168.1.46' in the input area then Enter,

- enter passwd 'jetson' into the next input area then Enter,

you will log onto Jetson nano as the user 'jetson' whose passwd is 'jetson'.

• Now on VSCODE main page, click on the menu item 'Start, Open folder' to remotely open a directory located on Jetson nano, and remotely edit/modify any files on Jetson nano, with VSCODE running on PC.

• Right-click anywhere in remote directories which are displayed on locla VSCODE, 'Open in Terminal' to start a new SSH terminal in VSCODE sub-window that is ssh'ed onto Jetson nano, and in the exact remote folder where the terminal has been opened.

• From that Terminal, software installation on Jetson nano can be continued.

In short, VSCODE serves as both remote IDE and remote SSH terminal, that operates on Jetson nano but it runs on local PC and its GUI is on local PC screen.

3) Install Pytorch and other Linux packages

From VSCODE remote terminal, with 'jetson' / 'jetson' as sudo user and password, we installed Pytorch by installing the nvidia sdk example 'jetson-inference':

cd
git clone https://github.com/dusty-nv/jetson-inference
cd jetson-inference
git submodule update --init

sudo apt-get install python-numpy python-scipy
cd ~/jetson-inference/build
./install-pytorch.sh

Verify the installation:

python
Python 2.7.17 (default, Nov  7 2019, 10:07:09)
>>> import torch
>>> torch.__version__
'1.1.0'
>>> import torchvision
>>> torchvision.__version__
'0.3.0'

Additional Linux packages:

pip install typing
pip install pathlib
pip install pandas

4) Customize Jetson nano for training (mandatory operation)

To give Jetson nano Linux enough (virtual) memory to support Pytorch training, we must add additional swap disk (on the booting 64G sdcard of the board).
Always in VSCODE remote SSH terminal:

sudo fallocate -l 4G /mnt/4GB.swap
sudo mkswap /mnt/4GB.swap
sudo swapon /mnt/4GB.swap
sudo vi /etc/fstab # append one new line:
    /mnt/4GB.swap  none  swap  sw 0  0

By default Jetson nano consumes 10W, without an additional noisy and annoying fan, Jetson nano shuts itself down after 10 minutes running continuously the training. We retrained the board to 5W by decreasing its performance by an acceptable 30%:

sudo nvpmodel -m 1
Current mode: NV Power Mode: 5W
max_freq: REAL_VAL: 921600000 reduced to-> 640000000

5) Install our project "detect-recyclable" from github

Run
git clone https://github.com/minghuang81/detect-recyclable.git
in VSCODE remote SSH terminal:

jetson@jetson:~$ git clone https://github.com/minghuang81/detect-recyclable.git
Cloning into 'detect-recyclable'...
remote: Enumerating objects: 270, done.
remote: Total 270 (delta 0), reused 0 (delta 0), pack-reused 270
Receiving objects: 100% (270/270), 525.49 MiB | 13.94 MiB/s, done.
Resolving deltas: 100% (9/9), done.
Checking out files: 100% (253/253), done.

There are two folders in the project:

jetson@jetson:~$ ls detect-recyclable
dataset_recyclable  py  README.md

The datasets along with their annotations:

jetson@jetson:~$ ls detect-recyclable/dataset_recyclable
sub-test-annotations-bbox.csv  sub-validation-annotations-bbox.csv train
sub-train-annotations-bbox.csv test                                validation

The application code of our project:

jetson@jetson:~$ ls detect-recyclable/py
draw_annotations.py   models                      run_example.py
draw_results.py       open_images_downloader.py   train_recycle.py
eval_recycle.py       open_images_downloader.sh  'VGG Image Annotator.html'
LICENSE               README.md                   vision
log.py

Model training for recyclable items detection

1) Image annotation py/VGG Image Annotator.html

Image annotation consists in drawing a tight box around each recyclable item on photos in training, validation and test sets. The photos in training set are read by the training program to train the model (train_recycle.py); the validation set is also used during the training for error metrics; the test set is not used in model training, you don't have to use them, but you can use them to run the generated model (Inference) on them to see how the model is efficient.

From VSCODE file explorer, copy the dataset 'dataset_reclable/train' and the tool 'VGG Image Annotator.html' from remote jetson nano board to the local ubuntu PC. Then we do annotation on local PC, because it is way too slow to run the tool on jetson nano board. The annotation tool is a standalone .html file running in web browser.

On local ubuntu PC, double-click on 'VGG Image Annotator.html' to run the annotation tool in any Browser:

• 'Image>Load or Add Images' to open all images in dataset_recyclable/train at once (multiple-selection)

• box the cyclables by press-and-drag left mouse button

• 'Keyboard Shortcuts' +/- : press arrow key '->' to go to the next image among loaded images
• Box again, '->' again
• 'Annotation > Save as CSV' to save under one of the following file names and under these names only, because the training program (py/train_recycle.py) has hard-coded the Open Images naming convention:
  * sub-test-annotations-bbox.csv
  * sub-validation-annotations-bbox.csv
  * sub-train-annotations-bbox.csv

Also for reason of Open Images naming convention, the folders of the dataset are:
* train
* test
* validation

2) Run py/train_recycle.py to train a model

Train a new model on images in detect-recyclable/dataset_recyclable/train. The training starts with the pre-trained model checkpoint, namely py/models/pretrained-ssdv1.pth. The path to the pre-trained model is hard-coded.

Being a transfer-learning, the training leaves the base-net of the model checkpoint unchanged, it modifies the SSD part of the model parameters for re-converging the classification to either 'background' or our 'recyclable' classes; and it also modifies the SSD part of the model parameters for boxes coordinates because our recyclables have new locations on images, comparing to those objects that were used to train pretrained-ssdv1.pth.

Recall the MobileNet-SSD-v1 structure is a MobileNet v1 (https://arxiv.org/abs/1704.04861v1) + a SSD v1 (https://arxiv.org/abs/1512.02325).

Check (remotely) on Jetson nano board that 'train_recycle.py' file is there:

jetson@jetson:~/detect-recyclable$ cd ~/detect-recyclable/py
jetson@jetson:~/detect-recyclable/py$ ls
draw_annotations.py   models                      train_recycle.py
draw_results.py       open_images_downloader.py  'VGG Image Annotator.html'
eval_recycle.py       open_images_downloader.sh   vision
LICENSE               README.md
log.py                run_example.py

For lack of timely keyboard input, the SSH session between jetson nano and the local PC expires for inactivity after about 30 mins of training; we see the same behavior with SSH sessions etablished from MobileXterm from Windows PC too. Without sitting all the time in front of the PC and always typing something into the SSH terminal to jetson nano, the SSH session does not last enough long to finish a training. We have to directly connect a keyboard and a HDMI screen to Jetson nano board for the training.

With direct keyboard and HDMI screen to jetson nano, run on Jetson nano board:

python train_recycle.py --datasets ../dataset_recyclable --num_epochs 20

The output of the training are the trained model and a list of object classes that have been annotated on training images:

• models/outModel.pth
• models/outModel-labels.txt

Terminal output:

- train_recycle.py:171 Stored labels into file models/outModel-labels.txt
...
- train_recycle.py:243 Epoch: 19,Validation Loss: 8.02826786041, Validation Regression Loss 5.57354521751, Validation Classification Loss: 2.4547226429
- train_recycle.py:247 Saved model models/outModel.pth

For our purpose, we have only annotated a single class : 'recyclable' during the step 1) Image annotation. All recyclable items being it cups or bottles or cans are labelled equally 'recyclable'. Thus outModel-labels.txt contains only two classes, with 'background' to denote any objects that are not our recyclables:

cat ~/detect-recyclable/py/models/outModel-labels.txt
BACKGROUND
recyclable

If you wonder why we detect 'background' objects any way, it is because the 1000 classes used to train the pre-trained model are translated into our 'background' class. Of course that pre-training was done by other people on their powerful GPU, we just downloaded the result and use it as the pretrained model (pretrained-ssdv1.pth) for transfer-learning.

If the traing is interrupted for any reason, for instance after 5 successful epochs, resume the training from the latest intermediate auto check-point, specify --num_epochs for the remaining number of epochs:

python train_recycle.py --datasets ../dataset_recyclable --num_epochs 15 --resume models/outModel.pth

3) Validation of our new model by one example

Using the newly trained model, let's make inference on one example of the test data:

cd ~/detect-recyclable/py
python run_example.py  models/outModel.pth models/outModel-labels.txt ../dataset_recyclable/test/IMG_20200210_133430.jpg

Terminal output:

Found 2 objects. The output image is run_example_output.jpg

to visualize the detection outcome:

xdg-open run_example_output.jpg

where the Linux 'xdg-open' is the command line equivalence of double-click on jpg file in ubuntu desktop or Windows environment.

4) Evaluate our new model on entire test set

To evaluate the model on the test subset of a Open Images dataset, namely ~/detect_recyclable/dataset_recyclable/test/*.jpg:

python eval_recycle.py --trained_model models/outModel.pth  --label_file models/outModel-labels.txt --dataset ../dataset_recyclable --eval_out /tmp

Terminal output:

results are printed into file
/tmp/det_test_recyclable.txt
format:
ImageID,xxx,LabelName,Confidence,XMin,XMax,YMin,YMax,0,0,0,0,0,LabelName,LabelName
Average Precision Per-class:
recyclable: 0.227272727273
Average Precision Across All Classes:
0.227272727273
Average Time for loading an image:
0.175064590242
Average Time for prediction:
0.127658261193

all images in "dataset_recyclable/test" directory are passed one by one into the predictor (inference) using the parameters of our new model. A machine learning metrics - precision, is shown in the Terminal output. However, the detected boxes are not shown in the Terminal output, they are cumulated in one and single annotation text file: : /tmp/det_test_recyclable.txt.

To visually check the inference result, we need to overlay the detected boxes onto the images:

python draw_annotations.py /tmp/det_test_recyclable.txt ../dataset_recyclable/test /tmp/drawn5

the redrawn images are placed into the folder /tmp/drawn5:

jetson@jetson:~/detect-recyclable$ cd /tmp/drawn5
jetson@jetson:/tmp/drawn5$ ls
IMG_20200210_133430.jpg  IMG_20200210_133435.jpg  IMG_20200210_133441.jpg  
IMG_20200210_134140.jpg  IMG_20200210_134141.jpg  IMG_20200210_134149.jpg  
IMG_20200210_134150.jpg

.