AI Apps Integrating Linux with RTOS for Real Time Detection

INTRO

This system is a power-optimized, battery-operated camera system with AI/ML functionality, for object detection and classification. The camera can be powered from USB Type- C, solar cells, or batteries. Motion detection functionality ensures that the camera is only turned on when the object is detected.

System Benefits

• Low sleep power for extended battery life
• Low power sensor monitoring, triggering trigger MPU wake up
• Reduced power stand-by current of PMIC via power enable signal and MCU
• Power-optimized RA2E1 MCU with only 0.25mA standby current (without RTC) and 0.36mA with RTC
• Selection of low leakage power devices to extend battery life
• MPU fast boot option within 500ms
• Certified BLE module DA14531MOD for fast design cycle.

Target Applications:

• Target in Motion Detection
• Battery powered camera

System Diagram

Boot Flow: Overview

• U-boot is removed in the boot sequence.
• ATF kicks CM33 core.
• CM33 core initializes the camera and starts capture image.

Boot Flow: ATF

After removing u-boot, ATF will load RTOS image + Kernel Image + Device tree from the Flash to RAM.

Software Configuration

The software configuration consists of two main steps:

• Prepare rootfs in eMMC/microSD Card

Prepare rootfs in eMMC/microSD Card at "/dev/mmcblk0p1" (By set as default device for booting). Linux will automatically mount rootfs from that partition.

• Flash Images to QSPI

Flashing Bootloader will take time due to the big size of the Image (fip-rzboard.srec – It includes all binaries of Linux and RTOS).

Prepare rootfs in eMMC/microSD Card

Hardware setup

In our use-case, we have to prepare the optimized rootfs in eMMC. Booting via TFTP server and NFS-kernel-server first is necessary. By that, we will be able to write the optimized rootfs into eMMC. And booting via eMMC later (just prepare rootfs only).

In order to boot via Ethernet, the Linux kernel must support ethernet booting. Unfortunately, our optimized package do not support this feature. That’s why we share the “anvet_normal_package.zip”for booting via Ethernet.

All the files can be downloaded in the link here: (RZ-AVNET_LinuxBSP_RTOS-Integration_Package_Rev0.5).

Extract Normal Package

On Linux Host, extract file “avnet_normal_package.zip"

bn@nc:~$ unzip avnet_normal_package.zip
bn@nc:~$ cd avnet_normal_package/
bn@nc:~$ ls

Follow the instructions on this hackster.io project regarding network booting.

Login with root

After setting up the network boot login in RzBoard with root privileges.

On the RZBoard, press and hold the S1 button to power on the RZ/V2L. Verify that the U-boot/Linux boot messages display via serial cable.

Poky (Yocto Project Reference Distro) 3.1.14 rzboard ttySC0
rzboard login: root
Password: avnet

PARTITION emmc

Follow the next steps to partition the emmc

1. To show current partitions.

root@rzboard:~# fdisk -l

2. Delete 2 old partitions:

root@rzboard:~# fdisk /dev/mmcblk0

3. Create a new partition:

root@rzboard:~# fdisk /dev/mmcblk0

4. Format partition 1 to ext4:

root@rzboard:~# mkfs.ext4 /dev/mmcblk0p1
root@rzboard:~# fdisk -l

Copy the rootfs to emmc

Login into the Linux Host where you have the network rootfs.

After login into Linux userland, you can see that all the files in board rootfs also can be seen on Linux host.

bn@nc:~$ ls /nfs/rzv2l/
bin   dev  home  lib64  mnt   run   sys  usr
boot  etc  lib   media  proc  sbin  tmp  var

Next, we will mount eMMC device, copy and tar the optimized rootfs into it for boot via eMMC later.

Please find the optimized rootfs “avnet-core-image-rzboard.tar.bz2” in the “RZ-AVNET_LinuxBSP_RTOS-Integration_Package.zip”and copy it to “/nfs/rzv2l/”.

bn@nc:~$ ls /nfs/rzv2l/
avnet-core-image-rzboard.tar.bz2  bin   dev  home  lib64  mnt   
run   sys  usr  boot  etc  lib   media  proc  sbin  tmp  var

On RzBoard, run the following commands:

root@rzboard:~# mkdir –p /mnt/rootfs
root@rzboard:~# mount /dev/mmcblk0p1 /mnt/rootfs
root@rzboard:~# cd /mnt/rootfs
root@rzboard:/mnt/rootfs# rm –rf *
root@rzboard:/mnt/rootfs# cp /avnet-core-image-rzboard.tar.bz2 .
root@rzboard:/mnt/rootfs# tar xf avnet-core-image-rzboard.tar.bz2
root@rzboard:/mnt/rootfs# sync
root@rzboard:/mnt/rootfs# cd
root@rzboard:~# umount /mnt/rootfs

Now the optimized rootfs is available in eMMC. We need to write the integrated Bootloader again.

After that, just press the power button, and the optimized system will boot up.

Flash Images to QSPI

In this section, we will go through, how to flash images to the RzBoard, working either from a Linux host or a windows Host.

Download the flashing tools, with the pre-built binaries here (RZBoard_dashboard_trailcam_05.zip)

Hardware Setup

Verify that the USB to serial cable is connected properly.

1. RXD is RzBoard's Receiver, which means the transmitting cable to Rzboard goes there. The Same for TXD which is RzBoard's Transmitter to the other device. The RED cable connector should not be connected.

2. Set the bootmode to SCIF serial mode: This done by setting BOOT2=1 by strapping J19-pin1 to +5V(ie. connect fly-wire from J19-pin1 to J1-pin2 on the 40pin header)

Linux Host Images Flashing

If you are working on a Linux host, download and extract files in “RZBoard_dashboard_trailcam_05.zip"

bn@nc:~$ unzip RZBoard_dashboard_trailcam_05.zip
bn@nc:~$ cd RZBoard_dashboard_trailcam_05/images
bn@nc:~/RZBoard_dashboard_trailcam_05/images$ chmod a+x flash_util.py

Make sure:

• RZboard is powered down
• Then run the following command.

bn@nc:~/RZBoard_dashboard_trailcam_05/images$ flash_util.py --trailcam

Wait till the flashing is complete ( <50 min).

Windows Host Images Flashing

If not already installed, download and install Tera Term serial terminal software: https://osdn.net/projects/ttssh2/downloads/74780/teraterm-4.106.exe/.

Plug in the USB-to-Serial Debug Console Cable to the PC. Ensure it is recognized by the Windows Device Manager. It should come up under Ports (Com & LPT).

Note! The Comm port number assigned by the PC may differ from the image above.

Prior to launching either of the.bat files, make sure of the following:All Tera Term windows are closed!

• USB serial cable from PC has TX, RX, GND fly-leads connected to correct J19 header pins and the COM port has enumerated on the PC.
• The config.ini file has been edited to list: - the correct COM port for the attached USB-Serial cable- the correct name for the.wic Linux image file (if an updated version is used).

With RZBoard powered OFF, from Windows file explorer run flash_bootloader.bat (this launches the applicable Tera Term macro using the edited config.ini settings).

Choose QSPI as the flash to program, the macro then waits for power up...Press and hold S1 for 2 seconds to power on RZBoard, the macro will now proceed. Wait for this to complete (<50 min). Make sure to shut down the Tera Term window after this has been completed.

Boot

After completing, change again to Boot mode by:

• Power off
• Disconnect jump cable
• Change SW-1: 1-OFF, 2-OFF

Trail camera Demo Setup

Steps to connect RZ board with FBP-RA4E1/ FPB-RA2E1

Step 1: Connect QCIOT-014 Motion Sensor board to PMOD1 on FPB-RA4E1 board.

Step 2: Connect the Switch to PMOD2 on FPB-RA4E1.

Step 3: Provide 5V-16V to Motion sensor board.

To provide 5V, Connect as follows:

Motion Board FPB-RA4E1

Positive J2-5

Negative J2-6

Note: Dot mark near J2 indicates pin1.

Step 4: Connect make wire connection between J6-3 to PMOD1 pin 2.

Step 5:Connect a jumper wire on J5-8. Connect the other end to CEN pin on Rz board.

Step 6: Short the SW1 switch on Rz board.

Step 7: Flash the code on FPB-RA4E1.

• Follow the instructions in this manual, section "2. Installation" on how to install e2 studio for RA devices.
• Download the Motion_Sensor_Demo.zip project and follow instructions in the section "3.3 Importing an Existing RA Project" to import it into e2 studio.
• Double-click on the "configuration.xml" file and agree with the suggestions, then follow the instructions in the section "4.2 Building a Sample Project" to build the project.

• Connect FPB-RA4E1 to Laptop using USB cable.
• Follow the instruction in the section "5.1 Changing an Existing Debug Configuration" then click debug without changing any configuration.

Step 9: Turn ON the FPB-RA4E1 and RZ board.

Confirm Rz features operation

Press the power button (S1) on the RZ-AVNET board, log in with “root” and password “avnet”.

Confirm FreeRTOS and Linux cooperation

Run the following command to choose the supported features:

root@rzboard: cd ~/app-run
root@rzboard: chmod a+x capture_v1.0.sh
root@rzboard: ./capture_v1.0.sh <option> <option2>

There are three options for running:

• -s: 1st image only
• -c: continuous display
• -a: proceed AI inference for a selected picture <option2>and display the result

To display the captured images, run the following command:

To process AI inference, run the following command: