In this project I'll build a tele-operated robot using the Kria KR260 development board, a rover platform and QGroundControl station. The idea is to implement object detection and tracking on the video stream relayed to the ground control station.
The project is structured in 5 parts:
1. Autopilot software stack design
2. Ground Control software configuration
3. Video camera configuration
4. MPSOC AI network and object tracking
5. Robot electrical integration
1. Autopilot softwareThe rover SW architecture is a service oriented architecture composed of multiple independent services communicating via message queues.
The current autopilot software stack receives steering and throttle messages from QGroundControl, which are then sent to a differential controller which converts the quantities to PWM and direction values for each of the motors.
The closed loop implementation makes use of a 9-DOF IMU which is used as a feedback element. The driver converts the raw gyro/accelerometer to Euler angles via a Mahony filter. The yaw and pitch outputs are then subtracted from the QGC setpoints and sent to two PID control loops. The output of each PID is sent to the differential controller as on the open loop implementation.
Autopilot stack description
The autopilot stack is composed of the following sub-modules:
- Optional IMU driver
- Mavlink functionality to communicate with QGroundControl
- Slew rate controller
- Low pass filter
- Gyroscope calibration
- Differential controller
- Mahony filter
The QgroundControl application is used to send commands to the rover via Joysticks.
sudo usermod -a -G dialout $USER
sudo apt update
sudo apt remove -y modemmanager
sudo apt install -y gstreamer1.0-plugins-bad gstreamer1.0-libav gstreamer1.0-gl libqt5gui5 libfuse2 libsdl2-dev
Install the app
cd ~/Desktop
wget https://d176tv9ibo4jno.cloudfront.net/latest/QGroundControl.AppImage
chmod +x QGroundControl.AppImage
cd ~/Desktop
./QGroundControl.AppImage
Select
the units.
Select custom autopilot stack.
The main dependencies of the autopilot stack are mavrouter which routes the Mavlink commands:
mkdir autopilot
cd autopilot
git clone https://github.com/intel/mavlink-router.git
cd mavlink-router
git submodule update --init --recursive
sudo apt install git meson ninja-build pkg-config gcc g++ systemd
meson setup build .
ninja -C build
sudo ninja -C build install
sudo mkdir /etc/mavlink-router
sudo nano /etc/mavlink-router/main.conf
and the python libraries:
pip install pyquaternion
pip install pyserial
pip install spidev
pip install pymavlink
pip install PCA9685-driver # Servo driver
Configure Autopilot IP
The current autopilot stack requires manual configuration of IP address.
1. Login using IP address of KR260 and root as username.
2. Change directory under /etc/mavlink-router
3. Open file: sudo nano main.conf
4. Change IP of tablet with QGroundControl or IP of laptop with QGC on the first or third entry or add another new entry with a descriptive name representing another tablet or laptop device.
5. Change directory to /home/root/KR260Autopilot_OpenLoop
6. Open main file and change IP at the top. sudo nano main.py
7. Check status of mavlink router by issuing
sudo systemctl status mavlink-router
If status is correct proceed to 8 else stop and disable service, edit mavlink router and restart service:
$ sudo systemctl stop mavlink-router
$ sudo systemctl disable mavlink-router
$ sudo systemctl enable mavlink-router
$ sudo systemctl start mavlink-router
8. Start open-loop autopilot code.
9. Note that autopilot only prints and sends data when commands form QGC are present in the queue
10. At this point, the robot will connect with QGC.
Note: This procedure is provisional as the aim is to automate the IP allocation on the autopilot to do this automatically.
sudo nano /lib/systemd/system/autopilot.service
The content of which is:
[Unit]
Description=Kria Autopilot
After=network.target
[Service]
ExecStart=python3 /usr/bin/Autopilot_OpenLoop/main.py
WorkingDirectory=/usr/bin/Autopilot_OpenLoop/
Restart=always
#User=your_username
#Group=your_groupname
[Install]
WantedBy=multi-user.target
Start System services
sudo systemctl daemon-reload
sudo systemctl start mavlink-router.service
sudo systemctl start autopilot.service
sudo systemctl start kria-video-stream.service
Enable services on boot
sudo systemctl enable mavlink-router.service
sudo systemctl enable autopilot.service
sudo systemctl enable kria-video-stream.service
Check status of autopilot
sudo systemctl status mavlink-router.service
sudo systemctl status autopilot.service
sudo systemctl status kria-video-stream.service
Start servicesudo systemctl start mavlink-router.service
sudo systemctl start autopilot.service
sudo systemctl start kria-video-stream.service
QGroundControl configurationTo configure QGroundControl the user needs to know the IP address of the robot.
Since the robot is assembled and there is no access to the serial header on the main board, the IP address of the robot can be read from the tablet or laptop equipped with a WIFI radio.
1. Open web-browser on the tablet
2. Navigate under network DHCP. The mdns service shows the host name and IP address of the robot.
3. Open QGroundControl
4. Click on the Q image Select Tool on the corner.
5 Open Application Settings → Communication Links
6. Select UDP as the protocol of choice
7. Click Automatically connect on start.
8. Enter port 24550 and IP address of radio.
9. Enter port 14550 and IP address of radio.
10. Click OK to validate.
11. Click connect to connect the robot to QGroundControl.
12. The QGC is ready to accept connection from rover.
13. Enable virtual joysticks
The robot rover makes use of a USB camera. After plugging the USB camera, the camera supported formats are determined:
v4l2-ctl --list-formats-ext
This specific camera supports:
- YUYV (YUYV 4:2:2)
- MJPG (Motion-JPEG, compressed)
- NV12 (Y/CbCr 4:2:0)
A Gstreamer pipeline was used to send a video feed to the QGroundControl station.
gst-launch-1.0 v4l2src device=/dev/video0 ! \
"video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1" ! \
videoconvert ! \
x264enc bitrate=15000 speed-preset=ultrafast tune=zerolatency ! \
rtph264pay timestamp-offset=0 pt=96 name=pay0 ! \
udpsink host=${kria_qgc_ip} port=5600
This pipeline was saved as a script and a service was created to start this automatically.
Create a file called start_video_stream.sh in /usr/local/bin:
sudo nano /usr/local/bin/start_video_stream.sh
The scripts contents are as follows:
#!/bin/bash
# Define your pipeline
gst-launch-1.0 v4l2src device=/dev/video0 ! \
"video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1" ! \
videoconvert ! \
x264enc bitrate=15000 speed-preset=ultrafast tune=zerolatency ! \
rtph264pay timestamp-offset=0 pt=96 name=pay0 ! \
udpsink host=${KRIA_QGC_IP} port=5600
Then we start the service:
sudo chmod +x /usr/local/bin/start_video_stream.sh
sudo nano /etc/systemd/system/video_stream.service
video_stream.service
The service is as follows:
[Unit]
Description=Video Stream Service
After=network.target
[Service]
ExecStart=/usr/local/bin/start_video_stream.sh
Restart=always
User=your_username
Environment=KRIA_QGC_IP=your_kria_qgc_ip
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=video_stream
[Install]
WantedBy=multi-user.target
And it can be started and the status cane be checked as follows:
sudo systemctl daemon-reload
sudo systemctl start video_stream
sudo systemctl enable video_stream
sudo systemctl status video_stream
journalctl -u video_stream
Configure UDP port 5600 and select UDP h.264 Video Stream.
Note that the DHCP server can lease a different IP address to the Ground control desktop in which case the user has to change the IP address of the autopilot again or configure with a static IP.
Check status of autopilot
systemctl status autopilot.service
Start service:
systemctl start mavlink-router.service
systemctl start autopilot.service
systemctl start video-stream.service
Once the robot is configured the gstreamer pipeline is set to send H264 encoded video using RTP over a UDP connections to a specific client destination.
The gstreamer pipeline is started by kria-video-stream.service located in /lib/systemd/system/kria-video-stream.service
The service file uses the Address entry under UdpEndpoint QGCDesktop section in /etc/mavlink-router/main.conf for the destination IP address for the streamed video.
[General]
TcpServerPort=5760
ReportStats=false
MavlinkDialect=auto
[UdpEndpoint QGCDesktop]
Mode = Normal
Address = 192.168.168.163
The following gstream pipeline command is executed by kria-video-stream.service. Note the entry udpsink host=${cam_qgc_ip}, the cam_gc_ip value is read from the /etc/mavlink-router/main.conf.
To start streaming video to QGroundControl. Where cam_qgc_ip is the QGroundControll’s IP Address.
To support AI network on the PL the PYNQ library was installed:
git clone https://github.com/Xilinx/Kria-PYNQ.git
cd Kria-PYNQ/
sudo ./install.sh -b 'KR260'
kria:9090/lab
pip3 install pynq-dpu --no-build-isolation
cd $PYNQ_JUPYTER_NOTEBOOKS
pynq get-notebooks pynq-dpu -p .
5. Robot integrationSince Kria KR260 does not have WIFI we can add one using a PMOD or RPI daughterboard via SPI. The KR260 serves as the brain for the rover. The differential drive autopilot data (Z and R components ) are sent to the servo.
There are two components in progress:
A) Adding a WIFI module to KR260
B) Modifying HW design with I2C to support servo
C) Electrical integration.
The KR250 is fed from a LI-Ion battery supplying 5V. This needs to be bumped up to 12V to meet the wall wart specs. At the moment I'm waiting for a SMPS part to arrive.
ConclusionThis project implemented a tele-operated robot with AI capabilities using Kria SOM. A custom autopilot using mavlink protocol was implemented so that the robot could operate with QgroundControl application.
Comments