Remote Control 360° Camera from KR260

I tried controlling the RICOH THETA V 360° camera from the KR260 using PYNQ.

In this project, I will introduce the installation method and provide examples of how to execute the control commands.

Subproject for the AMD Pervasive AI Developer Contest

This project is part of a subproject for the AMD Pervasive AI Developer Contest.

Be sure to check out the other projects as well.

***The main project is currently under submission. ***

0. Main project << under submission

1. PYNQ + GPIO(LED Blinking)

2. PYNQ + PWM(DC-Motor Control)

3. Object Detection(Yolo) with DPU-PYNQ

4. Implementation DPU, GPIO, and PWM

5. Remote Control 360° Camera << this project

6. GStreamer + OpenCV with 360°Camera

7. 360 Live Streaming + Object Detect(DPU)

8. ROS2 3D Marker from 360 Live Streaming

9. Control 360° Object Detection Robot Car

10. Improve Object Detection Speed with YOLOX

11. Benchmark Architectures of the DPU

12. Power Consumption of 360° Object Detection Robot Car

13. Application to Vitis AI ONNX Runtime Engine (VOE)

14. Appendix: Object Detection Using YOLOX with a Webcam

Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.

https://github.com/amd/Kria-RoboticsAI

Introduction

I successfully controlled the RICOH THETA V 360° camera from the KR260 via USB.

Here are the photos taken by the 360° camera:

Below is the test video.

You can see that the 360° camera is controlled from an.ipynb file.

Installation Steps

The 360° camera used is the RICOH THETA V. It is a user-friendly 360° camera with various APIs and libraries available.

Connecting THETA(360° camera) to KR260

Connect the THETA(360° camera) to the KR260 using a USB cable.

Checking the logs with dmesg shows the successful connection.

ubuntu@kria:~$ sudo su
root@kria:/home/ubuntu# dmesg | grep usb

[    8.214013] usb 1-1.2: New USB device found, idVendor=05ca, idProduct=0368, bcdDevice= 1.00
[    8.222380] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    8.229697] usb 1-1.2: Product: RICOH THETA V
[    8.234055] usb 1-1.2: Manufacturer: Ricoh Company, Ltd.
[    8.239367] usb 1-1.2: SerialNumber: 00119628

Building and Installing the Library

Follow the steps below to build and install the library from GitHub.

https://github.com/codetricity/libptp2-theta

sudo apt install build-essential libtool automake pkg-config subversion libusb-dev

git clone https://github.com/codetricity/libptp2-theta
cd libptp2-theta/
./configure 
make

autoreconf -i
./configure 
automake
make
sudo make install
sudo ldconfig -v

Controlling THETA via USB from KR260

After installation, you can use the theta command. For example, to check the help, use theta -h, and to read the camera (device) information, use theta -i.

root@kria:/home/ubuntu# theta -h
root@kria:/home/ubuntu# theta -i

THETA Device Info
==================
Model: RICOH THETA V
  manufacturer: Ricoh Company, Ltd.
  serial number: '00119628'
  device version: 1.00.2
  extension ID: 0x00000006
  image formats supported: 0x00000004
  extension version: 0x006e

Jupyter Notebook

Using the Kria-PYNQ Jupyter Notebook, you can control the camera from Python.

This example shows one possible method, but various operations are possible using the official RICOH USB API information.

https://github.com/ricohapi/theta-api-specs/tree/main/theta-usb-api

Below is the test.ipynb file. It has been uploaded to GitHub.

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/blob/main/jupyter_notebooks/theta-check/theta_camera_check.ipynb

ipynb File Example

To check the camera information in Python, use the following code:

from subprocess import getoutput
import time
theta_info = getoutput('theta --info')
print(theta_info)

THETA Device Info
==================
Model: RICOH THETA V
  manufacturer: Ricoh Company, Ltd.
  serial number: '00119628'
  device version: 1.00.2
  extension ID: 0x00000006
  image formats supported: 0x00000004
  extension version: 0x006e

The output will show detailed information about the connected THETA camera.

Capturing Images

To wake up the THETA from power-saving mode. The camera's LED light will turn blue.

theta_wakeup = getoutput('theta --set-property=0xD80E --val=0x00')
print(theta_wakeup)

Camera: RICOH THETA V
'UNKNOWN' is set to: 0
Changing property value to 0x00 [(null)] succeeded.

Switch to camera shooting mode. The camera icon will light up blue.

theta_camera_mode = getoutput('theta --set-property=0x5013 --val=0x0001')
print(theta_camera_mode)

Camera: RICOH THETA V
'Still Capture Mode' is set to: [Normal]
Changing property value to 0x0001 [(null)] succeeded.

All that's left is to take the photo (capture).

I checked the time it takes for the THETA V 360° camera to capture an image, and it was about 3 seconds.

time1 = time.time()
theta_capture = getoutput('theta --capture')
time2 = time.time()
capture_time = time2 - time1
print("Performance: {} (s)".format(capture_time))
print(theta_capture)

Performance: 3.308269500732422 (s)
Initiating capture...
Object added 0x000000f2
Capture completed successfully!

You can check the files stored on the THETA using the command theta -L.

The retrieved data includes the following photos.

theta_list = getoutput('theta -L')
print(theta_list)

Listing files...
Camera: RICOH THETA V
Handler:           Size: 	Captured:      	name:
0x000000ad:      2042821	2024-03-30 23:35	R0010146.JPG
0x000000ae:      2021885	2024-03-30 23:36	R0010147.JPG
0x000000af:      1801502	2024-03-31 07:01	R0010148.JPG
0x000000b0:      1861412	2024-03-31 08:06	R0010149.JPG
0x000000b1:      1716924	2024-03-31 09:18	R0010150.JPG
0x000000b2:      1723153	2024-03-31 09:22	R0010151.JPG

You can specify a file (or all files) and save them to the KR260.

Use the commands theta --get-file=HANDLE or theta --get-all-files.

The files will be transferred from the THETA to the current directory.

theta_get_file = getoutput('theta --get-file=0x00000088')
print(theta_get_file)

Camera: RICOH THETA V
Skipping file: "R0010155.JPG", file exists!

Displaying Captured Images

To display the captured images using OpenCV, use the following code:

import cv2
from matplotlib import pyplot as plt

image_path = 'R0010109.JPG'
image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.show()

Reference

Many thanks for reference articles.

https://github.com/codetricity/libptp2-theta

https://github.com/ricohapi/theta-api-specs/tree/main/theta-usb-api

Conclusion

I successfully controlled the RICOH THETA V 360° camera from the KR260.

However, it takes about 3 seconds to capture one image, which is not suitable for real-time control.

Next time, I will introduce a method to stream real-time video to OpenCV using GStreamer.

6. GStreamer + OpenCV with 360°Camera << next project