Marker-Based Augmented Reality on Ultra96-V2

Introduction

Augmented Reality (AR) is a live view of a real-world environment which has been augmented with computer-generated graphics. There are two ways of implementing AR:

• marker-based AR
• marker-less AR

Marker-less AR is more complex and relies on the real environment to determine references points. This usually involves recognizing objects and or background cues such as floors and walls.

Marker-based AR is simpler to implement and relies on square markers located in the scene.

This project will describe how to implement marker-based AR on the Ultra96-V2 using OpenCV.

Let's get started !

Inspiration

Before getting started, I wanted to share the inspiration and motivation for this project.

I was inspired by the following innovative products/tutorials:

• Pantone, Color Matching Card, https://www.pantone.com/pantone-color-match-card
• Adrian Rosebrock, Detecting ArUco markers with OpenCV and Python, PyImageSearch, https://www.pyimagesearch.com/2020/12/21/detecting-aruco-markers-with-opencv-and-python/ (accessed on 07 May, 2021)

Motivation

My motivation for implementing something similar was to be able to use the markers to automatically trigger some kind of calibration, such as:

• white balance, using a white reference chart
• stereo calibration, using a checkerboard reference chart

In order to achieve this, I created the following three charts (with Microsoft Word) to experiment with.

In this project, I will detect the presence of these charts, and perform additional processing, depending on the chart:

• Chart 1 - Draw a green box around the checkboard pattern
• Chart 2 - Measure the average values of B, G.R pixels in the area within the markers, and display bar graph with values on chart
• Chart 2 - Measure the color histograms in the area within the markers, and display histograms on chart

For more information on generating these markers, please refer to the following excellent tutorials:

• Adrian Rosebrock, Generating ArUco markers with OpenCV and Python, PyImageSearch, https://www.pyimagesearch.com/2020/12/14/generating-aruco-markers-with-opencv-and-python/ (accessed on 07 May, 2021)

Detecting Markers in the Scene

With our goals defined, and our charts with markers created, we are ready to jump into the implementation.

I chose to implement this project in C++, as a gstreamer plug-in. I need to thank Tom Simpson for setting up the ground work for creating gstreamer plug-ins.

The first step is to detect the markers in the scene. This is accomplished with OpenCV.

/* Aruco Markers */
#include <opencv2/aruco.hpp>

...

//
// Detect ARUCO markers
//   ref : https://docs.opencv.org/master/d5/dae/tutorial_aruco_detection.html
//
std::vector<int> markerIds;
std::vector<std::vector<cv::Point2f>> markerCorners, rejectedCandidates;
cv::Ptr<cv::aruco::DetectorParameters> parameters = cv::aruco::DetectorParameters::create();
cv::Ptr<cv::aruco::Dictionary> dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_ARUCO_ORIGINAL);
cv::aruco::detectMarkers(img, dictionary, markerCorners, markerIds, parameters, rejectedCandidates);
if ( markerIds.size() > 0 )
{
  cv::aruco::drawDetectedMarkers(img, markerCorners, markerIds);
}

At this point, you will notice that I am using the DICT_ARUCO_ORIGINAL series of markers. The markers contain a grid of 5x5 squares, surrounded by a black border.

For each detected marker, the OpenCV API returns the following information:

• markerIds : identification of each marker
• markerCorners : coordinates of each marker, in following format: [0] top-left [1] top-right [2] bottom-right [3] bottom-left

The following table and images illustrate which markers are used in the charts I created.

I use a switch case state to scan through the detected markers, and identify the markers of interest.

...
  if (markerIds.size() >= 4 )
  {
    int tl_id = 0;
    int tr_id = 0;
    int bl_id = 0;
    int br_id = 0;
    cv::Point2f tl_xy, tr_xy, bl_xy, br_xy;
    for ( unsigned i = 0; i < markerIds.size(); i++ )
    {
      switch ( markerIds[i] )
      {
      case 923:
        tl_id = markerIds[i];
        tl_xy = markerCorners[i][3]; // bottom left corner of top left marker
        break;
      case 1001:
      case 1002:
      case 1003:
      case 1004:
      case 1005:
      case 1006:
        tr_id = markerIds[i];
        tr_xy = markerCorners[i][2]; // bottom right corner of top right marker
        break;
      case 1007:
        bl_id = markerIds[i];
        bl_xy = markerCorners[i][0]; // top left corner of bottom left marker
        break;
      case 241:
        br_id = markerIds[i];
        br_xy = markerCorners[i][1]; // top right corner of bottom right marker
        break;
      default:
        break;
      }
    }
    ...
  }
  ...

The following figure illustrates which IDs and X/Y coordinates I am preserving to define a region of interest (ROI) for the additional processing.

Detecting a specific chart is accomplished with the following simple conditional statements.

// Chart 1 - Checkerboard (9x7)
if ( (tl_id==923) && (tr_id==1001) && (bl_id==1007) && (br_id==241) )
{
   ...
}
// Chart 2 - White Reference
if ( (tl_id==923) && (tr_id==1002) && (bl_id==1007) && (br_id==241) )
   ...
}
{
// Chart 3 - Histogram
if ( (tl_id==923) && (tr_id==1003) && (bl_id==1007) && (br_id==241) )
{
   ...
}

Adding Computer Generated Graphics to the Scene

For "Chart 1 - CheckerBoard (9x7)", a green rectangle is drawn to identify the region of interest.

// Extract ROI (area, ideally within 4 markers)
      std::vector<cv::Point> polygonPoints;
      polygonPoints.push_back(cv::Point(tl_xy.x,tl_xy.y));
      polygonPoints.push_back(cv::Point(tr_xy.x,tr_xy.y));
      polygonPoints.push_back(cv::Point(br_xy.x,br_xy.y));
      polygonPoints.push_back(cv::Point(bl_xy.x,bl_xy.y));
      // Draw border around "checkerboard"
      cv::polylines(img, polygonPoints, true, cv::Scalar (0, 255, 0), 2, 16);

For "Chart 2 - White Reference", the region of interest is used to calculate the average value for each of the blue (B), green (G), and red (R) components. The following code uses a mask, which supports a ROI that is not a perfect rectangle.

//
      // Calculate color gains
      //   ref : https://stackoverflow.com/questions/32466616/finding-the-average-color-within-a-polygon-bound-in-opencv
      //
      cv::Point pts[1][4];
      pts[0][0] = cv::Point(tl_xy.x,tl_xy.y);
      pts[0][1] = cv::Point(tr_xy.x,tr_xy.y);
      pts[0][2] = cv::Point(br_xy.x,br_xy.y);
      pts[0][3] = cv::Point(bl_xy.x,bl_xy.y);
      const cv::Point* points[1] = {pts[0]};
      int npoints = 4;
      // Create the mask with the polygon
      cv::Mat1b mask(img.rows, img.cols, uchar(0));
      cv::fillPoly(mask, points, &npoints, 1, cv::Scalar(255));
      // Calculate mean in masked area
      auto bgr_mean = cv::mean( img, mask );
      double b_mean = bgr_mean(0);
      double g_mean = bgr_mean(1);
      double r_mean = bgr_mean(2);

With the average values calculated, a plotImage is created which contains a bar plot image with the average values for B, G, and R pixels.

// Draw bars
int plot_w = 100, plot_h = 100;
cv::Mat plotImage( plot_h, plot_w, CV_8UC3, cv::Scalar(255,255,255) );
int b_bar = int((b_mean/256.0)*80.0);
int g_bar = int((g_mean/256.0)*80.0);
int r_bar = int((r_mean/256.0)*80.0);
// layout of bars : |<-10->|<---20-->|<-10->|<---20-->|<-10->|<---20-->|<-10->|
cv::rectangle(plotImage, cv::Rect(10,(80-b_bar),20,b_bar), cv::Scalar(255, 0, 0), cv::FILLED, cv::LINE_8);
cv::rectangle(plotImage, cv::Rect(40,(80-g_bar),20,g_bar), cv::Scalar(0, 255, 0), cv::FILLED, cv::LINE_8);
cv::rectangle(plotImage, cv::Rect(70,(80-r_bar),20,r_bar), cv::Scalar(0, 0, 255), cv::FILLED, cv::LINE_8);
std::stringstream b_str;
std::stringstream g_str;
std::stringstream r_str;
b_str << int(b_mean);
g_str << int(g_mean);
r_str << int(r_mean);
cv::putText(plotImage, b_str.str(), cv::Point(10,90), cv::FONT_HERSHEY_PLAIN, 0.75, cv::Scalar(255,0,0), 1, cv::LINE_AA);
cv::putText(plotImage, g_str.str(), cv::Point(40,90), cv::FONT_HERSHEY_PLAIN, 0.75, cv::Scalar(0,255,0), 1, cv::LINE_AA);
cv::putText(plotImage, r_str.str(), cv::Point(70,90), cv::FONT_HERSHEY_PLAIN, 0.75, cv::Scalar(0,0,255), 1, cv::LINE_AA);

Finally, this plot image warped onto the region of interest, using the following OpenCV functions:

• findHomography : calculates transformation matrix between source and destination coordinates
• warpPerspective : warps the source image to the destination coordinates

The warped bar plot image is then combined with the live image using a mask.

// Calculate transformation matrix
std::vector<cv::Point2f> srcPoints;
std::vector<cv::Point2f> dstPoints;
srcPoints.push_back(cv::Point(       0,       0)); // top left
srcPoints.push_back(cv::Point(plot_w-1,       0)); // top right
srcPoints.push_back(cv::Point(plot_w-1,plot_h-1)); // bottom right
srcPoints.push_back(cv::Point(       0,plot_h-1)); // bottom left
dstPoints.push_back(tl_xy);
dstPoints.push_back(tr_xy);
dstPoints.push_back(br_xy);
dstPoints.push_back(bl_xy);
cv::Mat h = cv::findHomography(srcPoints,dstPoints);
// Warp plot image onto video frame
cv::Mat img_temp = img.clone();
cv::warpPerspective(plotImage, img_temp, h, img_temp.size());
cv::Point pts_dst[4];
for( int i = 0; i < 4; i++)
{
pts_dst[i] = dstPoints[i];
}
cv::fillConvexPoly(img, pts_dst, 4, cv::Scalar(0), cv::LINE_AA);
img = img + img_temp;

For the "Chart 3 - Histogram", a similar technique as "Chart 2" is used. Instead of displaying a bar plot of the color averages, a histogram for each color component is displayed.

With all the theory behind us, it's time for the embedded implementation on Ultra96-V2.

Step 0 - Print the Charts

In order to run this example, you will need the three charts, which have been provided in PDF format in the Schematics section.

• Chart 1 - CheckerBoard (9x7)
• Chart 2 - White Reference
• Chart 2 - Histogram

Step 1 - Create the SD card

Pre-built Vitis-AI 1.3 SD card images have been provided for the following Avnet platforms:

• u96v2_sbc_base : Ultra96-V2 Development Board
• uz7ev_evcc_base : UltraZed-EV SOM (7EV) + FMC Carrier Card
• uz3eg_iocc_base : UltraZed-EG SOM (3EG) + IO Carrier Card

The download links for the pre-built SD card images can be found here:

• Vitis-AI 1.3 Flow for Avnet Vitis Platforms : https://avnet.me/vitis-ai-1.3-project

Once downloaded, and extracted, the.img file can be programmed to a 16GB micro SD card.

0. Extract the archive to obtain the .img file

1. Program the board specific SD card image to a 16GB (or larger) micro SD card

a. On a Windows machine, use Balena Etcher or Win32DiskImager (free opensource software)

b. On a linux machine, use Balena Etcher or use the dd utility

$ sudo dd bs=4M if=Avnet-{platform}-Vitis-AI-1-3-{date}.img of=/dev/sd{X} status=progress conv=fsync

Where {X} is a smaller case letter that specifies the device of your SD card. You can use “df -h” to determine which device corresponds to your SD card.

Step 2 - Clone the source code repository

The source code used in this project can be obtained from the following repositories:

• https://github.com/AlbertaBeef/vitis_ai_gstreamer_plugins

If you have an active internet connection, you can simply clone the repositories to the root directory of your embedded platform:

$ cd ~
$ git clone https://github.com/AlbertaBeef/vitis_ai_gstreamer_plugins

Step 3 - Compile and Install the gstreamer plug-in

The gstreamer plug-in can be built on the Ultra96-V2 embedded platform using the make command:

$ cd vitis_ai_gstreamer_plugins
$ cd markerdetect
$ make

Once compiled, the gstreamer plug-in can be installed as follows:

$ cp libgstmarkerdetect.so /usr/lib/gstreamer-1.0/.

The installation of the gstreamer plug-in can be verified with the gst-inspect-1.0 utility:

$ gst-inspect-1.0 | grep markerdetect
markerdetect:  markerdetect: Marker detection using the OpenCV Library

$ gst-inspect-1.0 markerdetect
Factory Details:
  Rank                     none (0)
  Long-name                Marker detection using the OpenCV Library
  Klass                    Video Filter
  Description              Marker Detection
  Author                   FIXME <fixme@example.com>

Plugin Details:
  Name                     markerdetect
  Description              Marker detection using the OpenCV Library
  Filename                 /usr/lib/gstreamer-1.0/libgstmarkerdetect.so
  Version                  0.0.0
  License                  LGPL
  Source module            markerdetect
  Binary package           OpenCV Library
  Origin URL               http://avnet.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----GstVideoFilter
                               +----GstMarkerDetect

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGR }
                  width: [ 1, 1920 ]
                 height: [ 1, 1080 ]
              framerate: [ 0/1, 2147483647/1 ]

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGR }
                  width: [ 1, 1920 ]
                 height: [ 1, 1080 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:e--
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "markerdetect0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: true

Step 4 - Execute the example with a live video feed

In order to facilitate launching the example, create the following launch script (launch_usb_markerdetect.sh) on your embedded platform:

#!/bin/sh

gst-launch-1.0  \
  v4l2src device=/dev/video0 io-mode=4 ! \
  video/x-raw, width=640, height=480, format=YUY2, framerate=30/1 ! \
  videoconvert ! \
  video/x-raw, format=BGR ! \
  queue ! markerdetect ! queue ! \
  videoconvert ! \
  fpsdisplaysink sync=false text-overlay=false fullscreen-overlay=true \
  \
  -v

Before launching the example, we want to define our DISPLAY environment variable, and configure the resolution of our DisplayPort monitor.

$ export DISPLAY=:0.0
$ xrandr --output DP-1 --mode 800x600

The example can be launch using the script we just created:

$ ./launch_usb_markerdetect.sh

My first execution of this "markerdetect" gstreamer plug-in with a USB camera surprised me. The USB camera I used was a Logitech C720 which has auto white balance, so I was expecting to see the average values of blue, green, and red to be approximately the same.

It turns out that the DisplayPort monitor is generating a blue hue that is being picked up by the chart, and skewing the results slightly.

I ran the same test, this time away from the monitor, and the results aligned better with my expectations.

This time, the blue, green, and red color averages were approximately the same.

Conclusion

I hope this tutorial will inspire you to experiment wtih augmented reality (AR) on the Ultra96-V2.

That other applications can you think of that would use these type of markers ?

If there is any other related content that you would like to see, please share your thoughts in the comments below.

Revision History

2021/05/10 - Initial Version

Acknowledgements

Thank you Tom Simpson for your excellent contributions on gstreamer plug-ins:

• Creating a Vitis-AI GStreamer Plugin for the Ultra96-V2

Thank you Pantone for your innovative and inspiring color matching app:

• Pantone, Color Matching Card, https://www.pantone.com/pantone-color-match-card

Thank you Adrian Rosebrock for your excellent tutorials:

• Adrian Rosebrock, Generating ArUco markers with OpenCV and Python, PyImageSearch,https://www.pyimagesearch.com/2020/12/14/generating-aruco-markers-with-opencv-and-python/ (accessed on 07 May, 2021)
• Adrian Rosebrock, Detecting ArUco markers with OpenCV and Python, PyImageSearch, https://www.pyimagesearch.com/2020/12/21/detecting-aruco-markers-with-opencv-and-python/ (accessed on 07 May, 2021)

Thank you Kevin Keryk for making me realize that my monitor was creating the unexpected blue offset in the initial results.