Camera modules for Self-Driving Car.

Before to start.

First.

I start this project from get start. Update OpenSDA software. (StartFragment

J-Link OpenSDA - Board-Specific Firmwares EndFragment)

Second.

Read and to do instruction. I compile and load code to board. Trought skype how write to instruction - I don't get video, but I use VLC player and get video from board.

Recognize

For recognize object I find two libraries - famous OpenCV and dlib. I know Java language and I have been working with him long. But I like microcontrollers kind of AVR and PIC and i want better know ARM controller.

Than I start expirement with OpenCV library for Java. Recognite face:

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Point;
import org.opencv.core.Rect;
import org.opencv.core.Scalar;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.objdetect.CascadeClassifier;
//
// Detects faces in an image, draws boxes around them, and writes the results
// to "faceDetection.png".
//
class DetectFaceDemo {
public void run() {
 System.out.println("\nRunning DetectFaceDemo");
 // Create a face detector from the cascade file in the resources
 // directory.
 CascadeClassifier faceDetector = new CascadeClassifier(getClass().getResource("/lbpcascade_frontalface.xml").getPath());
 Mat image = Imgcodecs.imread(getClass().getResource("/lena.png").getPath());
 // Detect faces in the image.
 // MatOfRect is a special container class for Rect.
 MatOfRect faceDetections = new MatOfRect();
 faceDetector.detectMultiScale(image, faceDetections);
 System.out.println(String.format("Detected %s faces", faceDetections.toArray().length));
 // Draw a bounding box around each face.
 for (Rect rect : faceDetections.toArray()) {
     Imgproc.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(0, 255, 0));
 }
 // Save the visualized detection.
 String filename = "faceDetection.png";
 System.out.println(String.format("Writing %s", filename));
 Imgcodecs.imwrite(filename, image);
}
}
public class HelloOpenCV {
public static void main(String[] args) {
 System.out.println("Hello, OpenCV");
 // Load the native library.
 System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
 new DetectFaceDemo().run();
}
}

Result:

Next experiment :

package recognite;

import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import javax.imageio.ImageIO;
import javax.swing.ImageIcon;
import javax.swing.JFrame;
import javax.swing.JLabel;
import org.opencv.core.*;
import org.opencv.highgui.Highgui;        
import org.opencv.highgui.VideoCapture;        
import org.opencv.imgproc.Imgproc;
public class VideoCap {
 public static void main (String args[]){
 	System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
 	VideoCapture camera = new VideoCapture(1);
 	if(!camera.isOpened()){
 		System.out.println("Error");
 	}
 	else {
 		Mat frame = new Mat();
  	    while(true){
 	    	if (camera.read(frame)){
 	       		camera.set(10, 99);
 		    	camera.set(11, 70);
//    		    	camera.set(15, -8.0);
 	    		System.out.println("Frame Obtained");
 	    		System.out.println("Captured Frame Width " + 
 	    		frame.width() + " Height " + frame.height());
 	    		//Highgui.imwrite("camera.jpg", frame);
 	    		showResult(frame);
 	    		System.out.println("OK");
 	    		break;
 	    	}
 	    }	
 	}
 	camera.release();
 }
public static void showResult(Mat img) {
 Imgproc.resize(img, img, new Size(640, 480));
 MatOfByte matOfByte = new MatOfByte();
 Highgui.imencode(".jpg", img, matOfByte);
 byte[] byteArray = matOfByte.toArray();
 BufferedImage bufImage = null;
 try {
     InputStream in = new ByteArrayInputStream(byteArray);
     bufImage = ImageIO.read(in);
     JFrame frame = new JFrame();
     frame.getContentPane().add(new JLabel(new ImageIcon(bufImage)));
     frame.pack();
     frame.setVisible(true);
 } catch (Exception e) {
     e.printStackTrace();
 }
}
}   

Capture image with FRDM-K82 board my face:

But I don't understand why image has not very quality. Mayby my camera sensor not quality.

Next time I search solve recognite sign and i find it in dlib library. I attached url github in Code part.

For compilation dlib need install CMake util. I did it on Win 7. I needed install MinGW enviropment for assemble project.

In guide need make sequences command:

cd examples
mkdir build
cd build
cmake ..
cmake --build . --config Release

For enviropment MinGW I added how need generator use.

cmake -G "MinGW Makefiles"

After that i begin think how this code will ported to FRDM platform. Inside code i see what it content strange compressed binary code what difficult for mind. Second problem i find when try create project with him. He has a lot dependency one with other. I did come back to OpenCV code.

I use CMake tools compile OpenCV library and ported java code for recognize face to C++. On site OpenCV library for C++ use different example.

#include "opencv2/objdetect/objdetect.hpp"

 
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include <iostream>
#include <stdio.h>
using namespace std;
using namespace cv;
// Function Headers
void detectAndDisplay(Mat frame);
// Global variables
// Copy this file from opencv/data/haarscascades to target folder
string face_cascade_name = "c:/progwork/opencv/haarcascade_frontalface_alt.xml";
CascadeClassifier face_cascade;
string window_name = "Capture - Face detection";
int filenumber; // Number of file to be saved
string filename;
// Function main
int main(void)
{
  // Load the cascade
  if (!face_cascade.load(face_cascade_name)){
      printf("--(!)Error loading\n");
      return (-1);
  }
  // Read the image file
  Mat frame = imread("c:/progwork/opencv/lena.png");
  // Apply the classifier to the frame
  if (!frame.empty()){
      detectAndDisplay(frame);
  }
  else{
      printf(" --(!) No captured frame -- Break!");
      return 0;
  }
  int c = waitKey(10000);
  if (27 == char(c)){
    return 0;  
  }
}
// Function detectAndDisplay
void detectAndDisplay(Mat frame)
{
  std::vector<Rect> faces;
  Mat frame_gray;
  Mat crop;
  Mat res;
  Mat gray;
  string text;
  stringstream sstm;
  cvtColor(frame, frame_gray, COLOR_BGR2GRAY);
  equalizeHist(frame_gray, frame_gray);
  // Detect faces
  face_cascade.detectMultiScale(frame_gray, faces, 1.1, 2, 0 | CASCADE_SCALE_IMAGE, Size(30, 30));
  // Set Region of Interest
  cv::Rect roi_b;
  cv::Rect roi_c;
  size_t ic = 0; // ic is index of current element
  int ac = 0; // ac is area of current element
  size_t ib = 0; // ib is index of biggest element
  int ab = 0; // ab is area of biggest element
  for (ic = 0; ic < faces.size(); ic++) // Iterate through all current elements (detected faces)
  {
      roi_c.x = faces[ic].x;
      roi_c.y = faces[ic].y;
      roi_c.width = (faces[ic].width);
      roi_c.height = (faces[ic].height);
      ac = roi_c.width * roi_c.height; // Get the area of current element (detected face)
      roi_b.x = faces[ib].x;
      roi_b.y = faces[ib].y;
      roi_b.width = (faces[ib].width);
      roi_b.height = (faces[ib].height);
      ab = roi_b.width * roi_b.height; // Get the area of biggest element, at beginning it is same as "current" element
      if (ac > ab)
      {
          ib = ic;
          roi_b.x = faces[ib].x;
          roi_b.y = faces[ib].y;
          roi_b.width = (faces[ib].width);
          roi_b.height = (faces[ib].height);
      }
      crop = frame(roi_b);
      resize(crop, res, Size(128, 128), 0, 0, INTER_LINEAR); // This will be needed later while saving images
      cvtColor(crop, gray, CV_BGR2GRAY); // Convert cropped image to Grayscale
      // Form a filename
      filename = "";
      stringstream ssfn;
      ssfn << filenumber << ".png";
      filename = ssfn.str();
      filenumber++;
      imwrite(filename, gray);
      Point pt1(faces[ic].x, faces[ic].y); // Display detected faces on main window - live stream from camera
      Point pt2((faces[ic].x + faces[ic].height), (faces[ic].y + faces[ic].width));
      rectangle(frame, pt1, pt2, Scalar(0, 255, 0), 2, 8, 0);
  }
  // Show image
  sstm << "Crop area size: " << roi_b.width << "x" << roi_b.height << " Filename: " << filename;
  text = sstm.str();
  putText(frame, text, cvPoint(30, 30), FONT_HERSHEY_COMPLEX_SMALL, 0.8, cvScalar(0, 0, 255), 1, CV_AA);
  imshow("original", frame);
  if (!crop.empty())
  {
      imshow("detected", crop);
  }
  else
      destroyWindow("detected");
}
 

From this code I understand what for task recognize traffic sign, me needed two module: imgproc and objdetect. I create in KDS project and add all files Objdetect module. And I try compile it, but I have next error message:

source/subdir.mk:70: recipe for target 'source/linemod.o' failed

make: *** [source/linemod.o] Error 1
make: *** Waiting for unfinished jobs....
C:\Users\5A03~1\AppData\Local\Temp\ccfBCkw3.s: Assembler messages:
C:\Users\5A03~1\AppData\Local\Temp\ccfBCkw3.s:47: Error: selected FPU does not support instruction -- `vcvtr.s32.f64 s15,d7'
source/subdir.mk:70: recipe for target 'source/haar.o' failed
make: *** [source/haar.o] Error 1
C:\Users\5A03~1\AppData\Local\Temp\ccF20JV2.s: Assembler messages:
C:\Users\5A03~1\AppData\Local\Temp\ccF20JV2.s:47: Error: selected FPU does not support instruction -- `vcvtr.s32.f64 s15,d7'
source/subdir.mk:70: recipe for target 'source/hog.o' failed
make: *** [source/hog.o] Error 1
C:\Users\5A03~1\AppData\Local\Temp\ccDxWOE2.s: Assembler messages:
C:\Users\5A03~1\AppData\Local\Temp\ccDxWOE2.s:47: Error: selected FPU does not support instruction -- `vcvtr.s32.f64 s15,d7'
source/subdir.mk:70: recipe for target 'source/cascadedetect.o' failed
make: *** [source/cascadedetect.o] Error 1
04:21:51 Build Finished (took 4s.425ms)
 

Maybe my skills in C++ programing not big. I don't resolve problem.

To be continued ...