Structured Light "Slit" Scanner using ESP32-CAM with IoT capabilites.
- Scans an Object with a Line Lazer, by rotating it 360 degrees and capturing an image at all rotating intervals (Discrete Values)
- Posts these Images to a C++ Client Application with a TCP Server listening in
- Preprocessing & 3D Reconstruction of Objects are done on Client Application.
Showcase!
ESP-32 Data Acquisition & TCP Communications to Client C++ Application
As you can see in the **Machine Setup / Physical System**, there is a Rotating Plate on which an Object is placed on, there is also a Line Lazer fixed at 15 Degrees, and the ESP-32 Camera Board aligned with the center of the Rotating Plate. *(However the ESP-32 is higher than the Plate)*
For every Rotation, 3 Data Points need to be captured and sent to the C++ Application in Order, View the List Below.
1.) Image with Line Lazer Projection ( 0_88-6600.png) from monk dataset
2.) Image without Line Lazer Projection ( 1_88-6600.png) from monk dataset
3.) Acknowledgement | Current Rotation of Object (Angle in Degrees, as Float Value)
PostedCharacterBuffer: OK, I acknowledge these 2 images are the projection at 88.66 degrees!
---
The ESP-32 join's my WiFi network upon setup. In the loop function, the images & acknowledgements are sent in a particular sequence via TCP, as I need to know which image is which, and what acknowledgement they correspond to. TCP is especially good for lossless data transfer over the network.
The C++ Application, on the other end, runs a TCP Server which the ESP writes data to upon every loop iteration.
ESP-32 WiFi Connection via `WiFiClient` & `WiFiMulti` libraries.
I've got more details about the networking, but to keep this post concise I'm not including it. Please view my GitHub Repository's Main Readme.md
Decoding & Pre-Processing of Image Datasets
After network communications between the ESP-32 CAM and Client Application have completed, the client-side application has received a directory full of images with associated meta-data information in the title. *(Binary Type for Pair-Ordering & Angle in Degrees)*
In `struct.h`
```c++
struct LazerSlice {
cv::Mat off_img;
cv::Mat on_img;
cv::Mat processed_matrix;
std::vector<glm::vec3> list_3d_points;
float angle;
};
```
The first step is to parse/decode these images into an ordered structure, with decoded **angle** in degrees, off_img set to the `1_...` Image, and on_img set to `0_...` Image. This pair of images is casted to `cv::Mat` and is used in pre-processing, which returns the processed_matrix that is the matrix we'll be reference in the reconstruction algorithms later on.
**NOTE:** Whilst using the second Reconstruction method, you shouldn't use the Perspective Transformation, as the Camera Extrinsics are known and
all interpolation of 2D points are done by the Lazer's Planar Equation.
By the end of Decoding & Preprocessing, we should have a list of structs: `std::vector<LazerSlice> slices = {... }` where each pair represents 1 list entry. *(At this point, `list_3d_points` is still un-assigned)*
Transforming 2D Points into Cylindrical Coordinates for subsequent translation into 3D Cartesian SpaceAfter the Data Acquisition & Pre-processing Phase, We've got a set of `LazerSlice`s, where the `processed_matrix` parameter is essentially a **Binary Mask**, where the activated points is the projected Line Lazer at the specific `angle`.
More Pre-processing happens to the Binary Masks, such as extracting 1 X, Y Activated Point per Y Layer on the image, and an Undistortion Transformation to rectify Camera Distortion, since the ESP slighly barrels I find. *(Transformation done with cv::undistortPoints, with Camera Matrix & Distortion Coefficients found with Python Script)*
Now I've got clean 2D Laser Line Projection on the Object at all captured angles *(Discrete Values, from 0 to 360 deg)*. from here I choose an Origin Point *(On the X-axis)* to define the center for all images. Defined as X-Midpoint
The algorithm takes a 2D image and conceptualizes it as a cross-sectional slice at a specific laser angle, denoted as Θ. This slice is represented in a cylindrical coordinate system, centered around an origin defined by the midpoint of the X-axis in the image. The X and Z coordinates of this slice are then transformed into Cartesian coordinates by dividing them by tan(Θ). Subsequently, these Cartesian coordinates are normalized to fit within the OpenGL coordinate range.
Finally, the list of 3D points extrapolated from the slice undergoes a rotational transformation (Vector (x, y, z) multiplied by (3 x 3) Rotation Matrix with the slice's angle as theta `LazerSlice.angle`.
**Observation:** Going from Cylindrical to Cartesian, then doing a transformation for rotation of 3D points preserves the relative spatial relationships within the 3D points captured on each slice!
---
More Software Features for the Project:Command Line Interface where you're able to make certain configurations of 3D Renders
OpenGL 3D Renderer & Free Cam Feature to visualize Point Clouds generated by the Configurations & Dataset
Extension of this project was written in Python to gather ESP32 Camera Instrinsics, Extrinsics, Distortion Coefficients, etc..
View the GitHub Repo
More on the Hardware:This is the Circuit Diagram that connect all the modules. The VCC pins (-/+) onboard the ESP32 Step-down the 5V DC input into 3.3V (ESP32's operating voltage)
The 24V, 2A DC source is inputted directly into the A4988 Stepper Driver Module which channels it to Stepper Motor. The A4988 also runs on 3.3V logic, which is supplied by the ESP-32's 3.3V pinout. this 3.3V pinout is also connected to MS1
, MS2
& MS3
onboard, which allows to set the step size of the Stepper Motor which digital HI & LOW signals. when all 3 pins are HI, it steps at 1/16 step which is optimal for my use case of having small increments in angle for every data point I collect via ESP32 CAM sensor. (View Truth Table Below)
The Line Lazer Module is very straight forward, connected it to a Digital Pin, and toggle it as needed.
Conclusion / Message From Author:I made this project between my 2nd & 3rd year of University (Bachelors in Engineering) And had lots of fun with it !
I'd like to acknowledge Victor Gordan for his great OpenGL Tutorial! Really helped me out making the OpenGL Model viewer
Ressources & Inspiration for this project:Main Inspiration, and great resource for this project!
hackster.io ; 3D Scanning with Raspberry Pi and MATLAB
Very Useful Ressources:
ece.cornell.edu, ECE 5725: Embedded OS Project ; Laser 3D Scanner
Brown University ; The Laser Slit 3D Scanner
Interesting Resource (Although not as useful for this project):
Comments