Tutorial : TinyML with OV7670 Camera module!

This is my second project with Google Summer of Code (GSoC) under TensorFlow. There was no proper documentation on the internet to build a custom image recognition TinyML model, so my GSoC mentor, Paul Ruiz, suggested that I should try and solve it. Here's how you could also build an image recognition TinyML application. Happy Tinkering!

Click here to view my first GSoC project!

The idea behind the project:

I wanted to work on a problem with fewer variables as the documentation for how to work with the camera module and process its data wasn't great. I choose to build an MNIST TinyML model as, in this case, I wouldn't need to worry about the training data set, and it would allow me to focus on the essential parts of the project to get things up and running. But, now that I have figured out all the parts to build a custom image recognition project, I have documented how to collect training data sets using the camera module.

The theme/tone for the blog?

I want to warn you that this blog might get a bit complex to understand. There's a proper explanation for this: With an accelerometer-based application, it would be easy to do sanity checks by just printing out the accelerometer values of one axis on the serial monitor or plotter. In contrast, doing sanity checks for the image recognition application is at least 10x more tiresome because checking if a piece of code is doing the desired action cannot be visualized in real-time.

Some Comments

This blog might be a bit hard to understand due to the complexity of unit testing. I want to address any gaps in explanation with feedback from the readers. So, comment below with your doubts and questions regarding anything related to image recognition on embedded systems.

Does TinyML make sense at all?

I would recommend you to read through this fantastic article by Pete Warden, The author of the TinyML book, to understand why running machine learning models on microcontrollers makes sense and is the future of machine learning.

Even if TinyML makes sense, does image recognition make sense on TinyML?

The full VGA (640×480 resolution) output from the OV7670 camera we'll be using here is too big for current TinyML applications. uTensor runs handwriting detection with MNIST that uses 28×28 images. The person detection example in the TensorFlow Lite for Microcontrollers example uses 96×96 which is more than enough. Even state-of-the-art 'Big ML' applications often only use 320×320 images. In conclusion, running image recognition applications on tiny microcontrollers makes a lot of sense

This tutorial in a nutshell:

Wiring

Introductiontothe OV7670 camera module

RGB888 vs RGB565

Reintroductiontothe OV7670 camera module

Which display to choose?

Getting started with the OLED module

Getting started with the TFT LCD module

Integrating the camera and TFT LCD module

Building the MNIST TinyML model

Testing the TinyML model

Integration time

Problems with the project/ How to improve the project

Some helpful pointers to building your own image recognition project

Collecting training data using the OV7670 camera module

Conclusion

1. Wiring

1.a Arduino Nano 33 BLE Sense pinouts

1.b Schematics

1.cArduino Nano 33 BLE Sense - OV7670 Camera module

PinsonOV7670Camera Module - Pins onArduino Nano 33 BLE Sense

• 3.3 to 3.3V
• GND to GND
• SIOC to A5
• SIOD to A4
• VSYNC to 8
• HREF to A1
• PCLK to A0
• XCLK to 9
• D7 to 4
• D6 to 6
• D5 to 5
• D4 to 3
• D3 to 2
• D2 to 0 / RX
• D1 to 1 / TX
• D0 to 10

1.dArduino Nano 33 BLE Sense - TFTLCD module

Pinson1.44" TFT LCD display - Pins onArduino Nano 33 BLE Sense

Note: there's only one 3.3V on the Arduino board. Use a breadboard to have multiple connections to it.

• LED to 3.3V
• SCK to 13
• SDA to 11
• A0 to A6
• RESET to 7
• CS to A7
• GND to GND
• VCC to 5V

Note: The TFT LCD module connected to the Arduino board uses the hardware SPI pins.

SPI stands for Serial Peripheral Interface. It is used by the microcontrollers to communicate with one or more peripheral devices quickly. SPI communication is faster than I2C communication.

There are three common pins to all the peripheral devices:

• SCK-It stands for Serial Clock. This pin generates clock pulses, that is used to synchronize the transfer of data.
• MISO-It stands for Master Input/ Slave Output. This data line in the MISO pin is used to send the data to the master.
• MOSI-It stands for Master Output/ Slave Input. This line is used for sending data to the slaves/peripheral devices.

SPI pins on the board:

• D13-SCK
• D12-MISO
• D11-MOSI

We will only use the SCK and MOSI pins here as we'll be sending data to the TFT and won't need the MISO pin for the same.

2. Introduction to the OV7670 camera module

2.a General information about the OV7670 module

The OV7670 camera module is a low-cost 0.3-megapixel CMOS color camera module. It can output a 640x480 VGA resolution image at 30fps.

Features:

• High sensitivity for low-light operation
• Low operating voltage for embedded portable apps
• Lens shading correction
• Flicker (50/60 Hz) auto-detection
• De-noise level auto adjust
• Supports image sizes: VGA, CIF, and any size scaling down from CIF to 40x30
• VarioPixel method for sub-sampling
• Automatic image control functions include: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance(AWB), automatic band filter (ABF), and automatic black-level calibration (ABLC)
• ISP includes noise reduction and defect correction
• Supports LED and flash strobe mode
• Supports scaling
• Output support for Raw RGB, RGB (GRB 4:2:2, RGB565/555/444), YUV (4:2:2) and YCbCr (4:2:2) formats
• Image quality controls include color saturation, hue, gamma, sharpness (edge enhancement), and anti-blooming
• Saturation level auto adjust (UV adjust)
• Edge enhancement level auto adjust

Specifications:

• Photosensitive Array: 640 x 480.
• IO Voltage: 2.5V to 3.0V.
• Operating Power: 60mW/15fpsVGAYUV.
• Sleeping Mode: <20μA.
• Operating Temperature: -30 to 70 deg C.
• Output Format: YUV/YCbCr4:2:2 RGB565/555/444 GRB4:2:2 Raw RGB Data (8 digit).
• Lens Size: 1/6″.
• Vision Angle: 25 degrees.
• Max. Frame Rate: 30fps VGA.
• Sensitivity: 1.3V / (Lux-sec).
• Signal to Noise Ratio: 46 dB.
• Dynamic range: 52 dB.
• Browse Mode: By row.
• Electronic Exposure: 1 to 510 rows.
• Pixel Coverage: 3.6μm x 3.6μm.
• Duck Current: 12 mV/s at 60℃.
• PCB Size (L x W): Approx. 1.4 x 1.4 inch / 3.5 x 3.5 cm.

2.b Software Setup: Installing the "Arduino_OV767x" library

First you will need the Arduino IDE installed. Next, under the Tools section, click the Manage Libraries, search for OV7670, select the Arduino_OV767x library and click Install.

Supported image configurations in the OV767X library:

• VGA – 640 x 480
• CIF – 352 x 240
• QVGA – 320 x 240
• QCIF – 176 x 144

2.c Software Setup: Installing Processing

Processing is a simple programming environment that was created by graduate students at MIT Media Lab to make it easier to develop visually oriented applications with an emphasis on animation and providing users with instant feedback through interaction.

Download and install Processing using this link.

Why do I need to download this software? We'll use this application to visualize the camera output sent via the serial port by the OV7670 camera module.

2.d Using Processing: Test pattern

Github Link for this subsection.

Open an Arduino sketch, and copy and paste the below sketch into the sketch, upload it to your board.

Processing_test_pattern.ino:

/*
  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10
*/

#include <Arduino_OV767X.h>

int bytesPerFrame;

byte data[320 * 240 * 2]; // QVGA: 320x240 X 2 bytes per pixel (RGB565)

void setup() {
  Serial.begin(115200);
  while (!Serial);

  if (!Camera.begin(QVGA, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  bytesPerFrame = Camera.width() * Camera.height() * Camera.bytesPerPixel();

  Camera.testPattern();
}

void loop() {
  Camera.readFrame(data);

  Serial.write(data, bytesPerFrame);
}

Once you are done uploading the above sketch to your Arduino board, open the Processing application and copy-paste the below code into a new file.

processingSketch:

import processing.serial.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

Serial myPort;

// must match resolution used in the sketch
final int cameraWidth = 320;
final int cameraHeight = 240;

final int cameraBytesPerPixel = 2;

final int bytesPerFrame = cameraWidth * cameraHeight * cameraBytesPerPixel;

PImage myImage;

void setup()
{
  size(320, 240);

  // if you have only ONE serial port active
  //myPort = new Serial(this, Serial.list()[0], 9600); // if you have only ONE serial port active

  // if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

  // wait for full frame of bytes
  myPort.buffer(bytesPerFrame);  

  myImage = createImage(cameraWidth, cameraHeight, RGB);
}

void draw()
{
  image(myImage, 0, 0);
}

void serialEvent(Serial myPort) {
  byte[] frameBuffer = new byte[bytesPerFrame];

  // read the saw bytes in
  myPort.readBytes(frameBuffer);

  // create image to set byte values
  PImage img = createImage(cameraWidth, cameraHeight, RGB);

  // access raw bytes via byte buffer
  ByteBuffer bb = ByteBuffer.wrap(frameBuffer);
  bb.order(ByteOrder.BIG_ENDIAN);

  int i = 0;

  img.loadPixels();
  while (bb.hasRemaining()) {
    // read 16-bit pixel
    short p = bb.getShort();

    // convert RGB565 to RGB 24-bit
    int r = ((p >> 11) & 0x1f) << 3;
    int g = ((p >> 5) & 0x3f) << 2;
    int b = ((p >> 0) & 0x1f) << 3;

    // set pixel color
    img.pixels[i++] = color(r, g, b);
  }
  img.updatePixels();

  // assign image for next draw
  myImage = img;
}

Now, uncomment the line, specific to your operating system, in the above. and Click on the Run button.

// if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  //myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

You should get an output like the one below:

2.e Explanation: Test Pattern

Processing_test_pattern.ino:

byte data[320 * 240 * 2]; // QVGA: 320x240 X 2 bytes per pixel (RGB565)

This line of code sets up an array of type byte. We'll be using the RGB565 color formal, so we'll need 2 bytes for every pixel, and the image format we'll be using here is QVGA, which is 320x240 pixels in size. Therefore, the size of the array will be the height * width * bytes required for each pixel's color. In practice, it translates to 320 * 240 * 2.

Serial.begin(115200);
  while (!Serial);

This line of code sets up the serial port to transmit data between the computer and the microcontroller.

if (!Camera.begin(QVGA, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

The above line of code sets up the OV7670 camera module. In this example, we have initialized it to use the QVGA image format and the RGB565 color format.

Camera.testPattern();

This line of code sets the camera to send a test image via the serial port.

Camera.readFrame(data);

This line of code reads one frame from the camera and stores it in the array we declared before.

Serial.write(data, bytesPerFrame);

Finally, this line of code writes the array's values onto the serial monitor.

processingSketch:

// must match resolution used in the sketch
final int cameraWidth = 320;
final int cameraHeight = 240;

These lines of code set up the cameraWidth and cameraHeight to match the size in the Arduino sketch.

// if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  //myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

These lines of code specify the serial port through which data is transmitted between the microcontroller and the computer.

// convert RGB565 to RGB 24-bit
    int r = ((p >> 11) & 0x1f) << 3;
    int g = ((p >> 5) & 0x3f) << 2;
    int b = ((p >> 0) & 0x1f) << 3;

These lines of code convert the RGB565 color format to RGB888 format to display on your computer screen. This will be explained in detail in further sections.

2.f Using Processing: Live image

Github Link for this subsection.

Open an Arduino sketch, and copy and paste the below sketch into the sketch, upload it to your board.

Processing_ov7670_live_image.ino

/*
  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10
*/

#include <Arduino_OV767X.h>

int bytesPerFrame;

byte data[320 * 240 * 2]; // QVGA: 320x240 X 2 bytes per pixel (RGB565)

void setup() {
  Serial.begin(115200);
  while (!Serial);

  if (!Camera.begin(QVGA, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  bytesPerFrame = Camera.width() * Camera.height() * Camera.bytesPerPixel();

  Camera.testPattern();
}

void loop() {
  Camera.readFrame(data);

  Serial.write(data, bytesPerFrame);
}

Once you are done uploading the above sketch to your Arduino board, open the Processing application and copy-paste the below code into a new file.

processingSketch:

import processing.serial.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

Serial myPort;

// must match resolution used in the sketch
final int cameraWidth = 320;
final int cameraHeight = 240;

final int cameraBytesPerPixel = 2;

final int bytesPerFrame = cameraWidth * cameraHeight * cameraBytesPerPixel;

PImage myImage;

void setup()
{
  size(320, 240);

  // if you have only ONE serial port active
  //myPort = new Serial(this, Serial.list()[0], 9600); // if you have only ONE serial port active

  // if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

  // wait for full frame of bytes
  myPort.buffer(bytesPerFrame);  

  myImage = createImage(cameraWidth, cameraHeight, RGB);
}

void draw()
{
  image(myImage, 0, 0);
}

void serialEvent(Serial myPort) {
  byte[] frameBuffer = new byte[bytesPerFrame];

  // read the saw bytes in
  myPort.readBytes(frameBuffer);

  // create image to set byte values
  PImage img = createImage(cameraWidth, cameraHeight, RGB);

  // access raw bytes via byte buffer
  ByteBuffer bb = ByteBuffer.wrap(frameBuffer);
  bb.order(ByteOrder.BIG_ENDIAN);

  int i = 0;

  img.loadPixels();
  while (bb.hasRemaining()) {
    // read 16-bit pixel
    short p = bb.getShort();

    // convert RGB565 to RGB 24-bit
    int r = ((p >> 11) & 0x1f) << 3;
    int g = ((p >> 5) & 0x3f) << 2;
    int b = ((p >> 0) & 0x1f) << 3;

    // set pixel color
    img.pixels[i++] = color(r, g, b);
  }
  img.updatePixels();

  // assign image for next draw
  myImage = img;
}

Now, uncomment the line, specific to your operating system, in the above. and Click on the Run button.

// if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  //myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

You should get an output like the one below:

2.g Explanation: Live image

Processing_ov7670_live_image.ino:

byte data[320 * 240 * 2]; // QVGA: 320x240 X 2 bytes per pixel (RGB565)

This line of code sets up an array of type byte. We'll be using the RGB565 color format so we'll need 2 bytes for every pixel and the image format that we'll be using here is QVGA, which is 320x240 pixels in size. Therefore, the size of the array will be the height * width * bytes required for each pixel's color. In practice, it translates to 320 * 240 * 2.

Serial.begin(115200);
  while (!Serial);

This line of code sets up the serial port to transmit data between the computer and the microcontroller.

if (!Camera.begin(QVGA, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

The above line of code sets up the OV7670 camera module. In this example, we have initialized it to use the QVGA image format and the RGB565 color format.

Camera.testPattern();

This line of code sets the camera to send a test image via the serial port.

Camera.readFrame(data);

This line of code reads one frame from the camera and stores it in the array we declared before.

Serial.write(data, bytesPerFrame);

Finally, this line of code writes the array on the serial monitor.

processingSketch:

// must match resolution used in the sketch
final int cameraWidth = 320;
final int cameraHeight = 240;

These lines of code set up the cameraWidth and cameraHeight to match the size in the Arduino sketch.

// if you know the serial port name
  //myPort = new Serial(this, "COM5", 9600);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", 9600);             // Linux
  //myPort = new Serial(this, "/dev/cu.usbmodem14101", 9600);  // Mac

These lines of code specify the serial port through which data is transmitted between the microcontroller and the computer.

// convert RGB565 to RGB 24-bit
    int r = ((p >> 11) & 0x1f) << 3;
    int g = ((p >> 5) & 0x3f) << 2;
    int b = ((p >> 0) & 0x1f) << 3;

These lines of code convert the RGB565 color format to RGB888 format to display it on your computer screen. This will be explained in detail in further sections.

2.h Problems with this approach,and possible solutions

The processing application displays a zigzag test pattern instead of the actual test pattern and a broken/washed-out image instead of the correct live image. This has been discussed in Github discussions and Arduino forums. I have attached the links to the same below.

Link to the Github discussion

Link to the Arduino forum

Some of the suggested solutions:

1. Use shorter wires

• My take on it: I changed from 20cm wires to 10cm, but it didn't make a difference.

2. Try Ubuntu Linux

3. Change the FPS

• My take on it: I changed it to be 1/5/30 FPS but there was no improvement.

4. Change the Serial Rate

• My take on it: I changed my Serial Rate to 115200 bps from 9600 bps. but there was still no improvement in the problem

A plausible reason for the problem:

• Most people on the forums agree that it's the windows processing speed that causes the issue and that switching to Ubuntu should solve it.

3. RGB888 vs RGB565

3.a General information about RGB888

The RGB888 color model uses 8 bits to represent each color. The transparency(alpha) value is assumed to be the maximum(255).

The maximum value possible for red, blue, and green colors is 255.

Some examples:

• White: (R, G, B) = (255, 255, 255)
• Black: (R, G, B) = (0, 0, 0)

3.b General information about RGB565

RGB565 is used to represent colors in 16 bits, rather than the 24bit to specify colors. To make full use of the 16 bits, red and blue are encoded in 5 bits and green in 6 bits. This is due to the fact that human eyes have a better capacity to see more shades of green.

The maximum possible values for red and blue values in the RGB565 color format are 31 while the maximum value for the green color is 63.

Fun fact: RGB565 only has 0.39% (65k vs 16m) of the colors of RGB888

3.c Converting RGB888 values to RGB565

/*
Assumption:
r = 8 bits
g = 8 bits
b = 8 bits
*/
rgb565 = ((r & 0b11111000) << 8) | ((g & 0b11111100) << 3) | (b >> 3);

We shift:

• r left by 11 bits, and throw away the last 3 bits
• g left by 5 bits, and throw away the last 2 bits
• b shifted right by 3 bits to throw away the last 3 bits

We finally bitwise OR these 3 to join them into a single 16-bit representation.

Example:

let's convert the white color from RGB888 to RGB565 color space.

Since we already know the maximum possible values possible for both the color spaces, we should expect

• (255, 255, 255) in the RGB888 color space
• (31, 63, 31) in the RGB565 color space

In this problem,

• r = 255 in decimal or 0000000011111111 in binary
• g = 255 in decimal or 0000000011111111 in binary
• b = 255 in decimal or 0000000011111111 in binary

For red:

• r = 0000000011111111
• (r & 0b11111000) = 0000000011111000
• (r & 0b11111000) << 8) = 1111100000000000

For green:

• g = 0000000011111111
• (g & 0b11111100) = 0000000011111100
• ((g & 0b11111100) << 3) = 0000011111100000

For blue:

• b = 0000000011111111
• (b >> 3) = 0000000000011111

Combining these three equations:

• rgb565 = ((r & 0b11111000) << 8) | ((g & 0b11111100) << 3) | (b >> 3);
• rgb565 = ( 1111100000000000 | 0000011111100000 | 0000000000011111)
• rgb565 = 1111111111111111

In RGB565 color space,

• The first 5 bits correspond to the red value
• The next 6 bits correspond to the green value
• the last 5 bits correspond to the blue value
• This translates to (31, 63, 31), which is the expected output!

3.d Converting RGB565 values to RGB888

int r = ((p >> 11) & 0b00011111) << 3;
int g = ((p >> 5) & 0b00111111) << 2;
int b = ((p >> 0) & 0b00011111) << 3;

• For red, we left shift by 11 bits, bitwise AND with 0b00011111, and right shift by 3 bits
• For green, we left shift by 5 bits, bitwise AND with 0b00111111, and right shift by 2 bits
• For blue, we left shift by 0 bits, bitwise AND with 0b00011111, and right shift by 3 bits

Example:

let's convert the white color from RGB565 to RGB888 color space.

Since we already know the maximum possible values possible for both the color spaces, we should expect

• (31, 63, 31) in the RGB565 color space
• (248, 252, 248) in the RGB888 color space

Shouldn't we expect (255, 255, 255) in the RGB888 color space?

RGB565 only has 0.39% (65k vs 16m) of the colors of RGB888. Therefore it's unable to cover the entire spectrum of RGB888.

In this problem,

In RGB format, white = 1111111111111111

For red:

• p(here: white) = 1111111111111111
• (p >> 11) = 00011111
• ((p >> 11) & 0b00011111) = 00011111
• (((p >> 11) & 0b00011111) << 3) = 11111000

For green:

• p(here: white) = 1111111111111111
• (p >> 5) = 0000011111111111
• ((p >> 5) & 0b00111111) = 00111111
• (((p >> 5) & 0b00111111) << 2) = 11111100

For blue:

• p(here: white) = 1111111111111111
• (p >> 0) = 1111111111111111
• ((p >> 0) & 0b00011111) = 00011111
• ((p >> 0) & 0b00011111) << 3 = 11111000

Combining these three outputs:

• Final red value = 0b11111000 = 248
• Final green value = 0b11111100 = 252
• Final blue value = 0b11111000 = 248
• This is the expected output!

Link to a RGB565 color picker

RGB88 to RGB565 converter

4. Reintroduction to the OV7670 camera module

4.a Thegame plan /Approach

Now that we have no way of knowing what the camera is looking at or if it's possible to process the camera data to get a meaningful image, how do we move forward?

The approach I used to solve this is to capture a single image, store the pixel values in an array, use code to process(like applying grayscale filters) data from the array, and check if the piece of code is doing the "right" thing by cross verifying it using a visualizer on python.

4.a Gettinghex values via the serial port and visualizing it

Github Link for this subsection.

Copy and paste the below code into a new Arduino sketch, upload it to your board, open the serial monitor, and press c. Copy and paste the hex values into a text file.

Code explanation:

This sketch reads one frame from the camera and prints out the RGB565 values of all the pixels onto the serial monitor.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

unsigned short pixels[176 * 144]; // QCIF: 176x144 X 2 bytes per pixel (RGB565)

void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < numPixels; i++) {
        unsigned short p = pixels[i];
        if (p < 0x1000) {
          Serial.print('0');
        }
        if (p < 0x0100) {
         Serial.print('0');
        }
        if (p < 0x0010) {
          Serial.print('0');
        }
        Serial.print(p, HEX);
      }
  }
}

Link to the text file with HEX values

4.c ConvertingtheHEXvalues to RGB888 values

Github Link for this subsection.

Copy and paste the below code into a new sketch and upload it to your board.

Code explanation:

This sketch reads one frame from the camera and prints out the RGB888 values of all the pixels onto the serial monitor. The equation used here is extensively described in the earlier sections.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

uint16_t pixels[176 * 144]; // QCIF: 176x144 X 2 bytes per pixel (RGB565)

void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < 1; i++) {
      for(int j =0; j < 1; j++){
       uint16_t pixel = 512;
       Serial.println(pixel);
       Serial.println(pixel,HEX);
       //RGB888 data
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2;
       int blue  = ((pixel >> 0) & 0x1f) << 3;
       Serial.println(red);
       Serial.println(green);
       Serial.println(blue); 
      }
    }
  }
}

4.d Test:Serial HEX to RGB888

Github Link for this subsection.

One way you can manually check if the equation you've used is working well is to open up the rawpixels.net website and hover your mouse over the image. This would show the RGB values of the pixel of the image. Using the data from this website, you can cross-verify the values that show up on the serial monitor.

4.e Building a custom RG565 visualizer

Github Link for this subsection.

Why did I build a custom RGB565 visualizer?

I was too lazy to save the RGB565 values in a hex file, upload it to the rawpixels website and change the parameters every time to view a single image.

Code explanation:

string = "0D2AED29ED....B6B5B6B5"

This line of code stores all the RGB565 values as a string.

(len(string) * 4)

>>405504

Since each hex is equivalent to four binary digits, multiplying the length of the string by 4 gives us the total no of bits to process.

(len(string) * 4) / 16

>> 25344.0

Since each pixel requires 16 bits to represent its color, dividing the total no of bits by 16 gives us the no of pixels to process.

The output here comes out to be 25344, which matches the expected number of pixels.

n = 4
string2 = [string[i:i+n] for i in range(0, len(string), n)]
print("{", end='')
for i in string2:
print('0x'+i+',', end='')
print("}", end='')

>>{0x0D2A,0xED29,0xED29...0x95B5,0xB6B5,0xB6B5}

4 hexadecimal values grouped together forms 16 bits and represent one pixel's RGB565 value. The above code uses list comprehension to group 4 hex values together to put them in a new list named "string2".

scale = 16
r = []
g = []
b = []

These lines of code declare three empty lists to store the red, green, and blue values, respectively, and initialize the variable scale to 16.

Approach 1:

for i in range(len(string2)):
    t = string2[i]
    t_new = bin(int('1'+t, scale))[3:]
    
    r_new = t_new[:5]
    g_new = t_new[5:11]
    b_new = t_new[11:]
   
    
    r_new = int(r_new, 2)
    g_new = int(g_new, 2)
    b_new = int(b_new, 2)
    
    r.append(r_new)
    g.append(g_new)
    b.append(b_new)

The above lines of code loop through all the string elements in the list "string2", convert them into integers, subsequently into binary, and store them in the t_new variable.

It then extracts the RGB565 data from the 16-bit binary value using slicing and stores them into three different variables. Finally, the individual RGB values are converted back into integers from binary and appended into their respective lists.

Approach 2:

for i in range(len(string2)):
    t = string2[i]
    t_new = bin(int('1'+t, scale))[3:]

    r_new = ((t_new >> 11) & 0b00011111) << 3
    g_new = ((t_new >> 5) & 0b00111111) << 2
    b_new = ((t_new >> 0) & 0b00011111) << 3

    r.append(r_new)
    g.append(g_new)
    b.append(b_new)

The above lines of code loop through all the string elements in the list "string2", convert them into integers, subsequently into binary, and store them in the t_new variable.

It then extracts the RGB565 data from the 16-bit binary value using the equation explained in the earlier section and stores them into three different variables. Finally, the individual RGB values are converted back into integers from binary and appended into their respective lists.

width = 176
height = 144
import numpy as np
r_new = np.reshape(r, (height, width)).T
g_new = np.reshape(g, (height, width)).T
b_new = np.reshape(b, (height, width)).T

These lines of code reshape the red, blue, and green 1D array into a 2D array of size (176, 144) and store it in a new array.

print(r_new)
print(g_new)
print(b_new)

These lines are code print the values of the respective RGB arrays.

final_image = np.stack([r_new, g_new, b_new], axis=2)
print(final_image.shape)

These lines of code stack the RGB array into a single 3D array.

final_image = final_image * 2

This line of code scale up the values in the 3D array by 2. I used it as a short hack to increase the brightness of the image.

print(final_image[1][:][0])

This line of code prints out the RGB values of an individual pixel from the 3D array.

import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
figure(figsize=(8, 6))
plt.plot(180)
#plt.imshow(final_image)
plt.axis("off")
import scipy.ndimage as ndimage
new_data = ndimage.rotate(final_image, 90, reshape=True)
plt.imshow(new_data, origin = 'lower')

These lines of code use matplotlib and scipy to appropriately rotate the image and display it to the user.

Why does the custom visualizer output different images compared to rawpixels.net?

One possible reason is that I didn't account for the little-endian representation. But, it turns out we wouldn't need this visualizer often in building this particular application as we'll be working with grayscale images. Nevertheless, it was a fun exercise to try out.

4.f Grayscale

Why do we need to work with grayscale images?

MNIST has grayscale images for its training data, so it's essential to work with them to ensure that the test data is around the same distribution as the training data.

4.g Types of grayscale

We'll learn about these different formulas to convert RGB images into grayscale images in the below sections.

• Average method
• Weighted method/Luminosity method
• Weighted method - 2
• Weighted method - 3

4.h Theaverage method

Github Link for this subsection.

Code explanation:

int red   = ((pixel >> 11) & 0x1f) << 3;
int green = ((pixel >> 5) & 0x3f) << 2; 
int blue  = ((pixel >> 0) & 0x1f) << 3; 
grayscale = (red + blue + green)/3

This sketch first converts RGB565 values into RGB888 format and finally into grayscale using the average method.

Theory:

The average method is the most simple one. You just have to take the average of three colors. Since it's an RGB image, it means that you have added r with g with b and then divide it by 3 to get your desired grayscale image.

The problem with this approach is that we take the average of the three colors. Since the three different colors have three different wavelengths and contribute to the formation of the image, we have to take the average according to their contribution. Right now, we are doing this, 33% of Red, 33% of Green, and 33% of Blue. We are taking 33% of each, which means that each portion has the same contribution to the image. But in reality, that's not the case.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>


void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    //Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < 144; i++) {
      for(int j = 0; j < 176; j++){
       uint16_t pixel = pixels[176*i +j];
       //Serial.println(pixel);
      // Serial.println(pixel,HEX);
       //RGB565 data to RGB888
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
       
       int grayscale = (red + blue + green)/3;
     
       Serial.print(grayscale);
       Serial.print(",");
          
      }
    }
  }
}

4.i The weighted methodorluminositymethod

Github Link for this subsection.

Code explanation:

int red   = ((pixel >> 11) & 0x1f) << 3;
int green = ((pixel >> 5) & 0x3f) << 2; 
int blue  = ((pixel >> 0) & 0x1f) << 3; 
grayscale = (0.299* red + 0.587 * green + 0.114 * blue);

This sketch first converts RGB565 values into RGB888 format and finally into grayscale using the luminosity method.

Theory:

Since red color has more wavelength than all the other colors. It means that we have to decrease the contribution of the red color, increase the contribution of the green color, and put the blue color contribution in between these two.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>


void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    //Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < 144; i++) {
      for(int j = 0; j < 176; j++){
       uint16_t pixel = pixels[176*i +j];
       //Serial.println(pixel);
      // Serial.println(pixel,HEX);
       //RGB565 data to RGB888
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
       
       int grayscale = (0.299* red + 0.587 * green + 0.114 * blue);
     
       Serial.print(grayscale);
       Serial.print(",");
          
      }
    }
  }
}

4.j Weightedmethod -2

Github Link for this subsection.

Code explanation:

int red   = ((pixel >> 11) & 0x1f) << 3;
int green = ((pixel >> 5) & 0x3f) << 2; 
int blue  = ((pixel >> 0) & 0x1f) << 3; 
grayscale = 1*red + 0*green + 0*blue;

This sketch first converts RGB565 values into RGB888 format and finally into grayscale using the 2nd version of the weighted method. This equation can be used if the image is known beforehand.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>


void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    //Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < 144; i++) {
      for(int j = 0; j < 176; j++){
       uint16_t pixel = pixels[176*i +j];
       //Serial.println(pixel);
      // Serial.println(pixel,HEX);
       //RGB565 data to RGB888
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
       
       int grayscale = 1*red + 0*green + 0*blue;
     
       Serial.print(grayscale);
       Serial.print(",");
          
      }
    }
  }
}

4.k Weightedmethod -2

Github Link for this subsection.

Code explanation:

int red   = ((pixel >> 11) & 0x1f) << 3;
int green = ((pixel >> 5) & 0x3f) << 2; 
int blue  = ((pixel >> 0) & 0x1f) << 3; 
grayscale = 0.2126 * red +0.7152 * green +0.0722 * blue;

This sketch first converts RGB565 values into RGB888 format and finally into grayscale using the 3rd version of the weighted method. This equation is just a more refined version of the weighted method.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>


void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    //Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < 144; i++) {
      for(int j = 0; j < 176; j++){
       uint16_t pixel = pixels[176*i +j];
       //Serial.println(pixel);
      // Serial.println(pixel,HEX);
       //RGB565 data to RGB888
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
       
      
       int grayscale = 0.2126 * red +0.7152 * green +0.0722 * blue;
      
       Serial.print(grayscale);
       Serial.print(",");
          
      }
    }
  }
}

4.l Grayscale.ipynb

Github Link for this subsection.

Code explanation:

lst = [[107,174,174,174....,153,144,144,144,108,110,84,208]

This line of code stores the output from the serial monitor in a list.

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
arr = np.array(lst)
arr.resize((144,176))
arr = arr.astype(np.uint8)
im = Image.fromarray(arr)
im

These lines of code convert the list into a NumPy array, resize it to a 2D shape, and display it to the user.

5. Which display to choose?

5.a General information about the OLED module

Features :

• No backlight required
• low power requirement
• Large viewing angle
• Uses I2C protocol
• Better performance characteristics than traditional LCD and LED displays.
• Only needs 2 I/O Port for control

5.b General information about the TFT LCD module

Features :

• Supports analog SPI and hardware SPI
• Wide viewing angle with suitable contrast.
• Uses SPI protocol

6. Getting started with the SSD1306 OLED module

6.a OLED:Bitmaps

Github Link for this subsection.

Code explanation:

This sketch creates a 112x112 bitmap and randomly turns on and off each pixel.

#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>

#define OLED_RESET 4

#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

unsigned char myBitmap [] PROGMEM = {
0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x70, 0x00, 0x00, 0x01, 0xf0, 
0x00, 0x00, 0x01, 0xf0, 0x00, 0x00, 0x01, 0xf0, 0x00, 0x00, 0x03, 0xf0, 0x00, 0x00, 0x03, 0xf0, 
0x00, 0x00, 0x03, 0xf0, 0x00, 0x00, 0x03, 0xf0, 0x00, 0x00, 0x07, 0xf0, 0x00, 0x00, 0x07, 0xf0, 
0x00, 0x00, 0x07, 0xf0, 0x00, 0x00, 0x07, 0xf0, 0x00, 0x00, 0x0f, 0xf0, 0x00, 0x00, 0x07, 0xf0, 
0x00, 0x00, 0x0f, 0xf0, 0x00, 0x00, 0x0f, 0xf0, 0x00, 0x00, 0x1f, 0xf0, 0x00, 0x00, 0x1f, 0xf0, 
0x00, 0x00, 0x3f, 0xf0, 0x00, 0x00, 0x3f, 0xf0, 0x00, 0x00, 0x3f, 0xf0, 0x00, 0x00, 0x7f, 0xf0, 
0x00, 0x00, 0x7f, 0xf0, 0x00, 0x00, 0xff, 0xf0, 0x00, 0x00, 0xff, 0xf0, 0x00, 0x00, 0xff, 0xe0
};
void setup()   {                
  // On my display, I had to used 0x3C as the address, something to do with the RESET not being
  // connected to the Arduino. THe 0x3D address below is the address used in the original
  // Adafruit OLED example 
  display.begin(SSD1306_SWITCHCAPVCC, 0x3C);  // initialize with the I2C addr 0x3D (for the 128x64)
  
  display.clearDisplay(); // Make sure the display is cleared
  // Draw the bitmap:
  // drawBitmap(x position, y position, bitmap data, bitmap width, bitmap height, color)
  display.drawBitmap(0, 0, myBitmap, 28, 28, WHITE);

  // Update the display
  
}


void loop() {
  display.clearDisplay();
  for(int i =0; i < 112; i++){
    myBitmap[i] = random(255);
  }
  display.drawBitmap(0, 0, myBitmap, 28, 28, WHITE);
  display.display();
  delay(100);
}

Now that dynamic bitmap is possible, we need to figure out how to get Softwire I2C working as the hardware I2C pins are already occupied by the OV7670 camera module.

6.b OLED:Softwire I2C

Github Link for this subsection.

The standard I2C library for the Arduino is the Wire library. While this library is sufficient most of the time, there are situations when it cannot be used:

• the I2C pins A4/A5 (or SDA/SCL) are in use already for other purposes
• same I2C addresses devices are used

So we write the SoftwireI2C library to use digit port and analog port to enable multiple same I2C addresses devices to work on Arduino. It utilises the pinMode(), digitalWrite() and digitalRead() functions. The pins to be used for the serial data (SDA) and serial clock (SCL) control lines can be defined at run-time.

Code explanation:

This sketch uses the Softwire library to run the basic Adafruit_OLED example.

/**************************************************************************
 This is an example for our Monochrome OLEDs based on SSD1306 drivers

 Pick one up today in the adafruit shop!
 ------> http://www.adafruit.com/category/63_98

 This example is for a 128x64 pixel display using I2C to communicate
 3 pins are required to interface (two I2C and one reset).

 Adafruit invests time and resources providing this open
 source code, please support Adafruit and open-source
 hardware by purchasing products from Adafruit!

 Written by Limor Fried/Ladyada for Adafruit Industries,
 with contributions from the open source community.
 BSD license, check license.txt for more information
 All text above, and the splash screen below must be
 included in any redistribution.
 **************************************************************************/

#include <SPI.h>
#include <SoftWire.h>
#include <AsyncDelay.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>

#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels

// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
// The pins for I2C are defined by the Wire-library. 
// On an arduino UNO:       A4(SDA), A5(SCL)
// On an arduino MEGA 2560: 20(SDA), 21(SCL)
// On an arduino LEONARDO:   2(SDA),  3(SCL), ...
#define OLED_RESET     4 // Reset pin # (or -1 if sharing Arduino reset pin)
#define SCREEN_ADDRESS 0x3C ///< See datasheet for Address; 0x3D for 128x64, 0x3C for 128x32

#define I2C_SDA 25
#define I2C_SCL 26

SoftWire sw(25, 26);

Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &sw, OLED_RESET);

#define NUMFLAKES     10 // Number of snowflakes in the animation example

#define LOGO_HEIGHT   16
#define LOGO_WIDTH    16
static const unsigned char PROGMEM logo_bmp[] =
{ 0b00000000, 0b11000000,
  0b00000001, 0b11000000,
  0b00000001, 0b11000000,
  0b00000011, 0b11100000,
  0b11110011, 0b11100000,
  0b11111110, 0b11111000,
  0b01111110, 0b11111111,
  0b00110011, 0b10011111,
  0b00011111, 0b11111100,
  0b00001101, 0b01110000,
  0b00011011, 0b10100000,
  0b00111111, 0b11100000,
  0b00111111, 0b11110000,
  0b01111100, 0b11110000,
  0b01110000, 0b01110000,
  0b00000000, 0b00110000 };

void setup() {
  Serial.begin(9600);

  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
  if(!display.begin(SSD1306_SWITCHCAPVCC, SCREEN_ADDRESS)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;); // Don't proceed, loop forever
  }

  // Show initial display buffer contents on the screen --
  // the library initializes this with an Adafruit splash screen.
  display.display();
  delay(2000); // Pause for 2 seconds

  // Clear the buffer
  display.clearDisplay();

  // Draw a single pixel in white
  display.drawPixel(10, 10, SSD1306_WHITE);

  // Show the display buffer on the screen. You MUST call display() after
  // drawing commands to make them visible on screen!
  display.display();
  delay(2000);
  // display.display() is NOT necessary after every single drawing command,
  // unless that's what you want...rather, you can batch up a bunch of
  // drawing operations and then update the screen all at once by calling
  // display.display(). These examples demonstrate both approaches...

  testdrawline();      // Draw many lines

  testdrawrect();      // Draw rectangles (outlines)

  testfillrect();      // Draw rectangles (filled)

  testdrawcircle();    // Draw circles (outlines)

  testfillcircle();    // Draw circles (filled)

  testdrawroundrect(); // Draw rounded rectangles (outlines)

  testfillroundrect(); // Draw rounded rectangles (filled)

  testdrawtriangle();  // Draw triangles (outlines)

  testfilltriangle();  // Draw triangles (filled)

  testdrawchar();      // Draw characters of the default font

  testdrawstyles();    // Draw 'stylized' characters

  testscrolltext();    // Draw scrolling text

  testdrawbitmap();    // Draw a small bitmap image

  // Invert and restore display, pausing in-between
  display.invertDisplay(true);
  delay(1000);
  display.invertDisplay(false);
  delay(1000);

  testanimate(logo_bmp, LOGO_WIDTH, LOGO_HEIGHT); // Animate bitmaps
}

void loop() {
}

void testdrawline() {
  int16_t i;

  display.clearDisplay(); // Clear display buffer

  for(i=0; i<display.width(); i+=4) {
    display.drawLine(0, 0, i, display.height()-1, SSD1306_WHITE);
    display.display(); // Update screen with each newly-drawn line
    delay(1);
  }
  for(i=0; i<display.height(); i+=4) {
    display.drawLine(0, 0, display.width()-1, i, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  delay(250);

  display.clearDisplay();

  for(i=0; i<display.width(); i+=4) {
    display.drawLine(0, display.height()-1, i, 0, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  for(i=display.height()-1; i>=0; i-=4) {
    display.drawLine(0, display.height()-1, display.width()-1, i, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  delay(250);

  display.clearDisplay();

  for(i=display.width()-1; i>=0; i-=4) {
    display.drawLine(display.width()-1, display.height()-1, i, 0, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  for(i=display.height()-1; i>=0; i-=4) {
    display.drawLine(display.width()-1, display.height()-1, 0, i, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  delay(250);

  display.clearDisplay();

  for(i=0; i<display.height(); i+=4) {
    display.drawLine(display.width()-1, 0, 0, i, SSD1306_WHITE);
    display.display();
    delay(1);
  }
  for(i=0; i<display.width(); i+=4) {
    display.drawLine(display.width()-1, 0, i, display.height()-1, SSD1306_WHITE);
    display.display();
    delay(1);
  }

  delay(2000); // Pause for 2 seconds
}

void testdrawrect(void) {
  display.clearDisplay();

  for(int16_t i=0; i<display.height()/2; i+=2) {
    display.drawRect(i, i, display.width()-2*i, display.height()-2*i, SSD1306_WHITE);
    display.display(); // Update screen with each newly-drawn rectangle
    delay(1);
  }

  delay(2000);
}

void testfillrect(void) {
  display.clearDisplay();

  for(int16_t i=0; i<display.height()/2; i+=3) {
    // The INVERSE color is used so rectangles alternate white/black
    display.fillRect(i, i, display.width()-i*2, display.height()-i*2, SSD1306_INVERSE);
    display.display(); // Update screen with each newly-drawn rectangle
    delay(1);
  }

  delay(2000);
}

void testdrawcircle(void) {
  display.clearDisplay();

  for(int16_t i=0; i<max(display.width(),display.height())/2; i+=2) {
    display.drawCircle(display.width()/2, display.height()/2, i, SSD1306_WHITE);
    display.display();
    delay(1);
  }

  delay(2000);
}

void testfillcircle(void) {
  display.clearDisplay();

  for(int16_t i=max(display.width(),display.height())/2; i>0; i-=3) {
    // The INVERSE color is used so circles alternate white/black
    display.fillCircle(display.width() / 2, display.height() / 2, i, SSD1306_INVERSE);
    display.display(); // Update screen with each newly-drawn circle
    delay(1);
  }

  delay(2000);
}

void testdrawroundrect(void) {
  display.clearDisplay();

  for(int16_t i=0; i<display.height()/2-2; i+=2) {
    display.drawRoundRect(i, i, display.width()-2*i, display.height()-2*i,
      display.height()/4, SSD1306_WHITE);
    display.display();
    delay(1);
  }

  delay(2000);
}

void testfillroundrect(void) {
  display.clearDisplay();

  for(int16_t i=0; i<display.height()/2-2; i+=2) {
    // The INVERSE color is used so round-rects alternate white/black
    display.fillRoundRect(i, i, display.width()-2*i, display.height()-2*i,
      display.height()/4, SSD1306_INVERSE);
    display.display();
    delay(1);
  }

  delay(2000);
}

void testdrawtriangle(void) {
  display.clearDisplay();

  for(int16_t i=0; i<max(display.width(),display.height())/2; i+=5) {
    display.drawTriangle(
      display.width()/2  , display.height()/2-i,
      display.width()/2-i, display.height()/2+i,
      display.width()/2+i, display.height()/2+i, SSD1306_WHITE);
    display.display();
    delay(1);
  }

  delay(2000);
}

void testfilltriangle(void) {
  display.clearDisplay();

  for(int16_t i=max(display.width(),display.height())/2; i>0; i-=5) {
    // The INVERSE color is used so triangles alternate white/black
    display.fillTriangle(
      display.width()/2  , display.height()/2-i,
      display.width()/2-i, display.height()/2+i,
      display.width()/2+i, display.height()/2+i, SSD1306_INVERSE);
    display.display();
    delay(1);
  }

  delay(2000);
}

void testdrawchar(void) {
  display.clearDisplay();

  display.setTextSize(1);      // Normal 1:1 pixel scale
  display.setTextColor(SSD1306_WHITE); // Draw white text
  display.setCursor(0, 0);     // Start at top-left corner
  display.cp437(true);         // Use full 256 char 'Code Page 437' font

  // Not all the characters will fit on the display. This is normal.
  // Library will draw what it can and the rest will be clipped.
  for(int16_t i=0; i<256; i++) {
    if(i == '\n') display.write(' ');
    else          display.write(i);
  }

  display.display();
  delay(2000);
}

void testdrawstyles(void) {
  display.clearDisplay();

  display.setTextSize(1);             // Normal 1:1 pixel scale
  display.setTextColor(SSD1306_WHITE);        // Draw white text
  display.setCursor(0,0);             // Start at top-left corner
  display.println(F("Hello, world!"));

  display.setTextColor(SSD1306_BLACK, SSD1306_WHITE); // Draw 'inverse' text
  display.println(3.141592);

  display.setTextSize(2);             // Draw 2X-scale text
  display.setTextColor(SSD1306_WHITE);
  display.print(F("0x")); display.println(0xDEADBEEF, HEX);

  display.display();
  delay(2000);
}

void testscrolltext(void) {
  display.clearDisplay();

  display.setTextSize(2); // Draw 2X-scale text
  display.setTextColor(SSD1306_WHITE);
  display.setCursor(10, 0);
  display.println(F("scroll"));
  display.display();      // Show initial text
  delay(100);

  // Scroll in various directions, pausing in-between:
  display.startscrollright(0x00, 0x0F);
  delay(2000);
  display.stopscroll();
  delay(1000);
  display.startscrollleft(0x00, 0x0F);
  delay(2000);
  display.stopscroll();
  delay(1000);
  display.startscrolldiagright(0x00, 0x07);
  delay(2000);
  display.startscrolldiagleft(0x00, 0x07);
  delay(2000);
  display.stopscroll();
  delay(1000);
}

void testdrawbitmap(void) {
  display.clearDisplay();

  display.drawBitmap(
    (display.width()  - LOGO_WIDTH ) / 2,
    (display.height() - LOGO_HEIGHT) / 2,
    logo_bmp, LOGO_WIDTH, LOGO_HEIGHT, 1);
  display.display();
  delay(1000);
}

#define XPOS   0 // Indexes into the 'icons' array in function below
#define YPOS   1
#define DELTAY 2

void testanimate(const uint8_t *bitmap, uint8_t w, uint8_t h) {
  int8_t f, icons[NUMFLAKES][3];

  // Initialize 'snowflake' positions
  for(f=0; f< NUMFLAKES; f++) {
    icons[f][XPOS]   = random(1 - LOGO_WIDTH, display.width());
    icons[f][YPOS]   = -LOGO_HEIGHT;
    icons[f][DELTAY] = random(1, 6);
    Serial.print(F("x: "));
    Serial.print(icons[f][XPOS], DEC);
    Serial.print(F(" y: "));
    Serial.print(icons[f][YPOS], DEC);
    Serial.print(F(" dy: "));
    Serial.println(icons[f][DELTAY], DEC);
  }

  for(;;) { // Loop forever...
    display.clearDisplay(); // Clear the display buffer

    // Draw each snowflake:
    for(f=0; f< NUMFLAKES; f++) {
      display.drawBitmap(icons[f][XPOS], icons[f][YPOS], bitmap, w, h, SSD1306_WHITE);
    }

    display.display(); // Show the display buffer on the screen
    delay(200);        // Pause for 1/10 second

    // Then update coordinates of each flake...
    for(f=0; f< NUMFLAKES; f++) {
      icons[f][YPOS] += icons[f][DELTAY];
      // If snowflake is off the bottom of the screen...
      if (icons[f][YPOS] >= display.height()) {
        // Reinitialize to a random position, just off the top
        icons[f][XPOS]   = random(1 - LOGO_WIDTH, display.width());
        icons[f][YPOS]   = -LOGO_HEIGHT;
        icons[f][DELTAY] = random(1, 6);
      }
    }
  }
}

This library doesn't support bitmaps, so we'll need to try printing pixel by pixel.

6.c OLEDPrinting pixel-by-pixel

I tried printing each pixel of an image sequentially but the printing time wasn't satisfactory for the use case so I shifted to working with an LCD module.

7. Getting started with the TFT LCD module

7.a TFT:Basics

Github Link for this subsection.

Code explanation:

This sketch runs the basic TFT example code.

#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>


#define TFT_CS        10
#define TFT_RST        8 // Or set to -1 and connect to Arduino RESET pin
#define TFT_DC         9
#define TFT_MOSI 12  // Data out
#define TFT_SCLK 13  // Clock out

// For ST7735-based displays, we will use this call
Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_MOSI, TFT_SCLK, TFT_RST);

float p = 3.1415926;

void setup(void) {
  Serial.begin(9600);
  Serial.print(F("Hello! ST77xx TFT Test"));

  // Use this initializer if using a 1.8" TFT screen:
  tft.initR(INITR_BLACKTAB);      // Init ST7735S chip, black tab

  // OR use this initializer if using a 1.8" TFT screen with offset such as WaveShare:
  // tft.initR(INITR_GREENTAB);      // Init ST7735S chip, green tab
  //tft.setSPISpeed(40000000);

  Serial.println(F("Initialized"));

  uint16_t time = millis();
  tft.fillScreen(ST77XX_BLACK);
  time = millis() - time;

  Serial.println(time, DEC);
  delay(500);

  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  testdrawtext("Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur adipiscing ante sed nibh tincidunt feugiat. Maecenas enim massa, fringilla sed malesuada et, malesuada sit amet turpis. Sed porttitor neque ut ante pretium vitae malesuada nunc bibendum. Nullam aliquet ultrices massa eu hendrerit. Ut sed nisi lorem. In vestibulum purus a tortor imperdiet posuere. ", ST77XX_WHITE);
  delay(1000);

  // tft print function!
  tftPrintTest();
  delay(4000);

  // a single pixel
  tft.drawPixel(tft.width()/2, tft.height()/2, ST77XX_GREEN);
  delay(500);

  // line draw test
  testlines(ST77XX_YELLOW);
  delay(500);

  // optimized lines
  testfastlines(ST77XX_RED, ST77XX_BLUE);
  delay(500);

  testdrawrects(ST77XX_GREEN);
  delay(500);

  testfillrects(ST77XX_YELLOW, ST77XX_MAGENTA);
  delay(500);

  tft.fillScreen(ST77XX_BLACK);
  testfillcircles(10, ST77XX_BLUE);
  testdrawcircles(10, ST77XX_WHITE);
  delay(500);

  testroundrects();
  delay(500);

  testtriangles();
  delay(500);

  mediabuttons();
  delay(500);

  Serial.println("done");
  delay(1000);
}

void loop() {
  tft.invertDisplay(true);
  delay(500);
  tft.invertDisplay(false);
  delay(500);
}

void testlines(uint16_t color) {
  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=0; x < tft.width(); x+=6) {
    tft.drawLine(0, 0, x, tft.height()-1, color);
    delay(0);
  }
  for (int16_t y=0; y < tft.height(); y+=6) {
    tft.drawLine(0, 0, tft.width()-1, y, color);
    delay(0);
  }

  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=0; x < tft.width(); x+=6) {
    tft.drawLine(tft.width()-1, 0, x, tft.height()-1, color);
    delay(0);
  }
  for (int16_t y=0; y < tft.height(); y+=6) {
    tft.drawLine(tft.width()-1, 0, 0, y, color);
    delay(0);
  }

  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=0; x < tft.width(); x+=6) {
    tft.drawLine(0, tft.height()-1, x, 0, color);
    delay(0);
  }
  for (int16_t y=0; y < tft.height(); y+=6) {
    tft.drawLine(0, tft.height()-1, tft.width()-1, y, color);
    delay(0);
  }

  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=0; x < tft.width(); x+=6) {
    tft.drawLine(tft.width()-1, tft.height()-1, x, 0, color);
    delay(0);
  }
  for (int16_t y=0; y < tft.height(); y+=6) {
    tft.drawLine(tft.width()-1, tft.height()-1, 0, y, color);
    delay(0);
  }
}

void testdrawtext(char *text, uint16_t color) {
  tft.setCursor(0, 0);
  tft.setTextColor(color);
  tft.setTextWrap(true);
  tft.print(text);
}

void testfastlines(uint16_t color1, uint16_t color2) {
  tft.fillScreen(ST77XX_BLACK);
  for (int16_t y=0; y < tft.height(); y+=5) {
    tft.drawFastHLine(0, y, tft.width(), color1);
  }
  for (int16_t x=0; x < tft.width(); x+=5) {
    tft.drawFastVLine(x, 0, tft.height(), color2);
  }
}

void testdrawrects(uint16_t color) {
  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=0; x < tft.width(); x+=6) {
    tft.drawRect(tft.width()/2 -x/2, tft.height()/2 -x/2 , x, x, color);
  }
}

void testfillrects(uint16_t color1, uint16_t color2) {
  tft.fillScreen(ST77XX_BLACK);
  for (int16_t x=tft.width()-1; x > 6; x-=6) {
    tft.fillRect(tft.width()/2 -x/2, tft.height()/2 -x/2 , x, x, color1);
    tft.drawRect(tft.width()/2 -x/2, tft.height()/2 -x/2 , x, x, color2);
  }
}

void testfillcircles(uint8_t radius, uint16_t color) {
  for (int16_t x=radius; x < tft.width(); x+=radius*2) {
    for (int16_t y=radius; y < tft.height(); y+=radius*2) {
      tft.fillCircle(x, y, radius, color);
    }
  }
}

void testdrawcircles(uint8_t radius, uint16_t color) {
  for (int16_t x=0; x < tft.width()+radius; x+=radius*2) {
    for (int16_t y=0; y < tft.height()+radius; y+=radius*2) {
      tft.drawCircle(x, y, radius, color);
    }
  }
}

void testtriangles() {
  tft.fillScreen(ST77XX_BLACK);
  uint16_t color = 0xF800;
  int t;
  int w = tft.width()/2;
  int x = tft.height()-1;
  int y = 0;
  int z = tft.width();
  for(t = 0 ; t <= 15; t++) {
    tft.drawTriangle(w, y, y, x, z, x, color);
    x-=4;
    y+=4;
    z-=4;
    color+=100;
  }
}

void testroundrects() {
  tft.fillScreen(ST77XX_BLACK);
  uint16_t color = 100;
  int i;
  int t;
  for(t = 0 ; t <= 4; t+=1) {
    int x = 0;
    int y = 0;
    int w = tft.width()-2;
    int h = tft.height()-2;
    for(i = 0 ; i <= 16; i+=1) {
      tft.drawRoundRect(x, y, w, h, 5, color);
      x+=2;
      y+=3;
      w-=4;
      h-=6;
      color+=1100;
    }
    color+=100;
  }
}

void tftPrintTest() {
  tft.setTextWrap(false);
  tft.fillScreen(ST77XX_BLACK);
  tft.setCursor(0, 30);
  tft.setTextColor(ST77XX_RED);
  tft.setTextSize(1);
  tft.println("Hello World!");
  tft.setTextColor(ST77XX_YELLOW);
  tft.setTextSize(2);
  tft.println("Hello World!");
  tft.setTextColor(ST77XX_GREEN);
  tft.setTextSize(3);
  tft.println("Hello World!");
  tft.setTextColor(ST77XX_BLUE);
  tft.setTextSize(4);
  tft.print(1234.567);
  delay(1500);
  tft.setCursor(0, 0);
  tft.fillScreen(ST77XX_BLACK);
  tft.setTextColor(ST77XX_WHITE);
  tft.setTextSize(0);
  tft.println("Hello World!");
  tft.setTextSize(1);
  tft.setTextColor(ST77XX_GREEN);
  tft.print(p, 6);
  tft.println(" Want pi?");
  tft.println(" ");
  tft.print(8675309, HEX); // print 8,675,309 out in HEX!
  tft.println(" Print HEX!");
  tft.println(" ");
  tft.setTextColor(ST77XX_WHITE);
  tft.println("Sketch has been");
  tft.println("running for: ");
  tft.setTextColor(ST77XX_MAGENTA);
  tft.print(millis() / 1000);
  tft.setTextColor(ST77XX_WHITE);
  tft.print(" seconds.");
}

void mediabuttons() {
  // play
  tft.fillScreen(ST77XX_BLACK);
  tft.fillRoundRect(25, 10, 78, 60, 8, ST77XX_WHITE);
  tft.fillTriangle(42, 20, 42, 60, 90, 40, ST77XX_RED);
  delay(500);
  // pause
  tft.fillRoundRect(25, 90, 78, 60, 8, ST77XX_WHITE);
  tft.fillRoundRect(39, 98, 20, 45, 5, ST77XX_GREEN);
  tft.fillRoundRect(69, 98, 20, 45, 5, ST77XX_GREEN);
  delay(500);
  // play color
  tft.fillTriangle(42, 20, 42, 60, 90, 40, ST77XX_BLUE);
  delay(50);
  // pause color
  tft.fillRoundRect(39, 98, 20, 45, 5, ST77XX_RED);
  tft.fillRoundRect(69, 98, 20, 45, 5, ST77XX_RED);
  // play color
  tft.fillTriangle(42, 20, 42, 60, 90, 40, ST77XX_GREEN);
}

7.b TFT LCD: Bitmaps and printing pixel-by-pixel

This library also doesn't support displaying bitmaps so instead, I tried printing pixel by pixel and this time around, the time taken was actually satisfactory. So, l went with this module.

8. Integrating the camera and LCD module

8.a TFT + OV7670: Display test

Github Link for this subsection

Code explanation:

tft.fillScreen(ST77XX_BLACK);

This line of code fills the screen with black color.

for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);

These lines of code draw a 28x28 green colored block.

The sketch:

#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>


  // For the breakout board, you can use any 2 or 3 pins.
  // These pins will also work for the 1.8" TFT shield.
  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6

// For 1.44" and 1.8" TFT with ST7735 use:
Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);
#include <Arduino_OV767X.h>

uint16_t pixels[176 *144];
uint16_t color;
uint8_t red, blue, green;

void setup(void) {
  Serial.begin(9600);

  // Use this initializer if using a 1.8" TFT screen:
  tft.initR(INITR_BLACKTAB); 
  delay(1000);// Init ST7735S chip, black tab
  Serial.println(F("Initialized"));
 
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }
  Serial.println(F("Initialized"));
  


  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);
 tft.fillScreen(ST77XX_BLACK);
}

8.b TFT + OV7670: Static image test

Github Link for this subsection.

Now that we've figured out how to display a green block, let's display an image from an array of stored HEX values.

Code explanation:

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

This array stores the HEX values of all the pixels.

for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
       tft.drawPixel(i,j,pixel);
}
}

These lines of code loop through the array and draws the image on the TFT LCD display.

The sketch:

#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>

  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

uint16_t color, pixel;
uint8_t red, blue, green;

void setup() {
  Serial.begin(9600);
  while (!Serial)
    delay(10);
 // Serial.print(F("Hello! ST77xx TFT Test"));
  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
        red   = ((pixel >> 11) & 0x1f) << 3;
        green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      color = tft.color565(red+50, green+50, blue+50);
      Serial.print(red);
      Serial.print(", ");
      Serial.print(green);
      Serial.print(", ");
      Serial.println(blue); 
      tft.drawPixel(i,j,color);
      delay(0);
    }
  }
  delay(100);
  //tft.fillScreen(ST77XX_BLACK);
}

8.c TFT + OV7670: Liveimagetest

Github Link for this subsection.

Now that we've figured out how to display an image from an array of stored HEX values, let's expand it to display live images.

Code explanation:

uint16_t pixels[176 * 144];

This line of code declares an array to store the image captured from the camera.

Camera.readFrame(pixels);

This line of code reads one frame from the camera and stores it in the pixels array.

for (int i = 0; i < 112; i++) 
{
   for(int j = 0; j < 112; j++)
   {
      uint16_t pixel = pixels[176*i +j];
      tft.drawPixel(i,j,pixel);
   }
}

These lines of code loop through the pixels array and draws the image on the TFT LCD display.

The sketch:

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>
#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>

#define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);
    tft.fillScreen(ST77XX_BLACK);
    for (int i = 0; i < 112; i++) {
      for(int j = 0; j < 112; j++){
       uint16_t pixel = pixels[176*i +j];
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
      
      Serial.println(red);
      Serial.println(green);
      Serial.println(blue);
      tft.drawPixel(i,j,pixels[176*i+j]);
      }
    }
    delay(2000);
}

9. Building the MNIST TinyML model

Note: If you are new to TensorFlow or TinyML, I would highly recommend you to read through this Introduction to TinyML blog to get a good grasp of the concepts before diving into this section.

Now that we've got the camera and display working and integrated, let's build the machine learning model to recognize the digits.

9.a Exploringthe Colab notebook

Github Link for this subsection.

Code explanation:

import numpy as np                   # advanced math library
import matplotlib.pyplot as plt      # MATLAB like plotting routines
import random                        # for generating random numbers

from keras.datasets import mnist     # MNIST dataset is included in Keras
from keras.models import Sequential  # Model type to be used

from keras.layers.core import Dense, Dropout, Activation # Types of layers to be used in our model
from keras.utils import np_utils                         # NumPy related tools
import tensorflow as tf

tf.config.run_functions_eagerly(True)

These lines of code import the necessary library to build and visualize our model.

(X_train, y_train), (X_test, y_test) = mnist.load_data()
print("X_train shape", X_train.shape)
print("y_train shape", y_train.shape)
print("X_test shape", X_test.shape)
print("y_test shape", y_test.shape)

>>X_train shape (60000, 28, 28)
>>y_train shape (60000,)
>>X_test shape (10000, 28, 28)
>>y_test shape (10000,)

These lines of code load the MNIST test and train images into the right variables.

plt.rcParams['figure.figsize'] = (9,9) # Make the figures a bit bigger

for i in range(9):
    plt.subplot(3,3,i+1)
    num = random.randint(0, len(X_train))
    plt.imshow(X_train[num], cmap='gray', interpolation='none')
    plt.title("Class {}".format(y_train[num]))
    
plt.tight_layout()

These lines of code visualize nine different images from the training data of the MNIST data set.

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
     
matprint(X_train[num])

These lines of code display a random image from the training data as an array of values.

from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D, GlobalAveragePooling2D, Flatten
from keras.layers import BatchNormalization

These lines of code import the necessary layers to build the machine learning model.

# Again, do some formatting
# Except we do not flatten each image into a 784-length vector because we want to perform convolutions first

X_train = X_train.reshape(60000, 28, 28, 1) #add an additional dimension to represent the single-channel
X_test = X_test.reshape(10000, 28, 28, 1)

X_train = X_train.astype('float32')         # change integers to 32-bit floating point numbers
X_test = X_test.astype('float32')

#X_train /= 255                              # normalize each value for each pixel for the entire vector for each input
#X_test /= 255

print("Training matrix shape", X_train.shape)
print("Testing matrix shape", X_test.shape)

These lines of code pre-process the training and test data to work e.g: Normalization, Float64 to Float32 conversion, and reshaping.

# one-hot format classes

nb_classes = 10 # number of unique digits

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

These lines of code one-hot encode the labels of the training and testing images.

from keras.layers.convolutional import DepthwiseConv2D
from keras.backend import relu
from keras.activations import softmax
model = Sequential()                                 # Linear stacking of layers

model.add(DepthwiseConv2D((3,3),input_shape=(28,28,1)))
# Convolution Layer 1
model.add(Conv2D(2, (3, 3)))      # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer1 = Activation('relu')                     # activation
model.add(convLayer1)

# Convolution Layer 2
model.add(Conv2D(2, (3, 3)))                        # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer2 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer2)

# Convolution Layer 3
model.add(Conv2D(4,(3, 3)))                         # 4 different 3x3 kernels -- so 4 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer3 = Activation('relu')                     # activation
model.add(convLayer3)

# Convolution Layer 4
model.add(Conv2D(4, (3, 3)))                        # 4 different 3x3 kernels -- so 64 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer4 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer4)

model.add(Flatten())                                 
model.add(Dense(5,activation = relu))                           
model.add(Dense(10, activation = softmax))

These lines of code define the actual layers that go in the machine learning model.

model.summary()

This line of code displays information about the model architecture to the user.

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

This line of code defines the loss, optimizer and other metrics to be used by the machine learning model while training.

history = model.fit(X_train,Y_train, steps_per_epoch=60000//128, epochs=3, verbose=1, 
                    validation_data=(X_test,Y_test))

This line of code trains the machine learning model.

score = model.evaluate(X_test, Y_test)
print('Test score:', score[0])
print('Test accuracy:', score[1])

These lines of code evaluate the machine learning model and print the accuracy and score to the user.

!apt-get -qq install xxd
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model to disk
open("gesture_model.tflite", "wb").write(tflite_model)
  
import os
basic_model_size = os.path.getsize("gesture_model.tflite")
print("Model is %d bytes" % basic_model_size)

These lines of code convert the TensorFlow model into a TensorFlow Lite model.

!echo "const unsigned char model[] = {" > /content/model.h
!cat gesture_model.tflite | xxd -i      >> /content/model.h
!echo "};"                              >> /content/model.h

import os
model_h_size = os.path.getsize("model.h")
print(f"Header file, model.h, is {model_h_size:,} bytes.")
print("\nOpen the side panel (refresh if needed). Double click model.h to download the file.")

These lines of code convert the TensorFlow Lite model into a C file to be used by the microcontroller.

Now that we've built and trained our model, we now need to figure out how to send data from the camera into the TinyML model.

The camera's output is 176x144 in size and the model's input is 28x28 in size. There are two approaches we can try:

• 1. Cropping a 28x28 image from the camera's output.
• 2. Cropping a 128x128 image from the camera's output and reshaping it to 28x28

We'll try both approaches in the below sections.

9.b OV7670:Crop test

Github Link for this subsection.

Code explanation:

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};

This line of code stores the HEX values in an array of size 176X144.

for(int a = 0; a< 112; a++)
   {
    for(int b = 0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }

These lines of code loop through the array and print out the first 28x28 pixels of the image.

The sketch:

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};
int arr1[28*28];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();
  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    
    for(int a =0; a< 112; a++)
   {
    for(int b =0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }
   Serial.println("");
  }
}

9.c OV7670:Reshape test

Github Link for this subsection.

Code explanation:

for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");

These lines of code use a 4x4 pooling kernel, with stride 1, to go over a 112x112 2D array to output a 28x28 image.

The sketch:

#include "num.h"
float arr1[28 * 28];
int filterWidth = 4;
int filterheight = 4;
void setup() {
  Serial.begin(9600);
  
  
}

void loop() {
  // put your main code here, to run repeatedly:
  for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");
 
}

9.d Exploring the reshape.ipynb Colab notebook

Github Link for this subsection.

Code explanation:

from skimage.transform import resize
t = number28.reshape(28,28)
print(t.shape)
number112 = resize(t, (112, 112))
print(number112.dtype)
"""
for i in range(0,112):
  for j in range(0,112):
    if number112[i][j] < 10e-20:
      number112[i][j] = 0
      /
"""
for i in range(0,112):
  for j in range(0,112):
      number112[i][j] = number112[i][j] * 10e+19
      if number112[i][j] < 10:
       number112[i][j] = 0
np.amax(number112)
number112 = number112 /12
np.amax(number112)
plt.imshow(number112, cmap='gray', interpolation='none')
plt.imshow(t, cmap='gray', interpolation='none')

These lines of code upscale the 28X28 MNIST image into a 112X112 image.

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
matprint(number112)

These lines of code print the upscaled 112X112 MNIST image.

number28new = number28new.reshape(28,28)
for i in range(0,28):
  for j in range(0,28):
      if number28new[i][j] < 35:
       number28new[i][j] = 0
plt.imshow(number28new, cmap='gray', interpolation='none')

These lines of code print the reshaped 28x28 MNIST image.

10. Testing the TinyML model

Now that we have built a TinyML model and tested the two approaches to inputting data into our model, it's time to integrate the TinyML model into the main application.

10.a TinyML Model: Input test

Github Link for this subsection.

How do we send in the input data?

With a time-series-based model, it was pretty obvious to send in the input data as the input tensor was a 1-dimensional array.

//1D input tensor
tflInputTensor->data.f[i * 3 + 0] = (ax + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 1] = (ay + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 2] = (az + 8.0) / 16.0;

I didn't know how to send data into this particular TinyML model so I devised a test.

I stored the array values of an MNIST training data into a num.h file.

num.h:

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0};

model.h:

//MPU6050_model.ino
#include <TensorFlowLite.h>
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = &micro_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}

void test_func(){
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
 Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  */
  
}
void loop() {
  
}

Code Explanation: Input tensors:

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }

To test out the input tensors, I sent in input data using a 2D loop.

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

Then, I printed out the values stored in the input tensors of the TinyML model.

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

I then printed out the actual values stored in the loop.

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }

Finally, I printed out the difference between the values stored in the input tensors and the values stored in the array. If all the differences are zero, then the input has been properly stored in the input tensors just the way we want.

10.b TinyML Model: Model test

Github Link for this subsection.

Now that we have figured out how to send in input data, It's time to test out the model. This code is the same as above except that inference is called and the output tensors are printed out.

num.h file:

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0

The sketch:

//MPU6050_model.ino
#include <TensorFlowLite.h>
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = &micro_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}


void loop() {
  
    
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j]/255.0;
    } 
  }
  /*
   for(int i = 0; i < 1; i++){
    for(int j =0; j < 1; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 

  }
  */
  
  TfLiteStatus invokeStatus = tflInterpreter->Invoke();
  out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];
  
  float maxVal = out[0];
  int maxIndex = 0;
  for(int k =0; k < 10;k++){
    if (out[k] > maxVal) {
         maxVal = out[k];
         maxIndex = k;
      } 
  }
  Serial.print("Number ");
  Serial.print(maxIndex);
  Serial.println(" detected");
  Serial.print("Confidence: ");
  Serial.println(maxVal);
  
  Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);
  
}

Code explanation:

TfLiteStatus invokeStatus = tflInterpreter->Invoke();

These lines of code invoke inference on the data stored in the input tensors.

out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];

These lines of code access the output tensors and store them in an array to be used later.

float maxVal = out[0];
int maxIndex = 0;
for(int k =0; k < 10;k++){
  if (out[k] > maxVal) {
      maxVal = out[k];
      maxIndex = k;
    } 
  }
Serial.print("Number ");
Serial.print(maxIndex);
Serial.println(" detected");
Serial.print("Confidence: ");
Serial.println(maxVal);

These lines of code print out the class index and confidence value of the class with the highest output value.

Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);

These lines of code print out the confidence values for each of the individual classes.

10.c MNIST: Test images

Github Link for this subsection.

You can use these images, converted into NumPy arrays, to test out your MNIST TinyML model.

11. Integration time

11.a TinyML Model: Cropped input data

Github Link for this subsection.

Code explanation:

Camera.readFrame(pixels);

This line of code reads one frame from the camera and stores it in the pixels array.

for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
       pixel = pixels[176*i +j];
      tft.drawPixel(i,j,pixel);
    }
  }
  delay(1000);

These lines of code loop through the pixels array crop a 28x28 image from it and display it on the screen.

for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
       pixel = pixels[176*i +j];
       red   = ((pixel >> 11) & 0x1f) << 3;
       green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      grayscale = (red + blue + green)/3 ;
      if(grayscale <128){
        grayscale =0;
      }
     tflInterpreter->input(0)->data.f[28*i+j] = grayscale / 255;
      Serial.println(grayscale);
    }
  }

These lines of code loop through the pixels array crop a 28x28 image from it and sends it as input to the TinyML model.

The sketch:

//MPU6050_model.ino
#include <TensorFlowLite.h>
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>
#include <Arduino_OV767X.h>

const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

#define TFT_CS        A7
#define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
#define TFT_DC         A6

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

uint16_t pixels[176*144];
uint16_t color, pixel;
uint8_t red, blue, green;
float grayscale;

Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);

  tft.initR(INITR_BLACKTAB); 
  delay(100);
   if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }
  Serial.println(F("Initialized"));
     
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = &micro_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  
  tft.fillScreen(ST77XX_BLACK);
  delay(100);
 
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  Camera.readFrame(pixels);
  
  for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
       pixel = pixels[176*i +j];
      tft.drawPixel(i,j,pixel);
    }
  }
  delay(1000);
  for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
       pixel = pixels[176*i +j];
       red   = ((pixel >> 11) & 0x1f) << 3;
       green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      grayscale = (red + blue + green)/3 ;
      if(grayscale <128){
        grayscale =0;
      }
     tflInterpreter->input(0)->data.f[28*i+j] = grayscale / 255;
      Serial.println(grayscale);
    }
  }
  
  delay(1000);
  TfLiteStatus invokeStatus = tflInterpreter->Invoke();
  out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];

    float maxVal = out[0];
  int maxIndex = 0;
  for(int k =0; k < 10;k++){
    if (out[k] > maxVal) {
         maxVal = out[k];
         maxIndex = k;
      } 
  }
  Serial.print("Number ");
  Serial.print(maxIndex);
  Serial.println(" detected");
  Serial.print("Confidence: ");
  Serial.println(maxVal);
     
}

11.b TinyML Model: Reshaped input data

Github Link for this subsection.

Code explanation:

Camera.readFrame(pixels);

This line of code reads one frame from the camera and stores it in the pixels array.

for(int i =0; i<112;i++){
    for(int j =0;j<112;j++){
       
      tft.drawPixel(i,j,pixels[176*i+j]);
      Serial.print("");
    }
  }

These lines of code loop through the pixels array crop a 112x112 image from it and display it on the screen.

Serial.println("");
  for(int i =0; i< 28; i++)
   {
    for(int j =0; j < 28; j++)
    {
       int sum =0;
       for(int k =0; k<4;k++)
       {
        for(int l =0; l<4; l++)
        {
          sum += pixels[4*(176*i+j) + 176 * k + l];
        }
       }
      sum = sum /16;
      //arr1[i*28+j] = sum;
      tflInterpreter->input(0)->data.f[28*i+j] = float(sum / 255.0);
      Serial.print(sum);
      Serial.print(", ");
    }
    Serial.println("");
 }

These lines of code loop through the pixels array crop a 112x112 image, reshape it into a 28x28 image, and sends it to the TinyML model.

The sketch:

//MPU6050_model.ino
#include <TensorFlowLite.h>
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7735.h> // Hardware-specific library for ST7735
#include <SPI.h>
#include <Arduino_OV767X.h>

const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

#define TFT_CS        A7
#define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
#define TFT_DC         A6

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

uint16_t pixels[176*144];
uint16_t color, pixel;
uint8_t red, blue, green;
int grayscale;

Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

void setup() {

  Serial.begin(9600);
  while (!Serial)
    delay(10);

  tft.initR(INITR_BLACKTAB); 
  delay(1000);
   if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }
  Serial.println(F("Initialized"));
     
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = &micro_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  
  tft.fillScreen(ST77XX_BLACK);
  delay(100);
 
  
  
  
}

void loop() {
  Camera.readFrame(pixels);
  tft.fillScreen(ST77XX_BLACK);
  for(int i =0; i<112;i++){
    for(int j =0;j<112;j++){
       
      tft.drawPixel(i,j,pixels[176*i+j]);
      Serial.print("");
    }
  }
 // delay(1000);
  for(int i =0; i<112;i++){
    for(int j =0;j<112;j++){
       pixel = pixels[176*i +j];
       red   = ((pixel >> 11) & 0x1f) << 3;
       green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      grayscale = (red + blue + green)/3 ;
      if(grayscale <160){
        grayscale =0;
      }
      pixels[176*i +j] = grayscale;
    
     //tflInterpreter->input(0)->data.f[28*i+j] = grayscale / 255;
    }
    }

  Serial.println("");
  for(int i =0; i< 28; i++)
   {
    for(int j =0; j < 28; j++)
    {
       int sum =0;
       for(int k =0; k<4;k++)
       {
        for(int l =0; l<4; l++)
        {
          sum += pixels[4*(176*i+j) + 176 * k + l];
        }
       }
      sum = sum /16;
      //arr1[i*28+j] = sum;
      tflInterpreter->input(0)->data.f[28*i+j] = float(sum / 255.0);
      Serial.print(sum);
      Serial.print(", ");
    }
    Serial.println("");
   }
  
  delay(1000);
  TfLiteStatus invokeStatus = tflInterpreter->Invoke();
  out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];

    float maxVal = out[0];
  int maxIndex = 0;
  for(int k =0; k < 10;k++){
    if (out[k] > maxVal) {
         maxVal = out[k];
         maxIndex = k;
      } 
  }
  Serial.print("Number ");
  Serial.print(maxIndex);
  Serial.println(" detected");
  Serial.print("Confidence: ");
  Serial.println(maxVal);
     
}

12. Problems with the project/ How to improve the project

12.a Color space of the LCD display doesn't match the ov7670

When displaying images from the live feed of the camera, a variety of color gradients pop up, I'm not entirely sure why this happens but my guess is that it's due to a loss of color space information inbetween conversions.

12.b LCDrefreshes after printing every pixel

The approach I used basically prints pixel by pixel. The problem with the Adafruit_st7735 is that it sends the buffer automatically after printing a pixel. I assume it's an easy fix to comment out the line of code, that sends the buffer, in the library.

12.cWhere is the camera pointing at

One of the major pain points while building this example was trying to figure out where the camera was pointing at. If a small 3D printed rectangular piece of plastic that helps eyeball roughly where the camera is looking will help a lot while collecting training data and testing the application.

13. Some helpful pointers to building your own image recognition project

Why this section?

You might have become confused with the thousand steps and detours to building this application so here's a list of things to simplify building your next image recognition application.

• Decide on an idea
• Decide on the components
• Collect training data
• Keep two formats of each training image( PNG file, HEX file)
• Build and train a TinyML model
• Test TinyML model
• Integrate the TinyML model into the main application
• Test application in the real world

14. Collecting training data using the OV7670 camera module

14.a The sketch

This sketch reads a frame from the camera and outputs the RGB565 values on the serial monitor.

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

unsigned short pixels[176 * 144]; // QCIF: 176x144 X 2 bytes per pixel (RGB565)

void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);

    int numPixels = Camera.width() * Camera.height();

    for (int i = 0; i < numPixels; i++) {
        unsigned short p = pixels[i];
        if (p < 0x1000) {
          Serial.print('0');
        }
        if (p < 0x0100) {
         Serial.print('0');
        }
        if (p < 0x0010) {
          Serial.print('0');
        }
        Serial.print(p, HEX);
      }
  }
}

15. Conclusion

I thank my GSoC mentor, Paul Ruiz, for guiding me throughout the project!

Links

• Part 1/4
• Part 2/4
• Part 3/4
• Part 4/4
• Full tutorial