PROBLEM DESCRIPTION
THE IDEA
PROOF OF CONCEPT
USED KEY FEATURES FOR THE PROJECT
PROJECT DOCUMENTATION
- Creation of the physical prototype
- Image capture and ML model creation in Edge Impulse
- Free parking spaces detection firmware program
THE CODE

Published August 7, 2022

Free Car Parking Monitoring Device

Detection and monitoring with AWS free parking spaces through the Sony Spresense board and pre-trained ML models thanks to Edge Impulse.

AdvancedFull instructions provided1,796

Judges Choice

Spresense Developer Challenge 2022

Edge Impulse Special Prize

Spresense Developer Challenge 2022

Things used in this project

Hardware components

Sony Spresense boards (main & extension)

Spresense Main Board. Used as main element to manage all the peripherals.

Sony Spresense camera board

Used to take photos from the street.

Sony Spresense LTE extension board

Used for send/receive data.

Micro SD Card

Used to save the photos taken with the Camera Board and to save the AWS Certificates.

IoT SIM Card - Truphone

SIM card for IoT provided by hacskter to be able to use the LTE board and send data.

SG90 Micro-servo motor

Used to rotate the device at various angles.

Jumper wires (generic)

3 Male to Female Jumper Wires. Used to connect the ServoMotor to the Spresense Main Board.

Anti-Slip Pad

Used for a better grip of the part to the surface.

Software apps and online services

Arduino IDE

Used to develop the firmware and program the spresense main board.

Edge Impulse Studio

Used for the creation and trainning of the object detection model.

Amazon Web Services AWS IoT

Used to Send/Receive data to/with Amazon AWS IoT. Then, this data may be processed through other AWS services.

Autodesk Tinkercad

Used to generate the 3D model to print.

Hand tools and fabrication machines

3D Printer (generic)

Used to make the casing for the device.

Story

PROBLEM DESCRIPTION

Nowadays, we hear everywhere about the "Smart city" concept. It is a concept that is booming and many people and many businesses are focusing on this in order to make cities smarter and more efficient.

One of the things that I miss and that causes me more headaches (I guess like many other people), is the issue of parking in the city.

Smart cities should be prepared to control and monitor city parking spaces, thus improving traffic and making people's lifes easier.

One of the biggest problems I have in my day to day, is the difficulty to park in the area where I live and know if there are free places and when.That's why I decided to address this issue by making this project.

THE IDEA

A possible idea to solve this problem could be through small wireless devices that monitor parking spaces (e.g. anchored to street lamps), and send the data to a central station to simultaneously display in real time the current parking status of a street.

When you get to the street, you could see through a panel the number of free spaces in that street, you could even get that same data from the Internet and know it remotely before arriving at the street itself.

Idea of an intelligent street monitoring the parking of a Smart City.

PROOF OF CONCEPT

To develop this idea I have decided to make a proof of concept using the Sony Spresense board. It has a camera and I can add machine learning models to detect objects (in this case, free parking places and cars), thanks to Edge Impulse.

For this proof of concept I have decided toplace the device on the balcony of my house, from where I can see perfectly the parking area of my street.

In the following image you can see the device checking the parking zone.

Device scanning the street through the balcony

USED KEY FEATURES FOR THE PROJECT

The central pillar of this project resides in the Sony Spresense board that, together with a few added components, will allow me to make an ideal prototype for this proof of concept.

Board Basic Connection

Used Hardware Spresense key features:

Spresense main board
Spresense camera
Spresense LTE
Spresense SDCard
Spresense RTC
Spresense LEDs

Used other components:

Servomotor
3D printed case/base/support

Used Spresense Firmware libraries:

Camera
SDCard
LTE
RTC

Other Firmware libraries used for the project:

MQTT
Servomotor

Used Software:

Edge impulse
Arduino IDE
AWS IoT
Tinkercad

PROJECT DOCUMENTATION

The project could be divided into 3 parts:

Physical prototype design.
Image capture and ML model creation with Edge Impulse. (Needed to make the device intelligent and allow it to detect cars and free spaces).
Free parking spacesdetection firmware program.

- Creation of the physical prototype

Knowing already, which are the physical components that are necessary for the project, the first thing is to make a prototype with which to make the tests.

The 3D model for printing has been made in tinkercad and in the following images you can see the parts have been generated (Static and Rotor Base, LTE board case). The 3D files are ….

3D view of the physical components used to create the device.

3D view to check the printed cases.

3D view with transparent cases which allow to see the internal assembly.

3D view of the device.

Final physical prototype

- Image capture and ML model creation in Edge Impulse

Once we have the physical part created, it is necessary to start with the programming part.

In this part the idea is to capture a large amount of image samples of the street (with Sony Spresense board)in order to be able to transfer them to Edge Impulse and train the model to detect cars and free spaces.

This program will perform the following functions:

Move the camera to various angles to capture a wider range of the whole street. The servomotor is used for this purpose.
Capture images and save them on the SD card.
Program the RTC alarm to perform the action periodically, so we can capture photos at various times and get images of the car park throughout the day.
Copy the images to Edge Impulse and generate a model capable of detecting cars and free spaces.

In order to select this project, simply uncomment the PROJECT_TAKE_PHOTOS directive.

.h with the option to select which project to compile and flash to the board

There is a section at the end of this documentation where I explain how the code is organised to make it more understandable.

After running this project we will get a large amount of images on the SDCard and we can go to Edge Impulse to load and tag them to generate the model capable of detecting the cars.

EDGE IMPULSE

These are the steps to follow to generate the intelligent model that detects cars and free spaces through Edge Impulse.

Go to Data acquisition and upload all the images.

Data acquisiton Edge Impulse

Label all photos indicating what is a car and what is a free space (the more samples there are the more accurate the detection).

Image labeling in Edge Impulse

Create the impulse indicating the image size to be used.

Creation of the impulse

Generate features from the Image section, and then push Start training from the Object detection section. This will tell you how much space it takes up and the accuracy of the model.

Training the model in Edge Impulse

In the deployment section we will autogenerate the code to introduce in our board (It will generate a.zip file that will be necessary to introduce it to the arduino IDE as a library).

Deployment in Edge Impulse

- Free parking spaces detection firmware program

This is the final firmware that connects to AWS IoT using the MQTT protocol to warn of free parking spaces. It has the following twofeatures:

Launch a request through AWS to scan the street. (scans the street only once).
Enable automatic monitoring through AWS (scans the street periodically depending on the sent time in seconds).

In order to select this project, simply uncomment the PROJECT_CAR_PARKING_SPACES directive.

.h with the option to select which project to compile and flash to the board

In order to connect to AWS IoT it is important to have the CERTS folder on the SD card with your own certificates and add the address of your broker. The topic I have used is 'spresense/freeParking', but you can feel free to change the names in this.h with whatever you think is more suitable for your application.

MQTT configuration in spresense.h file

The 'request'parameter is used to trigger or not the request to scan the street. ('0' do nothing, '1' ask the board to scan the street and return the current parking status).

Response from the Spresense Board after check the current street status.

The 'alarm' parameter is used to launch a periodical check of the street automatically according to the value in seconds that we set.(In this case we ask it to check the parking in the street every 30 seconds).

Request to '1' to check the current state of the street. Alarm '30' to check periodically (30sec)

Spresense board scanning the street.

THE CODE

I would like to explain small details of the code to make it more understandable for anyone who wants to pick it up/check it.

The file structure is the next one:

Project Structure

It is a very easy to understand code that is based on two main files:

spresense.h (in BSP folder): This is the file used to configure everything necessary for the project. In this file you can choose the project you want to compile (TAKE_PHOTOS or CAR_PARKING_DETECTION), and on the other hand it contains all the configuration of the peripherals and features used (camera configuration, AWS certificates, LTE, servomotor pin, RTC times, MQTT...).The code is totally modular, so anyone who wants to do without a specific peripheral because they want to use another one or don't need it, can simply comment out any of the directives xxxxx_PRESENT.That's why anyone who needs to tweak the code, can start from this file for most of the important changes and to configure it to their liking.

Modular Firmware. Easy to select the peripherals used for each project.

ProyectoSpresenseParking.ino:This is the base file where the setup and loop functions typical of the Arduino IDE are located.This is where all the firmware logic resides.

The other files are simply used to maintain a better organisation of the code and separate the functions for each peripheral used by the Spresense board.

Note:The library used for MQTT is the ArduinoMqttClient library (version 0.1.5). The ArduinoJSON library version used is the 6.14.1. In newer versions, the compilation crashes, so I've decided to use this one.

Schematics

Code

/********************************************************************************
-----> INCLUDES
********************************************************************************/
#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */
#include "LIB/CAMERA/inc/CameraSpresense.h"

/********************************************************************************
-----> DEFINITIONS
********************************************************************************/

/********************************************************************************
-----> VARIABLES
********************************************************************************/
/********************************************************************************
-----> FUNCTIONS
********************************************************************************/
static void camera_fnCallBack_vo(CamImage vF_Image);

bool CAMERA_fnInitialize_b(void)
{
  bool bRet = true;
  CamErr err = CAM_ERR_SUCCESS;

#ifdef SERIAL_PRINT_DEBUG
  Serial.println("Prepare camera");
#endif /* SERIAL_PRINT_DEBUG */

  err = theCamera.begin(1, 
                        CAMERA_FPS_VALUE, 
                        CAMERA_IMAGE_HEIGHT, 
                        CAMERA_IMAGE_WIDTH, 
                        CAM_IMAGE_PIX_FMT_YUV422, 
                        1); /* begin() without parameters means that number of buffers = 1, 30FPS, QVGA, YUV 4:2:2 format */
  if (err != CAM_ERR_SUCCESS)
  {
    bRet = false;
  }

  return(bRet);
}

bool CAMERA_fnStart_b(void)
{
  bool bRet = true;
  CamErr err = CAM_ERR_SUCCESS;
  /* Start video stream.
   * If received video stream data from camera device,
   *  camera library call CamCB.
   */
#ifdef SERIAL_PRINT_DEBUG
  Serial.println("Start streaming");
#endif /* SERIAL_PRINT_DEBUG */
  err = theCamera.startStreaming(true, camera_fnCallBack_vo);
  if (err != CAM_ERR_SUCCESS)
  {
      bRet = false;
  }
  else
  {
    if(true == CAMERA_fnSetAutoWhiteBalanceMode_b())
    {
      if(false == CAMERA_fnSetStillPictureImageFormat_b())
      {
        bRet = false;
      }
    }
    else
    {
      bRet = false;
    }
  }

  return(bRet);
}

void CAMERA_fnStop_vo(void)
{
  Serial.println("Stop");
  theCamera.startStreaming(false, camera_fnCallBack_vo);
  //theCamera.end();
}

bool CAMERA_fnSetStillPictureImageFormat_b(void)
{
  bool bRet = true;
  CamErr err = CAM_ERR_SUCCESS;
  /* Set parameters about still picture.
   * In the following case, QUADVGA and JPEG.
   */
#ifdef SERIAL_PRINT_DEBUG
  Serial.println("Set still picture format");
#endif /* SERIAL_PRINT_DEBUG */
   
  err = theCamera.setStillPictureImageFormat(CAMERA_IMAGE_HEIGHT,
                                            CAMERA_IMAGE_WIDTH,
#ifdef PROJECT_TAKE_PHOTOS
                                            CAM_IMAGE_PIX_FMT_RGB565,
#else
                                            CAM_IMAGE_PIX_FMT_RGB565,
#endif /* PROJECT_TAKE_PHOTOS */ 
                                            1);
  if (err != CAM_ERR_SUCCESS)
  {
      bRet = false;
      Serial.println(err);
  }

  return(bRet); 
}

CamImage CAMERA_fnTakePicture_CamImage(void)
{
  CamImage Image;
#ifdef SERIAL_PRINT_DEBUG
  Serial.println("Picture taken");
#endif /* SERIAL_PRINT_DEBUG */
  Image = theCamera.takePicture();
  return(Image);
}

bool CAMERA_fnIsImageAvailable_b(CamImage vF_Image)
{
  return(vF_Image.isAvailable());
}

bool CAMERA_fnSetAutoWhiteBalanceMode_b(void)
{
  bool bRet = true;
  CamErr err = CAM_ERR_SUCCESS;
  /* Auto white balance configuration */
#ifdef SERIAL_PRINT_DEBUG
  Serial.println("Set Auto white balance parameter");
#endif /* SERIAL_PRINT_DEBUG */
  err = theCamera.setAutoWhiteBalanceMode(CAM_WHITE_BALANCE_AUTO);
  if (err != CAM_ERR_SUCCESS)
  {
      bRet = false;
  }

  return(bRet);
}

/****************************************************************************
 * Callback from Camera library when video frame is captured.
 ****************************************************************************/
static void camera_fnCallBack_vo(CamImage vF_Image)
{
  /* Check the img instance is available or not. */
  CamErr Error =  CAM_ERR_SUCCESS;
  if (vF_Image.isAvailable())
  {
    /* You can use image data directly by using getImgSize() and getImgBuff().
    * for displaying image to a display, etc. */
#ifdef SERIAL_PRINT_DEBUG
     Serial.print("Image data size = ");
     Serial.print(vF_Image.getImgSize(), DEC);
     Serial.print(" , ");

     Serial.print("buff addr = ");
     Serial.print((unsigned long)vF_Image.getImgBuff(), HEX);
     Serial.println("");
#endif /* SERIAL_PRINT_DEBUG */
  }
  else
  {
#ifdef SERIAL_PRINT_DEBUG
    Serial.println("Failed to get video stream image");
#endif /* SERIAL_PRINT_DEBUG */
  }
}

/********************************************************************************
-----> INCLUDES
********************************************************************************/
#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */

/********************************************************************************
-----> DEFINITIONS
********************************************************************************/

/********************************************************************************
-----> VARIABLES
********************************************************************************/
// initialize the library instance
LTEModem modem;
// initialize the library instance
LTE        lteAccess;
LTEScanner scannerNetworks;
unsigned long LocalTime;
// IMEI variable
String IMEI             =     "";
String VERSION          =     "";
LTENetworkRatType RAT   =     LTE_NET_RAT_UNKNOWN; /* LTE_NET_RAT_NBIOT, LTE_NET_RAT_CATM */
/********************************************************************************
-----> FUNCTIONS
********************************************************************************/
bool LTE_fnInitialize_b(void)
{
  bool bRet = false;
  char apn[LTE_NET_APN_MAXLEN] = APP_LTE_APN;
  LTENetworkAuthType authtype = APP_LTE_AUTH_TYPE;
  char user_name[LTE_NET_USER_MAXLEN] = APP_LTE_USER_NAME;
  char password[LTE_NET_PASSWORD_MAXLEN] = APP_LTE_PASSWORD;

  if(lteAccess.begin() != LTE_SEARCHING)
  {
#ifdef SERIAL_PRINT_DEBUG
    Serial.println("Could not transition to LTE_SEARCHING.");
#endif /* SERIAL_PRINT */
  }
  else
  {
    /* The connection process to the APN will start.
     * If the synchronous parameter is false,
     * the return value will be returned when the connection process is started.
     */
    if (lteAccess.attach(APP_LTE_RAT,
                         apn,
                         user_name,
                         password,
                         authtype,
                         APP_LTE_IP_TYPE) == LTE_READY) 
    {
#ifdef SERIAL_PRINT_DEBUG
      Serial.println("attach succeeded.");
#endif /* SERIAL_PRINT */
    }
  }

  return(bRet);
}

uint32_t LTE_fnGetTime_u32(void)
{
  uint32_t u32Ret = 0;
  u32Ret = lteAccess.getTime();
  /* Convertir segundos a calendario */
#ifdef SERIAL_PRINT_DEBUG
  Serial.println(u32Ret);
#endif /* SERIAL_PRINT */
  return(u32Ret);
}

void LTE_fnDoAttach_vo(void)
{
  char apn[LTE_NET_APN_MAXLEN] = APP_LTE_APN;
  LTENetworkAuthType authtype = APP_LTE_AUTH_TYPE;
  char user_name[LTE_NET_USER_MAXLEN] = APP_LTE_USER_NAME;
  char password[LTE_NET_PASSWORD_MAXLEN] = APP_LTE_PASSWORD;

  /* Set if Access Point Name is empty */
  if (strlen(APP_LTE_APN) == 0) {
    Serial.println("This sketch doesn't have a APN information.");
    //readApnInformation(apn, &authtype, user_name, password);
  }
  Serial.println("=========== APN information ===========");
  Serial.print("Access Point Name  : ");
  Serial.println(apn);
  Serial.print("Authentication Type: ");
  Serial.println(authtype == LTE_NET_AUTHTYPE_CHAP ? "CHAP" :
                 authtype == LTE_NET_AUTHTYPE_NONE ? "NONE" : "PAP");
  if (authtype != LTE_NET_AUTHTYPE_NONE) {
    Serial.print("User Name          : ");
    Serial.println(user_name);
    Serial.print("Password           : ");
    Serial.println(password);
  }

  while (true) {

    /* Power on the modem and Enable the radio function. */

    if (lteAccess.begin() != LTE_SEARCHING) {
      Serial.println("Could not transition to LTE_SEARCHING.");
      Serial.println("Please check the status of the LTE board.");
      for (;;) {
        sleep(1);
      }
    }

    /* The connection process to the APN will start.
     * If the synchronous parameter is false,
     * the return value will be returned when the connection process is started.
     */
    if (lteAccess.attach(APP_LTE_RAT,
                         apn,
                         user_name,
                         password,
                         authtype,
                         APP_LTE_IP_TYPE,
                         false) == LTE_CONNECTING) {
      Serial.println("Attempting to connect to network.");
      break;
    }
    /* If the following logs occur frequently, one of the following might be a cause:
     * - APN settings are incorrect
     * - SIM is not inserted correctly
     * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
     *   your LTE board may not support it.
     */
    Serial.println("An error has occurred. Shutdown and retry the network attach preparation process after 1 second.");
    lteAccess.shutdown();
    sleep(1);
  }
}

#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */
#include <ArduinoMqttClient.h>
#include "LIB/SDCARD/inc/SDCardSpresense.h"
#include "LIB/LTE/inc/LTESpresense.h"
#include <LTE.h>
#include <ArduinoJson.h>
#include <ArduinoJson.hpp>

#ifdef RTC_PRESENT
#include "LIB/RTC/inc/RTCSpresense.h"
#endif /* RTC_PRESENT */

#ifdef MQTT_PRESENT
LTETLSClient client;
MqttClient mqttClient(client);

char broker[] = BROKER_NAME;
int port      = BROKER_PORT;
char topic[]  = MQTT_TOPIC;

static bool flagMQTTRequest = false;

void messageHandler(char *topic, byte *payload, unsigned int length);

void MQTT_fnSendMessage_vo(String vF_sMessage, char *vF_cTopic)
{
    // send message, the Print interface can be used to set the message contents
    mqttClient.beginMessage(vF_cTopic);
    mqttClient.print(vF_sMessage);
    mqttClient.endMessage();
}

void MQTT_fnSetCACertificate_vo(void)
{
  File rootCertsFile = SDCARD_fnOpen_vo(ROOTCA_FILE);
  client.setCACert(rootCertsFile, rootCertsFile.available());
  rootCertsFile.close();
}
void MQTT_fnSetCertificate_vo(void)
{
  File certsFile = SDCARD_fnOpen_vo(CERT_FILE);
  client.setCertificate(certsFile, certsFile.available());
  certsFile.close();
  
}
void MQTT_fnSetPrivateKey_vo(void)
{
  File priKeyFile = SDCARD_fnOpen_vo(KEY_FILE);
  client.setPrivateKey(priKeyFile, priKeyFile.available());
  priKeyFile.close();
}

bool MQTT_fnConnect_b(char *vF_cBroker, int vF_iPort)
{
  bool bRet = false;

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(vF_cBroker);

  if (!mqttClient.connect(vF_cBroker, vF_iPort)) 
  {
    Serial.print("MQTT connection failed! Error code = ");
    Serial.println(mqttClient.connectError());
    // do nothing forevermore:
  }
  else
  {
    Serial.println("You're connected to the MQTT broker!");
    bRet = true;
  }

  return(bRet);
}

void MQTT_fnInitialize_vo(void)
{
  //Set certifications via a file on the SD card before connecting to the MQTT broker
  MQTT_fnSetCACertificate_vo();
  MQTT_fnSetCertificate_vo();
  MQTT_fnSetPrivateKey_vo();
  MQTT_fnConnect_b(broker, port);
}

void MQTT_fnReceiveInit_vo(void)
{
  // Subscribe to MQTT and register a callback
  mqttClient.onMessage(messageHandler);
  mqttClient.subscribe(MQTT_TOPIC);
}


void MQTT_fnPoll_vo(void)
{
  mqttClient.poll();
}

void messageHandler(char *topic, byte *payload, unsigned int length) 
{
  unsigned long RTCAlarm;
  bool CarParkCheckRequest;

#ifdef SERIAL_PRINT_DEBUG
  // we received a message, print out the topic and contents
  Serial.print("Received a message with topic '");
  Serial.print(mqttClient.messageTopic());
  Serial.println("'");
#endif /* SERIAL_PRINT_DEBUG */

  /* Check JSON received data. */
  StaticJsonDocument<200> doc;
  DeserializationError error = deserializeJson(doc, mqttClient);
  
  /* Get RTCAlarm value. */
  RTCAlarm = doc["alarm"];
  Serial.print("Alarm Time:");
  Serial.println(RTCAlarm);
  RTC_fnSetAlarm_vo(RTCAlarm);    /* Set RTC alarm for a periodic interruption */

  /* Get if there is a request to check the parking spaces. */
  CarParkCheckRequest = doc["request"];
  if(true == CarParkCheckRequest)
  {
    Serial.print("Request:");
    Serial.println(CarParkCheckRequest);
    flagMQTTRequest = true;
  }
}

bool MQTT_fnIsMQTTRequestActivated_b(void)
{
  return(flagMQTTRequest);
}

void MQTT_fnClearMQTTRequestFlag_vo(void)
{
  flagMQTTRequest = false;
}

#endif /* MQTT_PRESENT */

#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */
#include "LIB/RTC/inc/RTCSpresense.h"

static bool flagRTCAlarm = false;
uint32_t rtcAlarmTime = 0;
void rtc_fnCallBackAlarmExpired_vo();
void RTC_fnInitialize_vo(void)
{
  RTC.begin();
  RTC_fnAddAlarm_vo();
}

void RTC_fnAddAlarm_vo(void)
{
  // Set the RTC alarm handler
  RTC.attachAlarm(rtc_fnCallBackAlarmExpired_vo);
}

void RTC_fnSetAlarm_vo(uint32_t vF_u32Seconds)
{
  if(0 != vF_u32Seconds)
  {/* When alarm time is 0, alarm is disabled */
      rtcAlarmTime = vF_u32Seconds;
      RtcTime now = RTC.getTime();
      now += vF_u32Seconds;
      RTC.setAlarm(now);
  }
}

void RTC_fnSetTime_vo(RtcTime vF_u32Seconds)
{
  RTC.setTime(vF_u32Seconds);
}

bool RTC_fnIsAlarmActivated_b(void)
{
  return(flagRTCAlarm);
}

void RTC_fnClearAlarmFlag_vo(void)
{
  flagRTCAlarm = false;
}

RtcTime RTC_fnGetTime_vo(void)
{
  return(RTC.getTime());
}


void rtc_fnCallBackAlarmExpired_vo(void)
{
  RtcTime now = RTC.getTime();

  // Set the RTC alarm every rtcAlarmTime seconds
  RtcTime alm = now + rtcAlarmTime;
  RTC.setAlarm(alm);

  if(false == flagRTCAlarm)
  {
    flagRTCAlarm = true;
  }
}

/********************************************************************************
-----> INCLUDES
********************************************************************************/
#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */

/********************************************************************************
-----> DEFINITIONS
********************************************************************************/

/********************************************************************************
-----> VARIABLES
********************************************************************************/
SDClass  theSD;
/********************************************************************************
-----> FUNCTIONS
********************************************************************************/
bool SDCARD_fnInitialize_b(void)
{
  bool bRet = false;

  while (!theSD.begin()) 
  {
#ifdef SERIAL_PRINT_DEBUG
    Serial.println("Insert SD card.");  /* wait until SD card is mounted. */
#endif /* SERIAL_PRINT */
  }
  bRet = true;
  return(bRet);
}

void SDCARD_fnSaveFile_vo(char *vF_Filename, const char *vF_Buffer, size_t vF_Size)
{
  theSD.remove(vF_Filename);
  File myFile = theSD.open(vF_Filename, FILE_WRITE);
  myFile.write(vF_Buffer, vF_Size);
  myFile.close();
}

File SDCARD_fnOpen_vo(char *vF_Filename) 
{
  File rootCertsFile = theSD.open(vF_Filename, FILE_READ);
  return(rootCertsFile);
}

#include <stdio.h> 
#include <stdlib.h>
#include <Arduino.h>  //For Serial
#include "BSP/spresense.h"   /* Include board defines */
#include "LIB/MOT/inc/ServoSpresense.h"

static Servo s_servo; /**< Servo object */

void SERVO_fnInitialize_vo(void)
{
  s_servo.attach(SERVO_PIN);
}

void SERVO_fnMoveToAngle_vo(uint8_t vF_u8Angle)
{
  uint8_t u8Angle;
  u8Angle = s_servo.read();
  if(u8Angle < vF_u8Angle)
  {
    for (u8Angle; u8Angle <= vF_u8Angle; u8Angle += 1) 
    { // in steps of 1 degree
      s_servo.write(u8Angle);              // tell servo to go to position in variable 'pos'
      delay(15);                       // waits 15 ms for the servo to reach the position
    }
  }
  else
  {
    for (u8Angle; u8Angle > vF_u8Angle; u8Angle -= 1) 
    { // in steps of 1 degree
      s_servo.write(u8Angle);              // tell servo to go to position in variable 'pos'
      delay(15);                       // waits 15 ms for the servo to reach the position
    }
  }

  
}

/*
 *  spresense.h - Board definition file
 *  
 *
 */
#ifndef _SPRESENSE_H_
#define _SPRESENSE_H_

/********************************************************************************
-----> PROJECT SELECTOR (Select the project you want to compile and flash)
********************************************************************************/
//#define PROJECT_TAKE_PHOTOS                 /* This project takes photos and save them in a SD Card */
#define PROJECT_CAR_PARKING_SPACES            /* This project takes photos and checks if there is free car parking space. 
                                              /* Sends/Receives data through MQTT to AWS IoT */

/********************************************************************************
-----> PROJECT FEATURES
********************************************************************************/
/***** PROJECT_TAKE_PHOTOS *****/
#ifdef PROJECT_TAKE_PHOTOS
#define SERIAL_PRINT_DEBUG                      /* Used to check through the Serial Terminal what's happening */
#define CAMERA_PRESENT                          /* Used to activate camera functions to take photos */
#define SDCARD_PRESENT                          /* Used to save the taken photos to the SD Card */    
#define LTE_PRESENT                             /* Used to check current time and to load the RTC timer */
#define RTC_PRESENT                             /* Used to configure the RTC alarm to take photos periodically */
#define SERVO_PRESENT                           /* Used to move a servo motor and check different street angles */                
//#define MQTT_PRESENT                          /* Used for MQTT protocol with AWS IoT */
//#define EDGE_IMPULSE_OBJECT_DETECTION_PRESENT   /* Used to check free car parking space thank's to Edge Impulse library */ 
#define BRD_DEBUG 
#endif /* PROJECT_TAKE_PHOTOS */

/***** PROJECT_CAR_PARKING_SPACES *****/
#ifdef PROJECT_CAR_PARKING_SPACES
#define SERIAL_PRINT_DEBUG                      /* Used to check through the Serial Terminal what's happening */   
#define CAMERA_PRESENT                          /* Used to activate camera functions to take photos */
#define SDCARD_PRESENT                          /* Used to get the AWS Certificates for AWS IoT connection */ 
#define LTE_PRESENT                             /* Used for wireless communication and get current time */
#define RTC_PRESENT                             /* Used for periodical events for free space checking */
#define SERVO_PRESENT                           /* Used to move a servo motor and check different street angles */
#define MQTT_PRESENT                            /* Used for MQTT protocol with AWS IoT */
#define EDGE_IMPULSE_OBJECT_DETECTION_PRESENT /* Used to check free car parking space thank's to Edge Impulse library */ 
#define BRD_DEBUG
#endif /* PROJECT_CAR_PARKING_SPACES */



/********************************************************************************
-----> MQTT
********************************************************************************/
#ifdef MQTT_PRESENT

#define BROKER_NAME                 "" // replace with your broker
#define BROKER_PORT                 8883                                // port 8883 is the default for MQTT over TLS.
#define ROOTCA_FILE                 "CERTS/AmazonRootCA1_MyBoard.pem"   // Define the path to a file containing CA
                                                                        // certificates that are trusted. Define with the file saved in your SDCard.
#define CERT_FILE                   "CERTS/MyBoard-certificate.pem.crt" // Define the path to a file containing certificate
                                                                        // for this client, if required by the server. Define with the file saved in your SDCard.
#define KEY_FILE                    "CERTS/MyBoard-private.pem.key"     // Define the path to a file containing private key
                                                                        // for this client, if required by the server. Define with the file saved in your SDCard.

// MQTT topic
#define MQTT_TOPIC                  "spresense/freeParking"             /* Topic used to request the Car park check, */
                                                                        /* and to answer if there is free car park space */

#endif /* MQTT_PRESENT */
/********************************************************************************
-----> LTE
********************************************************************************/
#ifdef LTE_PRESENT
#include <LTE.h>                    /* Include for LTE */
#define APP_LTE_APN                 "iot.truphone.com"
#define APP_LTE_USER_NAME           ""
#define APP_LTE_PASSWORD            ""
// APN IP type
#define APP_LTE_IP_TYPE             (LTE_NET_IPTYPE_V4V6) // IP : IPv4v6
//#define APP_LTE_IP_TYPE           (LTE_NET_IPTYPE_V4) // IP : IPv4
// #define APP_LTE_IP_TYPE          (LTE_NET_IPTYPE_V6) // IP : IPv6
// APN authentication type
#define APP_LTE_AUTH_TYPE           (LTE_NET_AUTHTYPE_CHAP) // Authentication : CHAP
//#define APP_LTE_AUTH_TYPE         (LTE_NET_AUTHTYPE_PAP) // Authentication : PAP
//#define APP_LTE_AUTH_TYPE         (LTE_NET_AUTHTYPE_NONE) // Authentication : NONE

/* RAT to use
 * Refer to the cellular carriers information
 * to find out which RAT your SIM supports.
 * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
 */
#define APP_LTE_RAT                 (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1)
// #define APP_LTE_RAT              (LTE_NET_RAT_NBIOT) // RAT : NB-IoT
#endif /* LTE_PRESENT */

/********************************************************************************
-----> CAMERA
********************************************************************************/
#ifdef CAMERA_PRESENT
//Includes
#include <Camera.h>           /* Include for Camera */
#define CAMERA_IMAGE_HEIGHT         160                     /* Photo height */
#define CAMERA_IMAGE_WIDTH          160                     /* Photo width */
#define CAMERA_FPS_VALUE            CAM_VIDEO_FPS_6         /* Frames per seconds */
#endif /* CAMERA_PRESENT */

/********************************************************************************
-----> SDCARD
********************************************************************************/
#ifdef SDCARD_PRESENT
#include <SDHCI.h>            /* Include for SD card */  
#endif /* SDCARD_PRESENT */

/********************************************************************************
-----> RTC
********************************************************************************/
#ifdef RTC_PRESENT
#include <RTC.h>
#define DEFAULT_ALARM_TIME_SEC_U    30                      /* 30 seconds for default alarm time */
#endif /* RTC_PRESENT */

/********************************************************************************
-----> SERVO MOTOR
********************************************************************************/
#ifdef SERVO_PRESENT
#include <Servo.h>
#define SERVO_PIN                   PIN_D03                 /* Servomotor connected to this pin */
#endif /* SERVO_PRESENT */

#endif /* _SPRESENSE_H_ */

/* FreeCarDetection Firmware
 * Author: I.Nagore
 * Date: 07/08/2022
 */
/* Includes ---------------------------------------------------------------- */

/********************************************************************************
-----> INCLUDES
********************************************************************************/
#include "BSP/spresense.h"   /* Include board defines */
#include <Servo.h>

#ifdef CAMERA_PRESENT
#include "LIB/CAMERA/inc/CameraSpresense.h"
#endif /* CAMERA_PRESENT */

#ifdef SDCARD_PRESENT
#include "LIB/SDCARD/inc/SDCardSpresense.h"
#endif /* SDCARD_PRESENT */

#ifdef LTE_PRESENT
#include "LIB/LTE/inc/LTESpresense.h"
#endif /* LTE_PRESENT */

#ifdef RTC_PRESENT
#include "LIB/RTC/inc/RTCSpresense.h"
#endif /* RTC_PRESENT */

#ifdef SERVO_PRESENT
#include "LIB/MOT/inc/ServoSpresense.h"
#endif /* SERVO_PRESENT */

#ifdef MQTT_PRESENT
#include "LIB/MQTT/inc/MQTTSpresense.h"
#include <ArduinoJson.h>
#include <ArduinoJson.hpp>
#endif /* MQTT_PRESENT */

#ifdef EDGE_IMPULSE_OBJECT_DETECTION_PRESENT
#include <CarParking_160_inferencing.h>                     /* Add library autogenerated with EdgeImpulse for object detection */
#endif /* EDGE_IMPULSE_OBJECT_DETECTION_PRESENT */

/********************************************************************************
-----> DEFINITIONS
********************************************************************************/
#define BAUDRATE  (115200)
#define LED0_ON   digitalWrite(LED0, HIGH)                    /* Macros for Switch on/off Board LEDs */
#define LED0_OFF  digitalWrite(LED0, LOW)
#define LED1_ON   digitalWrite(LED1, HIGH)
#define LED1_OFF  digitalWrite(LED1, LOW)
#define LED2_ON   digitalWrite(LED2, HIGH)
#define LED2_OFF  digitalWrite(LED2, LOW)
#define LED3_ON   digitalWrite(LED3, HIGH)
#define LED3_OFF  digitalWrite(LED3, LOW)

#define PICTURE_SIGNALLING_ON   LED0_ON                       /* Switch on LED when take a photo */
#define PICTURE_SIGNALLING_OFF  LED0_OFF                      /* Switch off LED when taken photo */

#ifdef SERVO_PRESENT
#define _NUMBER_OF_ANGLES_U     4                             /* Number of angles for the servo move to take photos */
#endif /* SERVO_PRESENT */

/********************************************************************************
-----> VARIABLES
********************************************************************************/
#ifdef SERVO_PRESENT
const uint8_t a6u8AngleArray[_NUMBER_OF_ANGLES_U] = { 15, 
                                                      50, 
                                                      100, 
                                                      150};   /* Angles to take photos. Can be changed depending your application. */
#endif /* SERVO_PRESENT */
#ifdef PROJECT_TAKE_PHOTOS
static uint32_t system_u32PhotosNumber = 0;                   /* Variable used to save number of taken photos */
#endif /* PROJECT_TAKE_PHOTOS */


/********************************************************************************
-----> Static Functions Prototype
********************************************************************************/
/**
 * @brief      Initialize all the peripherals used for the project (RTC, LTE, SDCard, Camera, MQTT, Servo...)
 */
void system_fnPeripheralsInitialize_vo(void);

/**
 * @brief      Takes a photo and it saves or check the presence of a car depending of the project definition
 */
bool system_fnPhotoTakeAndSave_b(void);
/**
 * @brief      Takes photos and check them in different angles to check a bigger range
 */
void system_fnCameraProcess_vo(void);

#ifdef EDGE_IMPULSE_OBJECT_DETECTION_PRESENT
float features[25600];  /* Variable to save the taken photo and check it through EdgeImpulse */
/**
 * @brief      Check Free car parking spaces thanks to the parameter image
 */
uint8_t system_fnCheckFreeSpaces_u8(CamImage Image);
/**
 * @brief      Converts from RGB565 to RGB888
 */
void r565_to_rgb(uint16_t color, uint8_t *r, uint8_t *g, uint8_t *b) ;
int raw_feature_get_data(size_t offset, size_t length, float *out_ptr);
#endif /* EDGE_IMPULSE_OBJECT_DETECTION_PRESENT */

#ifdef MQTT_PRESENT
/**
 * @brief      Check the car park status and send the result through MQTT to AWS IoT
 */
void system_fnMQTTProcess_vo(CamImage Image);
#endif /* MQTT_PRESENT */

/********************************************************************************
-----> Static Functions definition
********************************************************************************/
#ifdef EDGE_IMPULSE_OBJECT_DETECTION_PRESENT
/**
 * @brief      Copy raw feature data in out_ptr
 *             Function called by inference library
 *
 * @param[in]  offset   The offset
 * @param[in]  length   The length
 * @param      out_ptr  The out pointer
 *
 * @return     0
 */
int raw_feature_get_data(size_t offset, size_t length, float *out_ptr) {
    memcpy(out_ptr, features + offset, length * sizeof(float));
    return 0;
}

void r565_to_rgb(uint16_t color, uint8_t *r, uint8_t *g, uint8_t *b) 
{
    *r = (color & 0xF800) >> 8;
    *g = (color & 0x07E0) >> 3;
    *b = (color & 0x1F) << 3;
}
#endif /* EDGE_IMPULSE_OBJECT_DETECTION_PRESENT */

/****************************************************************************
 * setup
 ****************************************************************************/
void setup() 
{
  pinMode(LED0, OUTPUT);                /* LED0 as output. Used to signal taking a photo */
  Serial.begin(BAUDRATE);               /* Open serial communications and wait for port to open */
  while (!Serial)
  {
    ;
  }                                     /* Wait for serial port to connect. Needed for native USB port only */
  system_fnPeripheralsInitialize_vo();  /* Initialize all the peripherals used for the project (RTC, LTE, SDCard, Camera, MQTT, Servo...) */
}

void loop() 
{
#ifdef MQTT_PRESENT
  if(true == (RTC_fnIsAlarmActivated_b() || MQTT_fnIsMQTTRequestActivated_b()))
#else
  if(true == (RTC_fnIsAlarmActivated_b()))
#endif /*MQTT_PRESENT  */
  {
    system_fnCameraProcess_vo();      /* Take and photos in various angles and checks free car spaces and send result to AWS through MQTT protocol.*/

    RTC_fnClearAlarmFlag_vo();        /* Ensure alarm flag is cleared. */
#ifdef MQTT_PRESENT
    MQTT_fnClearMQTTRequestFlag_vo(); /* Ensure MQTT request flag cleared */
#endif /* MQTT_PRESENT */
  }

#ifdef MQTT_PRESENT
  MQTT_fnPoll_vo();                   /* Used to check if there's any data where we are subscribed */
#endif /* MQTT_PRESENT */
    
    delay(1000);
}

void system_fnPeripheralsInitialize_vo(void)
{
#ifdef LTE_PRESENT
  LTE_fnInitialize_b();
#endif /* LTE_PRESENT */

#ifdef SDCARD_PRESENT
   /* Initialize SD */
  SDCARD_fnInitialize_b();
#endif /* SDCARD_PRESENT */

#ifdef CAMERA_PRESENT
  CAMERA_fnInitialize_b();  
#endif /* CAMERA_PRESENT */

#ifdef RTC_PRESENT
  RTC_fnInitialize_vo(); 
  // Set the temporary RTC time
  RtcTime TimeLTE;
#ifdef LTE_PRESENT
  uint32_t LTETime = LTE_fnGetTime_u32();
  TimeLTE.unixtime(LTETime);
#else
  #error "You need to define the time if not obtained by LTE"
#endif /* LTE_PRESENT */
  RTC_fnSetTime_vo(TimeLTE);
#ifdef PROJECT_TAKE_PHOTOS
  RTC_fnSetAlarm_vo(DEFAULT_ALARM_TIME_SEC_U);   
#else  
  /* For this project we do not enable RTC alarm until the user needs/request it. */
#endif /* PROJECT_TAKE_PHOTOS */ 
#endif /* RTC_PRESENT */

#ifdef SERVO_PRESENT
 SERVO_fnInitialize_vo();
#endif /* SERVO_PRESENT */

#ifdef MQTT_PRESENT
  MQTT_fnInitialize_vo(); //Set certifications via SDCard and connect to the broker.
  MQTT_fnReceiveInit_vo();
#endif /* MQTT_PRESENT */
}

 #ifdef EDGE_IMPULSE_OBJECT_DETECTION_PRESENT
uint8_t system_fnCheckFreeSpaces_u8(CamImage Image)
{
    uint8_t u8RetFreeSpaces = 0;
 
    memcpy(features, Image.getImgBuff(), Image.getImgSize());

    uint8_t r, g, b;
    for(int i = 0; i<25600; i++)
    {
      r565_to_rgb(features[i], &r, &g, &b);
      // then convert to out_ptr format
      float pixel_f = (r << 16) + (g << 8) + b;
      features[i] = pixel_f;
    }
    
    ei_printf("Edge Impulse standalone inferencing (Arduino)\n");

    if (sizeof(features) / sizeof(float) != EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE) {
        ei_printf("The size of your 'features' array is not correct. Expected %lu items, but had %lu\n",
            EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, sizeof(features) / sizeof(float));
        delay(1000);
        return;
    }

    ei_impulse_result_t result = { 0 };

    // the features are stored into flash, and we don't want to load everything into RAM
    signal_t features_signal;
    features_signal.total_length = sizeof(features) / sizeof(features[0]);
    features_signal.get_data = &raw_feature_get_data;

    // invoke the impulse
    EI_IMPULSE_ERROR res = run_classifier(&features_signal, &result, false /* debug */);
    ei_printf("run_classifier returned: %d\n", res);

    if (res != 0) return;

    // print the predictions
    ei_printf("Predictions ");
    ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
        result.timing.dsp, result.timing.classification, result.timing.anomaly);
    ei_printf(": \n");
    ei_printf("[");
    for (size_t ix = 0; ix < 10; ix++) {
        ei_printf("%.5f", result.bounding_boxes[ix].value);
        if (ix != EI_CLASSIFIER_LABEL_COUNT - 1) {
            ei_printf(", ");
        }
    }
    ei_printf("]\n");

    // human-readable predictions
    for (size_t ix = 0; ix < 15; ix++) 
    {
        ei_printf("    %s: x = %d y = %d\n", result.bounding_boxes[ix].label, result.bounding_boxes[ix].x, result.bounding_boxes[ix].y);
        if(1 == strcmp("FreeCarPark",result.bounding_boxes[ix].label))
        {
          u8RetFreeSpaces++;
        }
    }

    return(u8RetFreeSpaces);
}
#endif /* EDGE_IMPULSE_OBJECT_DETECTION_PRESENT */

bool system_fnPhotoTakeAndSave_b(void)  
{
  CamImage Image;
  CamErr Error =  CAM_ERR_SUCCESS;

#ifdef PROJECT_TAKE_PHOTOS
  char filename[16] = {0};
  sprintf(filename, "PICT%03d.JPG", system_u32PhotosNumber);  /* Name for the image */
#endif /* PROJECT_TAKE_PHOTOS */
  
  Image = CAMERA_fnTakePicture_CamImage();                    /* Take a photo */

  if(false == CAMERA_fnIsImageAvailable_b(Image))
  {
    Serial.println("Failed to take picture");
  }
  else
  {
    LED0_ON;
#ifdef PROJECT_TAKE_PHOTOS
    system_u32PhotosNumber++;
    Serial.println("Save image");    
    SDCARD_fnSaveFile_vo(filename, Image.getImgBuff() ,Image.getImgSize()); /* Save image in SDCard */
    LED0_OFF;
#endif /* PROJECT_TAKE_PHOTOS */
#ifdef MQTT_PRESENT
    system_fnMQTTProcess_vo(Image);                   /* Process the photo to send the result through MQTT to AWS IoT */
#endif /* MQTT_PRESENT */
  }
  return(true);
}

void system_fnCameraProcess_vo(void)
{
  uint8_t u8ArrayCounter = 0;
  CAMERA_fnStart_b();
  for(u8ArrayCounter=0; u8ArrayCounter<(_NUMBER_OF_ANGLES_U); u8ArrayCounter++)
  {
    SERVO_fnMoveToAngle_vo(a6u8AngleArray[u8ArrayCounter]);
    sleep(1);
    system_fnPhotoTakeAndSave_b();
  }
  CAMERA_fnStop_vo();
}

#ifdef MQTT_PRESENT
void system_fnMQTTProcess_vo(CamImage Image)
{
  StaticJsonDocument<300> jsonDoc;
  String payload = "";
  uint8_t u8NumberFreeSpaces = 0;

  u8NumberFreeSpaces = system_fnCheckFreeSpaces_u8(Image);
  jsonDoc["FreeCarSpaces"] = u8NumberFreeSpaces;
  serializeJson(jsonDoc, payload);
  MQTT_fnSendMessage_vo(payload, MQTT_TOPIC);   //Send MQTT message.
}
#endif /* MQTT_PRESENT */

Credits

Iñaki Nagore

2 projects • 9 followers

Comments

Awards

Judges Choice

Spresense Developer Challenge 2022

Edge Impulse Special Prize

Spresense Developer Challenge 2022

Free Car Parking Monitoring Device

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

PROBLEM DESCRIPTION

THE IDEA

PROOF OF CONCEPT

USED KEY FEATURES FOR THE PROJECT

PROJECT DOCUMENTATION

- Creation of the physical prototype

- Image capture and ML model creation in Edge Impulse

- Free parking spaces detection firmware program

THE CODE

Custom parts and enclosures

Rotor/Static Case for Spresense board with camera

LTE Board Base

Schematics

Schematic

Code

CameraSpresense.cpp

LTESpresense.cpp

MQTTSpresense.cpp

RTCSpresense.cpp

SDCardSpresense.cpp

ServoSpresense.cpp

spresense.h

ProyectoSpresenseParking.ino

Code added to github

Credits

Iñaki Nagore

Comments

Awards

Embed the widget on your own site

Free Car Parking Monitoring Device

Free Car Parking Monitoring Device

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

PROBLEM DESCRIPTION

THE IDEA

PROOF OF CONCEPT

USED KEY FEATURES FOR THE PROJECT

PROJECT DOCUMENTATION

- Creation of the physical prototype

- Image capture and ML model creation in Edge Impulse

- Free parking spaces detection firmware program

THE CODE

Custom parts and enclosures

Rotor/Static Case for Spresense board with camera

LTE Board Base

Schematics

Schematic

Code

CameraSpresense.cpp

LTESpresense.cpp

MQTTSpresense.cpp

RTCSpresense.cpp

SDCardSpresense.cpp

ServoSpresense.cpp

spresense.h

ProyectoSpresenseParking.ino

Code added to github

Credits

Iñaki Nagore

Comments

Awards

Related channels and tags