Artemis is the flagship microprocessor module, developed by Sparkfun that will be integrated into the next generation of efficient development boards. Based on Cortex-M4F with BLE 5.0 capability along with 96MHz peak operation, with as low power as 6uA per MHz (less than 5mW). The Apollo3 chip inside each Artemis module can be powered from a single coin-cell battery. The SparkFun Artemis Module is fully FCC/IC/CE certified. Moreover, it provides support for AI/ML development through TensorFlow Lite.
Sparkfun Artemis ATP(with form-factor of Arduino MEGA 2560)
Hence, utilizing the capabilities of Artemis ATP(All the Pins) board; with similar form-factor to the famous Arduino MEGA2560, I have developed a Smart Health Assistant that would provide the vital signals associated with cardiac health like Heart-Rate measurement, SPo2 and HRV measurement, Single channel ElectroCardioGram measurement along with ambient environment parameters: temperature, CO2 and TVOCs, humidity and associated altitude, that altogether will be augmented with the captured bio-signals to improve the efficiency and efficacy of the entire system.
The device also features WS2812B LED Ring, with gestures almost similar to Amazon's Alexa color-ring, that will be used for notifications, criticality, and sensor-updates to Raspberry Pi 3B+ based server. Moreover, the device utilizes the capabilities of on-board 'Always_ON' microphone, which has been exploited along with the TensorFlow Lite based micro-speech code, that will be used to wake the device from idle state, or to initiate the device for real-time measurements.
Shortnote...
Early detection of abnormal cardiac activity is vital to identify heart problems and avoid sudden cardiac death. It has been proved that people with similar heart conditions almost have similar electrocardiogram (ECG) signals. Arrhythmia detection has been heavily relied on observing morphological features of the ECG signals which are tedious and very time consuming. Automatic detection of arrhythmia is more preferable through the use of classifiers based on machine-learning, that can alltogether run on low consumption MCU.
Sinus Rhythm of normal heart
Brief Analogy:
The developed device integrates ambient environmental measurement based on Sparkfun CCS811/BME280 module while during the measurement (ECG & Spo2:HRV) is performed. I have also augmented the sensor values on the captured biosigs and have trained a classifier in MatLab based on 'MIT-BIH Arrhythmia Database' for predicting the heart condition and show early-signs of any associated cardiac-anomaly.
Building the entire system in one go...
I have initially developed a cardboard enclosure covered in black fabric; for replicating the alexa echo device, I have also designed a similar 3 part geometry in Fusion 360 which can be 3D-printed for giving the finishing looks to the device. The acrylic disc is mounted in the top part of the geometry that would be placed directly above the WS2812B ring with 6-7cm. gap for maximum diffusion. Note: For producing quality DIY diffusers one needs to grind one surface with sandpaper and paint the other surface pitch-black, such that the diffusion occurs only through the circumferential-ring.
Checking the individual sensors with the Artemis ATP
Final Assembly#1; SHA
Preliminary Tests for sensors and sync. with RPi
Each and every sensor is mounted over the top section; for providing the measurement-comfort and sake of the design. Except the ECG one, that exposes the measurement leads from one side. The device has also an optional feature to listen to your favorite music with the help of an external speaker that is connected to RPi.
Each and every sensor is mounted over the top section; for providing the measurement-comfort and sake of the design. Except the ECG one, that exposes the measurement leads from one side. The device has also an optional feature to listen to your favorite music with the help of an external speaker that is connected to RPi.
3 part geometry; Enclosure#2
SHA-Enclosure designed in Fusion360
Using the ESP32...WHAT!!!
Not quite sorry for using the ESP32, as I wasn't able to compile for the FastLED on the Artemis, I have implemented the DMA registers for artemis using adafruit_neopixel, but the end results were'nt satisfactory. The ESP32 is responsible for driving the Alexa based color gestures on our device, through trained 'YES/NO' Mic.-Output; one can also train the artemis for more commands like 'On/Off' by compiling the classifier on GPU assisted system through Colab or Jupyter Notebook.
Speech Classifier demonstration and Alexa based WS2812b render
VCNL4040 used as Proximity based Hot-Switch...
I have used the capability of Sparkfun Proximity sensor, as a hot-switch; whose sole-purpose is to sync. the last measurements to the RPi based Server. The response time for the programmed 'virtual-tap' is quite accurate for this job.
The Magic of uOLED:
Using the Sparkfun MicroOLED breakout with qwiic, various sensor data are shown graphically; to name a few: Screen1 displays CCS_CO2 measurements in ppm, Screen2 displays the Temperature(in Celsius), Altitude(in meters) and Humidity(%), while Screen3 shows the Average Heart-Rate value in beats per minute obtained from MAX30102 sensor, which can be further used to display ppg waveform over the cloud, Much more needs to be done with the OLED (What I think!)
Serious ECG measurement (based on AD8232):
The measured output values were sampled for different ADC resolutions (Artemis), for bringing out the initial 'clean' signal.
Primary ECG Ouput
After threshold has been applied on Artemis Output
AD8232 ECG Pulse Output Waveform
-Man vs Machine : What the physician sees on the ECG record and what the computer interprets are not the same. Modern ECG devices record only leads I and II and, using the equilateral triangle equation of Einthoven, instantaneously calculate the other 4 limb leads. The 6 precordial leads are recorded individually, but all 12 leads are recorded simultaneously. The computer, therefore, can review a full 10 seconds of each lead, whereas the reader, in the standard display, sees only a 2.5-second sample of each of the 12 leads. This enables the computer analysis to recognize beat-to-beat variations in wave form in all leads for the full 10 seconds, data routinely unavailable to the paper reader. In addition, lead simultaneity permits the identification of lead characteristics that are not readily apparent to the reader's eye. The reader still views the ECG as an analog recording, but the device digitizes the incoming analog signal 500 times per second and, for the purpose of identifying pacemaker stimulus artifacts, can even oversample at 1000 times per second. The reader looks at the waves individually and mentally categorizes beat-to-beat differences as important diagnostic findings or variations that are minimal and negligible. The computer composes a representative wave from each of the leads and analyzes this artifact-free “average” or “median” wave rather than analyzing each wave separately as the eye does.
Sensor compensation with HRV and ECG:
There is abundant evidence on the association between outdoor temperature and cardiovascular mortality. However, it is still not clear what is the mechanism underlying this association. Thus, there is increasing interest in exploring the mechanism behind this association, particularly because the extreme weathers will appear frequently under future climate change and the prevalence of cardiovascular diseases has been increasing. The increased heart rate (HR) and blood pressure (BP) are important cardiac function factors for cardiovascular disease and death. Therefore, the mechanism of temperature-related cardiovascular admission/death may be explained partly by the effects of temperature on HR and BP. Both extreme cold and hot temperatures, along with ambient CO2 levels have short-term impacts on heart rate and BP, which modifies the original vital-indication that further leads to errors in automated as well as human analysis.
Classifying the ECG signal: The Real Science..in detail:
Automatic cardiac Arrhythmia detection and classification can play a vital role in the monitoring of patients. ECG signal analysis mainly consists of three phases, ECG signal preprocessing, Feature Extraction and Classification. The Raspberry Pi is used to store the ECG signal in electrical from the serial of the Artemis, where the ECG is classified in to various classes, the EMD breaks down the nonlinear and non-stationary flag into IMFs which are symmetric, band limited and oscillatory in nature. Keeping in mind the end goal is to remove the data from the IMFs, time-space and recurrence area parameters which are registered from IMFs.
Functional Representation of AECGA
Automatic detection of ECG signal can be performed using RR-interval signals. The RR-interval signals are nonlinear and non-stationary in nature. Hence linear methods may not be able to capture the hidden information present in the signal. A new nonlinear method based on empirical mode decomposition (EMD) is proposed to discriminate between abnormal and normal RR-interval signals. The mean frequency parameter using Fourier Bessel series expansion (MFFB) and the two bandwidth parameters namely, amplitude modulation bandwidth (BAM) and frequency modulation bandwidth (BFM) extracted from the intrinsic mode functions (IMFs) obtained from the EMD of RR-interval signals are used to discriminate the two groups. Pre-processing steps performed on the ECG signals to obtain the RR-interval signals. The unwanted noise is filtered by passing the original ECG signals through a low pass filter (cut-off frequency = 15 Hz). Then, the signal is passed through high pass filter (cut-off frequency = 0.3 Hz) to remove baseline wander (Warlar Eswaran, 1991). The notch filter (cut-off frequency = 50 Hz) is employed to eliminate power-line interference noise. The median filter is used to determine the baseline wander of the processed ECG signal and this baseline wander has been subtracted from the processed ECG signal in order to obtain baseline wander free ECG signal. Then, the Pan-Tompkins algorithm (Pan Tompkins, 1985) is used to find the QRS complexes from the processed ECG signal. The RR-intervals are calculated as the time between two consecutive QRS complexes. Empirical mode decomposition (EMD) is an adaptive and data dependent method to decompose a nonlinear and non-stationary signal into a set of Amplitude and Frequency Modulated (AM-FM) signal components, called intrinsic mode functions (IMFs).
MITBIH Dataset sample visualisation in Matlab
Each IMF satisfies following two conditions –
1- Number of extrema (maxima and minima) and number of zero-crossings in a signal should be either equal or differ at most by one.
2-The mean value of two envelopes, one formed by connecting local maxima and other formed by local minima, at any point should be zero. IMFs can be derived from a signal x(t) using iterative process known as sifting process, which can be summarized as follows (Huang et al., 1998):
(I) Extract local maxima and local minima from signal x(t) in ECG signal.
(II) Obtain the envelope Emax(t) by connecting all the maxima and similarly obtain the envelope Emin(t) by connecting all the minima using cubic spline interpolation in EMD.
(III) Determine the average of Emax(t) and Emin(t) as: m(t) =[Emax(t)+Emin(t)]/2
(IV) Extract k(t) from x(t)as: k(t) = x(t)-m(t)
(V) Check whether the k(t) satisfies the above mentioned two basic conditions of IMF or not.
(VI) Repeat steps (I) to (V), for k(t) until it satisfies the conditions of IMF.
EMD based SVM Classifier Window
For our scope of work, The ECG signal is obtained from MIT-BIH data set and classified under MatLab using EMD. ECG signal is loaded and the displayed in physical units, ECG signal is sampled from MITBIH Arrhythmia Database and then performed for real-time recording of ECG from AD8232. ECG signal is pre-processed using Zero phase low pass filter to remove the baseline noise.This low pass filter sets ECG signal to the current baseline and then removes the external distortion.
Automated Classification result obtained in MatLab; (a) NORMAL, (b) BRADYCARDIA detection.
Connecting Everything to RPi through VNC...
Connecting Everything to RPi through VNC...
According to the circuit diagram of the entire system, both ESP32 and Artemis are plugged into the USB ports of RPi, which is further configured with VNC that has additional support for Android Application, for storing the data logs or simply stream over local WiFi hotspot. Serial data is acquired into RPi through the Sync. feature of the device.
RPi 3B+ streaming sensor data
Raspberry Pi 3B+; Sensor Readings streamed through VNC on Android
Conclusion
The Smart Health Assistant, powered by Artemis has potential applications both on professional and hobbyist scale. I have tried to depict the similarity of the device with Alexa; where the former one can be made much more powerful, while the latter is just an infotainment system. In the future version, I would try to offload much more task to Artemis (will eliminate ESP32, and discover for chances, if.. to eliminate RPi).
A 3 part design; consisting of top sensor mounting shell, bottom controller boards mounting, and the acrylic round diffuser for ws2812b LED ring, this design can be easily 3D printed, and covered with black poly fabric, to give unique look and feel of the product.
#include <Wire.h>#include "MAX30105.h"#include "heartRate.h"MAX30105 particleSensor;#include <SFE_MicroOLED.h> // Include the SFE_MicroOLED library#define PIN_RESET 9 #define DC_JUMPER 1 MicroOLED oled(PIN_RESET, DC_JUMPER);#define blinkPin LED_BUILTIN#include "SparkFun_VCNL4040_Arduino_Library.h"VCNL4040 proximitySensor;const byte RATE_SIZE=4; //Increase this for more averaging. 4 is good.
byte rates[RATE_SIZE]; //Array of heart rates
byte rateSpot=0;long lastBeat=0; //Time at which the last beat occurred
float beatsPerMinute;int beatAvg;// Undefine predefined Arduino max/min defs so they don't conflict with std methods#if defined(min)#undef min#endif#if defined(max)#undef max#endif#include <TensorFlowLite.h>#include "main_functions.h"#include "audio_provider.h"#include "command_responder.h"#include "feature_provider.h"#include "micro_features_micro_model_settings.h"#include "micro_features_tiny_conv_micro_features_model_data.h"#include "recognize_commands.h"#include "tensorflow/lite/experimental/micro/kernels/micro_ops.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"#include "tensorflow/lite/experimental/micro/micro_interpreter.h"#include "tensorflow/lite/experimental/micro/micro_mutable_op_resolver.h"#include "tensorflow/lite/schema/schema_generated.h"#include "tensorflow/lite/version.h"// Globals, used for compatibility with Arduino-style sketches.namespace {tflite::ErrorReporter* error_reporter = nullptr;const tflite::Model* model = nullptr;tflite::MicroInterpreter* interpreter = nullptr;TfLiteTensor* model_input = nullptr;FeatureProvider* feature_provider = nullptr;RecognizeCommands* recognizer = nullptr;int32_t previous_time = 0;// Create an area of memory to use for input, output, and intermediate arrays.// The size of this will depend on the model you're using, and may need to be
// determined by experimentation.
constexpr int kTensorArenaSize=10 * 1024;uint8_t tensor_arena[kTensorArenaSize];} // namespace
// The name of this function is important for Arduino compatibility.
void setup(){ // Set up logging. Google style is to avoid globals or statics because of
// lifetime uncertainty, but since this has a trivial destructor it's okay. // NOLINTNEXTLINE(runtime-global-variables) static tflite::MicroErrorReporter micro_error_reporter; error_reporter = µ_error_reporter; // Map the model into a usable data structure. This doesn't involve any
// copying or parsing, it's a very lightweight operation. model = tflite::GetModel(g_tiny_conv_micro_features_model_data); if (model->version() != TFLITE_SCHEMA_VERSION) { error_reporter->Report( "Model provided is schema version %d not equal " "to supported version %d.", model->version(), TFLITE_SCHEMA_VERSION); return; } // Pull in only the operation implementations we need. // This relies on a complete list of all the ops needed by this graph. // An easier approach is to just use the AllOpsResolver, but this will // incur some penalty in code space for op implementations that are not // needed by this graph. // // tflite::ops::micro::AllOpsResolver resolver; // NOLINTNEXTLINE(runtime-global-variables) static tflite::MicroMutableOpResolver micro_mutable_op_resolver; micro_mutable_op_resolver.AddBuiltin( tflite::BuiltinOperator_DEPTHWISE_CONV_2D, tflite::ops::micro::Register_DEPTHWISE_CONV_2D()); micro_mutable_op_resolver.AddBuiltin( tflite::BuiltinOperator_FULLY_CONNECTED, tflite::ops::micro::Register_FULLY_CONNECTED()); micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX, tflite::ops::micro::Register_SOFTMAX()); // Build an interpreter to run the model with. static tflite::MicroInterpreter static_interpreter( model, micro_mutable_op_resolver, tensor_arena, kTensorArenaSize, error_reporter); interpreter = &static_interpreter; // Allocate memory from the tensor_arena for the model's tensors.
TfLiteStatus allocate_status= interpreter->AllocateTensors();if(allocate_status != kTfLiteOk){ error_reporter->Report("AllocateTensors() failed");return;} // Get information about the memory area to use for the model's input. model_input = interpreter->input(0); if ((model_input->dims->size != 4) || (model_input->dims->data[0] != 1) || (model_input->dims->data[1] != kFeatureSliceCount) || (model_input->dims->data[2] != kFeatureSliceSize) || (model_input->type != kTfLiteUInt8)) { error_reporter->Report("Bad input tensor parameters in model"); return; } // Prepare to access the audio spectrograms from a microphone or other source // that will provide the inputs to the neural network. // NOLINTNEXTLINE(runtime-global-variables) static FeatureProvider static_feature_provider(kFeatureElementCount, model_input->data.uint8); feature_provider = &static_feature_provider; static RecognizeCommands static_recognizer(error_reporter); recognizer = &static_recognizer; previous_time = 0; Serial.begin(9600); if (!particleSensor.begin(Wire, I2C_SPEED_FAST)) //Use default I2C port, 400kHz speed { Serial.println("MAX30105 was not found. Please check wiring/power. "); while (1); } particleSensor.setup(); //Configure sensor with default settings particleSensor.setPulseAmplitudeRed(0x0A); //Turn Red LED to low to indicate sensor is running particleSensor.setPulseAmplitudeGreen(0); //Turn off Green LED if (proximitySensor.begin() == false) { Serial.println("Device not found. Please check wiring."); while (1); //Freeze! } pinMode(blinkPin, OUTPUT); Wire.begin(); oled.begin(); // Initialize the OLED oled.clear(ALL); // Clear the display's internal memory
oled.display(); // Display what's in the buffer (splashscreen) delay(1000); // Delay 1000 ms oled.clear(PAGE); // Clear the buffer.}// The name of this function is important for Arduino compatibility.void loop() { // Fetch the spectrogram for the current time. const int32_t current_time = LatestAudioTimestamp(); int how_many_new_slices = 0; TfLiteStatus feature_status = feature_provider->PopulateFeatureData( error_reporter, previous_time, current_time, &how_many_new_slices); if (feature_status != kTfLiteOk) { error_reporter->Report("Feature generation failed"); return; } previous_time = current_time; // If no new audio samples have been received since last time, don't bother
// running the network model.
if(how_many_new_slices==0){return;} // Run the model on the spectrogram input and make sure it succeeds.
TfLiteStatus invoke_status= interpreter->Invoke();if(invoke_status != kTfLiteOk){ error_reporter->Report("Invoke failed");return;} // Obtain a pointer to the output tensor
TfLiteTensor* output= interpreter->output(0); // Determine whether a command was recognized based on the output of inference
const char* found_command= nullptr; uint8_t score=0; bool is_new_command= false; TfLiteStatus process_status= recognizer->ProcessLatestResults( output, current_time, &found_command, &score, &is_new_command);if(process_status != kTfLiteOk){ error_reporter->Report("RecognizeCommands::ProcessLatestResults() failed");return;} // Do something based on the recognized command. The default implementation
// just prints to the error console, but you should replace this with your
// own functionfor a real application.
RespondToCommand(error_reporter, current_time, found_command, score,
is_new_command);{ long irValue= particleSensor.getIR();if(checkForBeat(irValue)==true){ //We sensed a beat!
long delta= millis() - lastBeat;lastBeat= millis();beatsPerMinute=60 / (delta / 1000.0);if(beatsPerMinute < 255&& beatsPerMinute > 20){ rates[rateSpot++]=(byte)beatsPerMinute; //Store this reading in the array
rateSpot %= RATE_SIZE; //Wrap variable
//Take average of readings
beatAvg=0;for(byte x=0; x < RATE_SIZE ; x++)beatAvg+= rates[x]; beatAvg /= RATE_SIZE;}}if(irValue > 60000){ //Serial.print("IR="); //Serial.print(irValue); Serial.print(" BPM="); Serial.print(beatsPerMinute); Serial.print(", Avg BPM="); Serial.print(beatAvg); Serial.println();} unsigned int proxValue= proximitySensor.getProximity();if(proxValue>14000){ Serial.println(""); Serial.println("Tap Activated"); Serial.print("...Sending the accumulated Sensor data to Local-Hub"); //Serial.print(proxValue); Serial.println(); delay(200); digitalWrite(blinkPin, HIGH); delay(150); digitalWrite(blinkPin, LOW); delay(400); digitalWrite(blinkPin, HIGH); delay(150); digitalWrite(blinkPin, LOW); delay(400); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240);}int middleZ=(oled.getLCDWidth() / 2)-14; int middleX= oled.getLCDWidth() / 2; int middleY= oled.getLCDHeight() / 2; oled.clear(PAGE); oled.setCursor(middleZ - (oled.getFontWidth() * (2)+3),
middleY - (oled.getFontHeight() / 2)-17);oled.print("HEART-RATE");// oled.setCursor(middleZ - (oled.getFontWidth() * (2)-6),
// middleY - (oled.getFontHeight() / 2)-8);//oled.print("Be Heart SMART!"); oled.setCursor(middleX - (oled.getFontWidth() * (2)+1),
middleY - (oled.getFontHeight() / 2)+17); oled.print(beatAvg); oled.print("bpm"); //delay(7000); oled.display(); oled.clear(PAGE);}}
Main Script#2
Arduino
arduino_audio_provider.cpp
// AM_BSP_NUM_LEDS : LED initialization and management per EVB target (# of// LEDs defined in EVB BSP)#include "audio_provider.h"#include <limits>// These are headers from Ambiq's Apollo3 SDK.#include "am_bsp.h" // NOLINT#include "am_mcu_apollo.h" // NOLINT#include "am_util.h" // NOLINT#include "micro_features_micro_model_settings.h"namespace {// These are the raw buffers that are filled by the PDM during DMAconstexpr int kPdmNumSlots = 1;constexpr int kPdmSamplesPerSlot = 256;constexpr int kPdmSampleBufferSize = (kPdmNumSlots * kPdmSamplesPerSlot);uint32_t g_ui32PDMSampleBuffer0[kPdmSampleBufferSize];uint32_t g_ui32PDMSampleBuffer1[kPdmSampleBufferSize];uint32_t g_PowerOff = 0;// Controls the double buffering between the two DMA buffers.int g_dma_destination_index = 0;// PDM Device Handle.static void* g_pdm_handle;// PDM DMA error flag.volatile bool g_pdm_dma_error;// So the interrupt can use the passed-in error handler to report issues.tflite::ErrorReporter* g_pdm_dma_error_reporter = nullptr;// Holds a longer history of audio samples in a ring buffer.constexpr int kAudioCaptureBufferSize = 16000;int16_t g_audio_capture_buffer[kAudioCaptureBufferSize] = {};int g_audio_capture_buffer_start = 0;int64_t g_total_samples_captured = 0;int32_t g_latest_audio_timestamp = 0;// Copy of audio samples returned to the caller.int16_t g_audio_output_buffer[kMaxAudioSampleSize];bool g_is_audio_initialized = false;//*****************************************************************************//// Globals////*****************************************************************************// ARPIT TODO : Implement low power configurationvoid custom_am_bsp_low_power_init(void) { // Initialize for low power in the power control block // am_hal_pwrctrl_low_power_init(); // Run the RTC off the LFRC. // am_hal_rtc_osc_select(AM_HAL_RTC_OSC_LFRC); // Stop the XTAL. // am_hal_clkgen_control(AM_HAL_CLKGEN_CONTROL_XTAL_STOP, 0); // Disable the RTC. // am_hal_rtc_osc_disable();} // am_bsp_low_power_init()// Make sure the CPU is running as fast as possible.void enable_burst_mode(tflite::ErrorReporter* error_reporter) { am_hal_burst_avail_e eBurstModeAvailable; am_hal_burst_mode_e eBurstMode; // Check that the Burst Feature is available. if (AM_HAL_STATUS_SUCCESS == am_hal_burst_mode_initialize(&eBurstModeAvailable)) { if (AM_HAL_BURST_AVAIL == eBurstModeAvailable) { error_reporter->Report("Apollo3 Burst Mode is Available\n"); } else { error_reporter->Report("Apollo3 Burst Mode is Not Available\n"); } } else { error_reporter->Report("Failed to Initialize for Burst Mode operation\n"); } // Put the MCU into "Burst" mode. if (AM_HAL_STATUS_SUCCESS == am_hal_burst_mode_enable(&eBurstMode)) { if (AM_HAL_BURST_MODE == eBurstMode) { error_reporter->Report("Apollo3 operating in Burst Mode (96MHz)\n"); } } else { error_reporter->Report("Failed to Enable Burst Mode operation\n"); }}} // namespace//*****************************************************************************// PDM configuration information.//*****************************************************************************am_hal_pdm_config_t g_sPdmConfig = { .eClkDivider = AM_HAL_PDM_MCLKDIV_1, .eLeftGain = AM_HAL_PDM_GAIN_P165DB, .eRightGain = AM_HAL_PDM_GAIN_P165DB, .ui32DecimationRate = 48, // OSR = 1500/16 = 96 = 2*SINCRATE --> SINC_RATE = 48 .bHighPassEnable = 1, .ui32HighPassCutoff = 0x2, .ePDMClkSpeed = AM_HAL_PDM_CLK_1_5MHZ, .bInvertI2SBCLK = 0, .ePDMClkSource = AM_HAL_PDM_INTERNAL_CLK, .bPDMSampleDelay = 0, .bDataPacking = 0, .ePCMChannels = AM_BSP_PDM_CHANNEL, .ui32GainChangeDelay = 1, .bI2SEnable = 0, .bSoftMute = 0, .bLRSwap = 0,};//*****************************************************************************// PDM initialization.//*****************************************************************************extern "C" void pdm_init(void) { // // Initialize, power-up, and configure the PDM. // am_hal_pdm_initialize(0, &g_pdm_handle); am_hal_pdm_power_control(g_pdm_handle, AM_HAL_PDM_POWER_ON, false); am_hal_pdm_configure(g_pdm_handle, &g_sPdmConfig); // // Configure the necessary pins. // am_hal_gpio_pinconfig(AM_BSP_PDM_CLOCK, g_AM_BSP_PDM_CLOCK); am_hal_gpio_pinconfig(AM_BSP_PDM_DATA, g_AM_BSP_PDM_DATA); // // Configure and enable PDM interrupts (set up to trigger on DMA // completion). // am_hal_pdm_interrupt_enable(g_pdm_handle, ( AM_HAL_PDM_INT_DERR | AM_HAL_PDM_INT_DCMP | AM_HAL_PDM_INT_UNDFL | AM_HAL_PDM_INT_OVF)); NVIC_EnableIRQ(PDM_IRQn); // Enable PDM am_hal_pdm_enable(g_pdm_handle);}// Start the DMA fetch of PDM samples.void pdm_start_dma(tflite::ErrorReporter* error_reporter) { // Configure DMA and target address. am_hal_pdm_transfer_t sTransfer; if (g_dma_destination_index == 0) { sTransfer.ui32TargetAddr = (uint32_t)g_ui32PDMSampleBuffer0; } else { sTransfer.ui32TargetAddr = (uint32_t)g_ui32PDMSampleBuffer1; } sTransfer.ui32TotalCount = 4 * kPdmSampleBufferSize; // PDM DMA count is in Bytes // Start the data transfer. if (AM_HAL_STATUS_SUCCESS != am_hal_pdm_dma_start(g_pdm_handle, &sTransfer)) { error_reporter->Report("Error - configuring PDM DMA failed."); } // Reset the PDM DMA flags. g_pdm_dma_error = false;}// Interrupt handler for the PDM.extern "C" void am_pdm_isr(void) { uint32_t ui32IntMask; // Read the interrupt status. if (AM_HAL_STATUS_SUCCESS != am_hal_pdm_interrupt_status_get(g_pdm_handle, &ui32IntMask, false)) { g_pdm_dma_error_reporter->Report("Error reading PDM0 interrupt status."); } // Clear the PDM interrupt. if (AM_HAL_STATUS_SUCCESS != am_hal_pdm_interrupt_clear(g_pdm_handle, ui32IntMask)) { g_pdm_dma_error_reporter->Report("Error clearing PDM interrupt status."); }#if USE_DEBUG_GPIO // DEBUG : GPIO flag polling. am_hal_gpio_state_write(31, AM_HAL_GPIO_OUTPUT_SET); // Slot1 AN pin#endif // If we got a DMA complete, set the flag. if (ui32IntMask & AM_HAL_PDM_INT_OVF) { am_util_stdio_printf("\n%s\n", "\nPDM ISR OVF."); } if (ui32IntMask & AM_HAL_PDM_INT_UNDFL) { am_util_stdio_printf("\n%s\n", "\nPDM ISR UNDLF."); } if (ui32IntMask & AM_HAL_PDM_INT_DCMP) { uint32_t* source_buffer; if (g_dma_destination_index == 0) { source_buffer = g_ui32PDMSampleBuffer0; g_dma_destination_index = 1; } else { source_buffer = g_ui32PDMSampleBuffer1; g_dma_destination_index = 0; } pdm_start_dma(g_pdm_dma_error_reporter); uint32_t slotCount = 0; for (uint32_t indi = 0; indi < kPdmSampleBufferSize; indi++) { g_audio_capture_buffer[g_audio_capture_buffer_start] = source_buffer[indi]; g_audio_capture_buffer_start = (g_audio_capture_buffer_start + 1) % kAudioCaptureBufferSize; slotCount++; } g_total_samples_captured += slotCount; g_latest_audio_timestamp = (g_total_samples_captured / (kAudioSampleFrequency / 1000)); } // If we got a DMA error, set the flag. if (ui32IntMask & AM_HAL_PDM_INT_DERR) { g_pdm_dma_error = true; }#if USE_DEBUG_GPIO // DEBUG : GPIO flag polling. am_hal_gpio_state_write(31, AM_HAL_GPIO_OUTPUT_CLEAR); // Slot1 AN pin#endif}TfLiteStatus InitAudioRecording(tflite::ErrorReporter* error_reporter) { // Set the clock frequency. if (AM_HAL_STATUS_SUCCESS != am_hal_clkgen_control(AM_HAL_CLKGEN_CONTROL_SYSCLK_MAX, 0)) { error_reporter->Report("Error - configuring the system clock failed."); return kTfLiteError; } // Individually select elements of am_bsp_low_power_init custom_am_bsp_low_power_init(); // Set the default cache configuration and enable it. if (AM_HAL_STATUS_SUCCESS != am_hal_cachectrl_config(&am_hal_cachectrl_defaults)) { error_reporter->Report("Error - configuring the system cache failed."); return kTfLiteError; } if (AM_HAL_STATUS_SUCCESS != am_hal_cachectrl_enable()) { error_reporter->Report("Error - enabling the system cache failed."); return kTfLiteError; } // Configure Flash wait states. CACHECTRL->FLASHCFG_b.RD_WAIT = 1; // Default is 3 CACHECTRL->FLASHCFG_b.SEDELAY = 6; // Default is 7 CACHECTRL->FLASHCFG_b.LPM_RD_WAIT = 5; // Default is 8 // Enable cache sleep states. uint32_t ui32LPMMode = CACHECTRL_FLASHCFG_LPMMODE_STANDBY; if (am_hal_cachectrl_control(AM_HAL_CACHECTRL_CONTROL_LPMMODE_SET, &ui32LPMMode)) { error_reporter->Report("Error - enabling cache sleep state failed."); } // Enable Instruction & Data pre-fetching. MCUCTRL->SRAMMODE_b.DPREFETCH = 1; MCUCTRL->SRAMMODE_b.DPREFETCH_CACHE = 1; MCUCTRL->SRAMMODE_b.IPREFETCH = 1; MCUCTRL->SRAMMODE_b.IPREFETCH_CACHE = 1; // Enable the floating point module, and configure the core for lazy stacking. am_hal_sysctrl_fpu_enable(); am_hal_sysctrl_fpu_stacking_enable(true); error_reporter->Report("FPU Enabled."); // Ensure the CPU is running as fast as possible. // enable_burst_mode(error_reporter); // Configure, turn on PDM pdm_init(); am_hal_interrupt_master_enable(); am_hal_pdm_fifo_flush(g_pdm_handle); // Trigger the PDM DMA for the first time manually. g_pdm_dma_error_reporter = error_reporter; pdm_start_dma(g_pdm_dma_error_reporter); error_reporter->Report("\nPDM DMA Threshold = %d", PDMn(0)->FIFOTHR); return kTfLiteOk;}TfLiteStatus GetAudioSamples(tflite::ErrorReporter* error_reporter, int start_ms, int duration_ms, int* audio_samples_size, int16_t** audio_samples) { if (!g_is_audio_initialized) { TfLiteStatus init_status = InitAudioRecording(error_reporter); if (init_status != kTfLiteOk) { return init_status; } g_is_audio_initialized = true; } // This should only be called when the main thread notices that the latest // audio sample data timestamp has changed, so that there's new data in the
// capture ring buffer. The ring buffer will eventually wrap around and
// overwrite the data, but the assumption is that the main thread is checking
// often enough and the buffer is large enough that this call will be made
// before that happens.
const int start_offset=(start_ms < 0) ? 0 : start_ms * (kAudioSampleFrequency / 1000); const int duration_sample_count= duration_ms * (kAudioSampleFrequency / 1000);for(int i=0; i < duration_sample_count; ++i){ const int capture_index=(start_offset + i) % kAudioCaptureBufferSize; g_audio_output_buffer[i]= g_audio_capture_buffer[capture_index];} *audio_samples_size = kMaxAudioSampleSize; *audio_samples = g_audio_output_buffer;return kTfLiteOk;}int32_t LatestAudioTimestamp(){return g_latest_audio_timestamp;}
Main Script#3
Arduino
arduino_command_responder.cpp
#include "command_responder.h"#include "am_bsp.h" // NOLINT// This implementation will light up the LEDs on the board in response to
// different commands.
void RespondToCommand(tflite::ErrorReporter* error_reporter,
int32_t current_time, const char* found_command,
uint8_t score, bool is_new_command){ static bool is_initialized= false;if(!is_initialized){ // Setup LED's as outputs // EAC am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_RED, g_AM_HAL_GPIO_OUTPUT_12); am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_BLUE, g_AM_HAL_GPIO_OUTPUT_12); // EAC am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_GREEN, g_AM_HAL_GPIO_OUTPUT_12); // EAC am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_YELLOW, g_AM_HAL_GPIO_OUTPUT_12); is_initialized = true; } static int count = 0; // Toggle the blue LED every time an inference is performed. //++count; //if (count & 1) { // am_hal_gpio_output_set(AM_BSP_GPIO_LED_BLUE); //} else { // am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE); //} // Turn on the yellow LED if 'yes' was heard. // EAC am_hal_gpio_output_clear(AM_BSP_GPIO_LED_RED); // EAC am_hal_gpio_output_clear(AM_BSP_GPIO_LED_YELLOW); // EAC am_hal_gpio_output_clear(AM_BSP_GPIO_LED_GREEN); if (is_new_command) { error_reporter->Report("Heard %s (%d) @%dms", found_command, score, current_time); if (found_command[0] == 'y') { error_reporter->Report("\nYES"); // EAC am_hal_gpio_output_set(AM_BSP_GPIO_LED_YELLOW); am_hal_gpio_output_set(AM_BSP_GPIO_LED_BLUE); //delay(3000); } if (found_command[0] == 'n') { error_reporter->Report("\nNO"); // EAC am_hal_gpio_output_set(AM_BSP_GPIO_LED_RED); am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE); } if (found_command[0] == 'u'){ error_reporter->Report("\nUNKNOWN"); // EAC am_hal_gpio_output_set(AM_BSP_GPIO_LED_GREEN); //am_hal_gpio_output_set(AM_BSP_GPIO_LED_BLUE); //am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE);}}}
Main Script#4
Arduino
arduino_main.cpp
#include "main_functions.h"// Arduino automatically calls the setup() and loop() functions in a sketch, so
// where other systems need their own main routine in this file, it can be left
// empty.
Main Script#5
Arduino
audio_provider.h
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_#include "tensorflow/lite/c/c_api_internal.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"// This is an abstraction around an audio source like a microphone, and is
// expected to return16-bit PCM sample data for a given point in time. The
// sample data itself should be used as quickly as possible by the caller, since
// to allow memory optimizations there are no guarantees that the samples won't// be overwritten by new data in the future. In practice, implementations should// ensure that there's a reasonable time allowed for clients to access the data
// before any reuse.
// The reference implementation can have no platform-specific dependencies, so
// it just returns an array filled with zeros. For real applications, you should
// ensure there's a specialized implementation that accesses hardware APIs.TfLiteStatus GetAudioSamples(tflite::ErrorReporter* error_reporter, int start_ms, int duration_ms, int* audio_samples_size, int16_t** audio_samples);// Returns the time that audio data was last captured in milliseconds. There's
// no contract about what time zero represents, the accuracy, or the granularity
// of the result. Subsequent calls will generally not return a lower value, but
// even that's not guaranteed if there's an overflow wraparound.
// The reference implementation of this function just returns a constantly
// incrementing value for each call, since it would need a non-portable platform
// call to access time information. For real applications, you'll need to write
// your own platform-specific implementation.
int32_t LatestAudioTimestamp();#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_
Main Script#6
Arduino
command_responder.h
// Provides an interface to take an action based on an audio command.
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_#include "tensorflow/lite/c/c_api_internal.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"// Called every time the results of an audio recognition run are available. The
// human-readable name of any recognized command is in the `found_command`// argument, `score` has the numerical confidence, and `is_new_command` is set// if the previous command was different to this one.
void RespondToCommand(tflite::ErrorReporter* error_reporter,
int32_t current_time, const char* found_command,
uint8_t score, bool is_new_command);#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_
Main Script#7
Arduino
feature_provider.cpp
#include "feature_provider.h"#include "audio_provider.h"#include "micro_features_micro_features_generator.h"#include "micro_features_micro_model_settings.h"FeatureProvider::FeatureProvider(int feature_size, uint8_t* feature_data) : feature_size_(feature_size),
feature_data_(feature_data),
is_first_run_(true){ // Initialize the feature data to default values.
for(int n=0; n < feature_size_; ++n){ feature_data_[n]=0;}}FeatureProvider::~FeatureProvider(){}TfLiteStatus FeatureProvider::PopulateFeatureData( tflite::ErrorReporter* error_reporter, int32_t last_time_in_ms,
int32_t time_in_ms, int* how_many_new_slices){if(feature_size_ != kFeatureElementCount){ error_reporter->Report("Requested feature_data_ size %d doesn't match %d",
feature_size_, kFeatureElementCount);return kTfLiteError;} // Quantize the time into steps as long as each window stride, so we can
// figure out which audio data we need to fetch.
const int last_step=(last_time_in_ms / kFeatureSliceStrideMs); const int current_step=(time_in_ms / kFeatureSliceStrideMs); int slices_needed= current_step - last_step; // If this is the first call, make sure we don't use any cached information.
if(is_first_run_){ TfLiteStatus init_status= InitializeMicroFeatures(error_reporter);if(init_status != kTfLiteOk){return init_status;}is_first_run_= false;slices_needed= kFeatureSliceCount;}if(slices_needed > kFeatureSliceCount){slices_needed= kFeatureSliceCount;} *how_many_new_slices = slices_needed; const int slices_to_keep= kFeatureSliceCount - slices_needed; const int slices_to_drop= kFeatureSliceCount - slices_to_keep; // If we can avoid recalculating some slices, just move the existing data
// up in the spectrogram, to perform something like this:
// last time= 80ms current time= 120ms
// +-----------+ +-----------+
// | data@20ms | --> | data@60ms | // +-----------+ -- +-----------+
// | data@40ms | -- --> | data@80ms | // +-----------+ -- -- +-----------+
// | data@60ms | -- -- | <empty> | // +-----------+ -- +-----------+
// | data@80ms | -- | <empty> | // +-----------+ +-----------+
if(slices_to_keep > 0){for(int dest_slice=0; dest_slice < slices_to_keep; ++dest_slice){ uint8_t* dest_slice_data= feature_data_ + (dest_slice * kFeatureSliceSize); const int src_slice= dest_slice + slices_to_drop; const uint8_t* src_slice_data= feature_data_ + (src_slice * kFeatureSliceSize);for(int i=0; i < kFeatureSliceSize; ++i){ dest_slice_data[i]= src_slice_data[i];}}} // Any slices that need to be filled in with feature data have their
// appropriate audio data pulled, and features calculated for that slice.
if(slices_needed > 0){for(int new_slice= slices_to_keep; new_slice < kFeatureSliceCount; ++new_slice){ const int new_step=(current_step - kFeatureSliceCount + 1) + new_slice; const int32_t slice_start_ms=(new_step * kFeatureSliceStrideMs); int16_t* audio_samples= nullptr; int audio_samples_size=0; // TODO(petewarden): Fix bug that leads to non-zero slice_start_ms
GetAudioSamples(error_reporter, (slice_start_ms > 0 ? slice_start_ms : 0),
kFeatureSliceDurationMs, &audio_samples_size,
&audio_samples);if(audio_samples_size < kMaxAudioSampleSize){ error_reporter->Report("Audio data size %d too small, want %d",
audio_samples_size, kMaxAudioSampleSize);return kTfLiteError;} uint8_t* new_slice_data= feature_data_ + (new_slice * kFeatureSliceSize); size_t num_samples_read; TfLiteStatus generate_status= GenerateMicroFeatures( error_reporter, audio_samples, audio_samples_size, kFeatureSliceSize,
new_slice_data, &num_samples_read);if(generate_status != kTfLiteOk){return generate_status;}}}return kTfLiteOk;}
Main Script#8
Arduino
feature_provider.h
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_#include "tensorflow/lite/c/c_api_internal.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"// Binds itself to an area of memory intended to hold the input features for an
// audio-recognition neural network model, and fills that data area with the
// features representing the current audio input, for example from a microphone.
// The audio features themselves are a two-dimensional array, made up of
// horizontal slices representing the frequencies at one point in time, stacked
// on top of each other to form a spectrogram showing how those frequencies
// changed over time.
class FeatureProvider { public:
// Create the provider, and bind it to an area of memory. This memory should
// remain accessible for the lifetime of the provider object, since subsequent
// calls will fill it with feature data. The provider does no memory
// management of this data.
FeatureProvider(int feature_size, uint8_t* feature_data); ~FeatureProvider(); // Fills the feature data with information from audio inputs, and returns how
// many feature slices were updated.
TfLiteStatus PopulateFeatureData(tflite::ErrorReporter* error_reporter,
int32_t last_time_in_ms, int32_t time_in_ms,
int* how_many_new_slices); private:
int feature_size_; uint8_t* feature_data_; // Make sure we don't try to use cached information if this is the first call
// into the provider.
bool is_first_run_;};#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_
Main Script#9
Arduino
main_functions.h
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_// Initializes all data needed for the example. The name is important, and needs
// to be setup()for Arduino compatibility.
void setup();// Runs one iteration of data gathering and inference. This should be called
// repeatedly from the application code. The name needs to be loop()for Arduino
// compatibility.
void loop();#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_
Main Script#10
Arduino
micro_features_micro_features_generator.cpp
#include "micro_features_micro_features_generator.h"#include <cmath>#include <cstring>#include "micro_features_micro_model_settings.h"#include "tensorflow/lite/experimental/microfrontend/lib/frontend.h"#include "tensorflow/lite/experimental/microfrontend/lib/frontend_util.h"// Configure FFT to output 16 bit fixed point.
#define FIXED_POINT 16namespace {FrontendState g_micro_features_state;bool g_is_first_time= true;} // namespace
TfLiteStatus InitializeMicroFeatures(tflite::ErrorReporter* error_reporter){ FrontendConfig config; config.window.size_ms = kFeatureSliceDurationMs; config.window.step_size_ms = kFeatureSliceStrideMs; config.noise_reduction.smoothing_bits =10; config.filterbank.num_channels = kFeatureSliceSize; config.filterbank.lower_band_limit =125.0; config.filterbank.upper_band_limit =7500.0; config.noise_reduction.smoothing_bits =10; config.noise_reduction.even_smoothing =0.025; config.noise_reduction.odd_smoothing =0.06; config.noise_reduction.min_signal_remaining =0.05; config.pcan_gain_control.enable_pcan =1; config.pcan_gain_control.strength =0.95; config.pcan_gain_control.offset =80.0; config.pcan_gain_control.gain_bits =21; config.log_scale.enable_log =1; config.log_scale.scale_shift =6;if(!FrontendPopulateState(&config, &g_micro_features_state,
kAudioSampleFrequency)){ error_reporter->Report("FrontendPopulateState() failed");return kTfLiteError;}g_is_first_time= true;return kTfLiteOk;}// This is not exposed in any header, and is only used for testing, to ensure
// that the state is correctly set up before generating results.
void SetMicroFeaturesNoiseEstimates(const uint32_t* estimate_presets){for(int i=0; i < g_micro_features_state.filterbank.num_channels; ++i){ g_micro_features_state.noise_reduction.estimate[i]= estimate_presets[i];}}TfLiteStatus GenerateMicroFeatures(tflite::ErrorReporter* error_reporter,
const int16_t* input, int input_size,
int output_size, uint8_t* output,
size_t* num_samples_read){ const int16_t* frontend_input;if(g_is_first_time){frontend_input= input;g_is_first_time= false;}else{frontend_input= input + 160;} FrontendOutput frontend_output= FrontendProcessSamples(&g_micro_features_state, frontend_input, input_size, num_samples_read);for(int i=0; i < frontend_output.size; ++i){ // These scaling values are derived from those used in input_data.py in the
// training pipeline.
constexpr int32_t value_scale=(10 * 255); constexpr int32_t value_div=(256 * 26); int32_t value=((frontend_output.values[i] * value_scale) + (value_div / 2)) /
value_div;if(value < 0){value=0;}if(value > 255){value=255;} output[i]= value;}return kTfLiteOk;}
Main Script#11
Arduino
micro_features_micro_features_generator.h
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_#include "tensorflow/lite/c/c_api_internal.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"// Sets up any resources needed for the feature generation pipeline.
TfLiteStatus InitializeMicroFeatures(tflite::ErrorReporter* error_reporter);// Converts audio sample data into a more compact form that's appropriate for// feeding into a neural network.
TfLiteStatus GenerateMicroFeatures(tflite::ErrorReporter* error_reporter,
const int16_t* input, int input_size,
int output_size, uint8_t* output,
size_t* num_samples_read);#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_// Keeping these as constant expressions allow us to allocate fixed-sized arrays
// on the stack for our working memory.
// The size of the input time series data we pass to the FFT to produce the
// frequency information. This has to be a power of two, and since we're dealing// with 30ms of 16KHz inputs, which means 480 samples, this is the next value.constexpr int kMaxAudioSampleSize = 512;constexpr int kAudioSampleFrequency = 16000;// All of these values are derived from the values used during model training,// if you change your model you'll need to update these constants.
constexpr int kFeatureSliceSize=40;constexpr int kFeatureSliceCount=49;constexpr int kFeatureElementCount=(kFeatureSliceSize * kFeatureSliceCount);constexpr int kFeatureSliceStrideMs=20;constexpr int kFeatureSliceDurationMs=30;constexpr int kCategoryCount=4;constexpr int kSilenceIndex=0;constexpr int kUnknownIndex=1;extern const char* kCategoryLabels[kCategoryCount];#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_
// This is a standard TensorFlow Lite model file that has been converted into a
// C data array, so it can be easily compiled into a binary for devices that
// don't have a file system. It was created using the command:
// xxd -i tiny_conv.tflite > tiny_conv_simple_features_model_data.cc
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_extern const unsigned char g_tiny_conv_micro_features_model_data[];extern const int g_tiny_conv_micro_features_model_data_len;#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_extern const int g_yes_micro_f2e59fea_nohash_1_width;extern const int g_yes_micro_f2e59fea_nohash_1_height;extern const unsigned char g_yes_micro_f2e59fea_nohash_1_data[];#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_
Main Script#20
Arduino
recognize_commands.cpp
#include "recognize_commands.h"#include <limits>RecognizeCommands::RecognizeCommands(tflite::ErrorReporter* error_reporter,
int32_t average_window_duration_ms,
uint8_t detection_threshold,
int32_t suppression_ms,
int32_t minimum_count) : error_reporter_(error_reporter),
average_window_duration_ms_(average_window_duration_ms),
detection_threshold_(detection_threshold),
suppression_ms_(suppression_ms),
minimum_count_(minimum_count),
previous_results_(error_reporter){previous_top_label_="silence";previous_top_label_time_= std::numeric_limits<int32_t>::min();}TfLiteStatus RecognizeCommands::ProcessLatestResults( const TfLiteTensor* latest_results, const int32_t current_time_ms,
const char** found_command, uint8_t* score, bool* is_new_command){if((latest_results->dims->size !=2)||(latest_results->dims->data[0] !=1)||(latest_results->dims->data[1] != kCategoryCount)){ error_reporter_->Report("The results for recognition should contain %d elements, but there are ""%d in an %d-dimensional shape",
kCategoryCount, latest_results->dims->data[1],
latest_results->dims->size);return kTfLiteError;}if(latest_results->type != kTfLiteUInt8){ error_reporter_->Report("The results for recognition should be uint8 elements, but are %d",
latest_results->type);return kTfLiteError;}if((!previous_results_.empty())&&(current_time_ms < previous_results_.front().time_)){ error_reporter_->Report("Results must be fed in increasing time order, but received a ""timestamp of %d that was earlier than the previous one of %d",
current_time_ms, previous_results_.front().time_);return kTfLiteError;} // Add the latest results to the head of the queue.
previous_results_.push_back({current_time_ms, latest_results->data.uint8}); // Prune any earlier results that are too old for the averaging window.
const int64_t time_limit= current_time_ms - average_window_duration_ms_;while((!previous_results_.empty())&& previous_results_.front().time_ < time_limit){ previous_results_.pop_front();} // If there are too few results, assume the result will be unreliable and
// bail.
const int64_t how_many_results= previous_results_.size(); const int64_t earliest_time= previous_results_.front().time_; const int64_t samples_duration= current_time_ms - earliest_time;if((how_many_results < minimum_count_)||(samples_duration < (average_window_duration_ms_ / 4))){ *found_command = previous_top_label_; *score =0; *is_new_command = false;return kTfLiteOk;} // Calculate the average score across all the results in the window.
int32_t average_scores[kCategoryCount];for(int offset=0; offset < previous_results_.size(); ++offset){ PreviousResultsQueue::Result previous_result= previous_results_.from_front(offset); const uint8_t* scores= previous_result.scores_;for(int i=0; i < kCategoryCount; ++i){if(offset==0){ average_scores[i]= scores[i];}else{ average_scores[i] += scores[i];}}}for(int i=0; i < kCategoryCount; ++i){ average_scores[i] /= how_many_results;} // Find the current highest scoring category.
int current_top_index=0; int32_t current_top_score=0;for(int i=0; i < kCategoryCount; ++i){if(average_scores[i] > current_top_score){current_top_score= average_scores[i];current_top_index= i;}} const char* current_top_label= kCategoryLabels[current_top_index]; // If we've recently had another label trigger, assume one that occurs too
// soon afterwards is a bad result.
int64_t time_since_last_top;if((previous_top_label_== kCategoryLabels[0])||(previous_top_label_time_== std::numeric_limits<int32_t>::min())){time_since_last_top= std::numeric_limits<int32_t>::max();}else{time_since_last_top= current_time_ms - previous_top_label_time_;}if((current_top_score > detection_threshold_)&&((current_top_label != previous_top_label_)||(time_since_last_top > suppression_ms_))){previous_top_label_= current_top_label;previous_top_label_time_= current_time_ms; *is_new_command = true;}else{ *is_new_command = false;} *found_command = current_top_label; *score = current_top_score;return kTfLiteOk;}
Main Script#21
Arduino
recognize_commands.h
#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_#include <cstdint>#include "tensorflow/lite/c/c_api_internal.h"#include "micro_features_micro_model_settings.h"#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"// Partial implementation of std::dequeue, just providing the functionality
// that's needed to keep a record of previous neural network results over a// short time period, so they can be averaged together to produce a more// accurate overall prediction. This doesn't use any dynamic memory allocation
// so it's a better fit for microcontroller applications, but this does mean// there are hard limits on the number of results it can store.class PreviousResultsQueue { public: PreviousResultsQueue(tflite::ErrorReporter* error_reporter) : error_reporter_(error_reporter), front_index_(0), size_(0) {} // Data structure that holds an inference result, and the time when it // was recorded. struct Result { Result() : time_(0), scores_() {} Result(int32_t time, uint8_t* scores) : time_(time) { for (int i = 0; i < kCategoryCount; ++i) { scores_[i] = scores[i]; } } int32_t time_; uint8_t scores_[kCategoryCount]; }; int size() { return size_; } bool empty() { return size_ == 0; } Result& front() { return results_[front_index_]; } Result& back() { int back_index = front_index_ + (size_ - 1); if (back_index >= kMaxResults) { back_index -= kMaxResults; } return results_[back_index]; } void push_back(const Result& entry) { if (size() >= kMaxResults) { error_reporter_->Report( "Couldn't push_back latest result, too many already!"); return; } size_ += 1; back() = entry; } Result pop_front() { if (size() <= 0) { error_reporter_->Report("Couldn't pop_front result, none present!"); return Result(); } Result result = front(); front_index_ += 1; if (front_index_ >= kMaxResults) { front_index_ = 0; } size_ -= 1; return result; } // Most of the functions are duplicates of dequeue containers, but this // is a helper that makes it easy to iterate through the contents of the // queue. Result& from_front(int offset) { if ((offset < 0) || (offset >= size_)) { error_reporter_->Report("Attempt to read beyond the end of the queue!"); offset = size_ - 1; } int index = front_index_ + offset; if (index >= kMaxResults) { index -= kMaxResults; } return results_[index]; } private: tflite::ErrorReporter* error_reporter_; static constexpr int kMaxResults = 50; Result results_[kMaxResults]; int front_index_; int size_;};// This class is designed to apply a very primitive decoding model on top of the// instantaneous results from running an audio recognition model on a single// window of samples. It applies smoothing over time so that noisy individual// label scores are averaged, increasing the confidence that apparent matches// are real.// To use it, you should create a class object with the configuration you// want, and then feed results from running a TensorFlow model into the// processing method. The timestamp for each subsequent call should be// increasing from the previous, since the class is designed to process a stream// of data over time.class RecognizeCommands { public: // labels should be a list of the strings associated with each one-hot score. // The window duration controls the smoothing. Longer durations will give a // higher confidence that the results are correct, but may miss some commands. // The detection threshold has a similar effect, with high values increasing // the precision at the cost of recall. The minimum count controls how many // results need to be in the averaging window before it's seen as a reliable
// average. This prevents erroneous results when the averaging window is
// initially being populated for example. The suppression argument disables
// further recognitions for a settime after one has been triggered, which can
// help reduce spurious recognitions.
explicit RecognizeCommands(tflite::ErrorReporter* error_reporter,
int32_t average_window_duration_ms=1000,
uint8_t detection_threshold=200,
int32_t suppression_ms=1500,
int32_t minimum_count=3); // Call this with the results of running a model on sample data.
TfLiteStatus ProcessLatestResults(const TfLiteTensor* latest_results,
const int32_t current_time_ms,
const char** found_command, uint8_t* score,
bool* is_new_command); private:
// Configuration
tflite::ErrorReporter* error_reporter_; int32_t average_window_duration_ms_; uint8_t detection_threshold_; int32_t suppression_ms_; int32_t minimum_count_; // Working variables
PreviousResultsQueue previous_results_; const char* previous_top_label_; int32_t previous_top_label_time_;};#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_
void setup(){ // initialize the serial communication:
Serial.begin(9600); pinMode(22, INPUT); //leads off detection LO +
pinMode(23, INPUT); //leads off detection LO -
}void loop(){if((digitalRead(22)==1)||(digitalRead(23)==1)){ Serial.println('^');}else{ Serial.println(analogRead(A29));} delay(50);}
Biosig acquistion HRV & Sync. OLED
Arduino
#include <Wire.h>#include "MAX30105.h"#include "heartRate.h"MAX30105 particleSensor;#include <SFE_MicroOLED.h> // Include the SFE_MicroOLED library#define PIN_RESET 9 #define DC_JUMPER 1 MicroOLED oled(PIN_RESET, DC_JUMPER);#define blinkPin LED_BUILTIN#include "SparkFun_VCNL4040_Arduino_Library.h"VCNL4040 proximitySensor;const byte RATE_SIZE=4; //Increase this for more averaging. 4 is good.
byte rates[RATE_SIZE]; //Array of heart rates
byte rateSpot=0;long lastBeat=0; //Time at which the last beat occurred
float beatsPerMinute;int beatAvg;void setup(){ Serial.begin(9600);if(!particleSensor.begin(Wire, I2C_SPEED_FAST)) //Use default I2C port, 400kHz speed
{ Serial.println("MAX30105 was not found. Please check wiring/power. ");while(1);} particleSensor.setup(); //Configure sensor with default settings
particleSensor.setPulseAmplitudeRed(0x0A); //Turn Red LED to low to indicate sensor is running
particleSensor.setPulseAmplitudeGreen(0); //Turn off Green LED
if(proximitySensor.begin()==false){ Serial.println("Device not found. Please check wiring.");while(1); //Freeze!
} pinMode(blinkPin, OUTPUT); Wire.begin(); oled.begin(); // Initialize the OLED
oled.clear(ALL); // Clear the display's internal memory oled.display(); // Display what's in the buffer (splashscreen) delay(1000); // Delay 1000 ms
oled.clear(PAGE); // Clear the buffer.
}void loop(){ long irValue= particleSensor.getIR();if(checkForBeat(irValue)==true){ //We sensed a beat!
long delta= millis() - lastBeat;lastBeat= millis();beatsPerMinute=60 / (delta / 1000.0);if(beatsPerMinute < 255&& beatsPerMinute > 20){ rates[rateSpot++]=(byte)beatsPerMinute; //Store this reading in the array
rateSpot %= RATE_SIZE; //Wrap variable
//Take average of readings
beatAvg=0;for(byte x=0; x < RATE_SIZE ; x++)beatAvg+= rates[x]; beatAvg /= RATE_SIZE;}}if(irValue > 60000){ //Serial.print("IR="); //Serial.print(irValue); Serial.print(" BPM="); Serial.print(beatsPerMinute); Serial.print(", Avg BPM="); Serial.print(beatAvg); Serial.println();} unsigned int proxValue= proximitySensor.getProximity();if(proxValue>14000){ Serial.println(""); Serial.println("Tap Activated"); Serial.print("...Sending the accumulated Sensor data to Local-Hub"); //Serial.print(proxValue); Serial.println(); delay(200); digitalWrite(blinkPin, HIGH); delay(150); digitalWrite(blinkPin, LOW); delay(400); digitalWrite(blinkPin, HIGH); delay(150); digitalWrite(blinkPin, LOW); delay(400); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240); digitalWrite(blinkPin, HIGH); delay(80); digitalWrite(blinkPin, LOW); delay(240);}int middleZ=(oled.getLCDWidth() / 2)-14; int middleX= oled.getLCDWidth() / 2; int middleY= oled.getLCDHeight() / 2; oled.clear(PAGE); oled.setCursor(middleZ - (oled.getFontWidth() * (2)+3),
middleY - (oled.getFontHeight() / 2)-17);oled.print("HEART-RATE");// oled.setCursor(middleZ - (oled.getFontWidth() * (2)-6),
// middleY - (oled.getFontHeight() / 2)-8);//oled.print("Be Heart SMART!"); oled.setCursor(middleX - (oled.getFontWidth() * (2)+1),
middleY - (oled.getFontHeight() / 2)+17); oled.print(beatAvg); oled.print("bpm"); //delay(7000); oled.display(); oled.clear(PAGE);}
AlexaRing script for ESP32 with support for TFlite mic.
Arduino
#include <Adafruit_NeoPixel.h>#define DATA_PIN 13#define ArtemisMIC 12#define SIZE 13 /* Echo (Dot) has 12 LEDs, but code adapts to higher numbers */Adafruit_NeoPixel ring= Adafruit_NeoPixel(24, DATA_PIN, NEO_GRB + NEO_KHZ800);const int DURATION=950; // animation duration (ms)const int STEP_SIZE= DURATION/SIZE;const uint32_t NOTHING= ring.Color(255, 0, 0);const uint32_t MUTE= ring.Color(153, 0, 0);const uint32_t BOOTING_BG= ring.Color(0, 0, 90);const uint32_t BOOTING_FG= ring.Color(0, 230, 230);const uint32_t SPEAK_MIN= ring.Color(0, 0, 255);const uint32_t SPEAK_MAX= ring.Color(0, 255, 255);const uint32_t ERR_MIN= ring.Color(17, 17, 0);const uint32_t ERR_MAX= ring.Color(255, 17, 0);const uint32_t MSG_MIN= ring.Color(17, 17, 0); // unverified
const uint32_t MSG_MAX= ring.Color(255, 255, 0); // unverified
const uint32_t CALL_MIN= ring.Color(0, 17, 0); // unverified
const uint32_t CALL_MAX= ring.Color(0, 255, 0); // unverified
// TODO: purple (do not disturb), orange spin (connecting)uint32_t interpolate(uint32_t c1, uint32_t c2, double t){ // assumes 0.0 <= t <=1.0
uint8_t r=(c1>>16 & 0xFF)*(1-t) + (c2>>16 & 0xFF)*t; uint8_t g=(c1>>8 & 0xFF)*(1-t) + (c2>>8 & 0xFF)*t; uint8_t b=(c1 & 0xFF)*(1-t) + (c2 & 0xFF)*t;return((uint32_t)r << 16) | ((uint32_t)g << 8) | b;}// for: nothing/listening, muted, do not disturbvoid fill(uint32_t colour) { for (int i=0; i<SIZE; i++) { ring.setPixelColor(i, colour); }}// for: bootingvoid spinner(uint32_t colour_bg, uint32_t colour_fg, int pos, int width=ceil(SIZE/12.0));void spinner(uint32_t colour_bg, uint32_t colour_fg, int pos, int width){ fill(colour_bg);for(int i=pos; i<pos+width; i++){ ring.setPixelColor(i % SIZE, colour_fg); // rotate clockwise
}}// for: speaking, error, don't disturb, incoming call, ..void pulse(uint32_t colour_min, uint32_t colour_max, double ratio, bool up) { double t = (up) ? ratio : 1 - ratio; uint32_t colour = interpolate(colour_min, colour_max, t); fill(colour);}/*** SKETCH ***/int counter = 0; // step function (spinner)double ratio = 0; // continuous (pulse)bool up = true; // direction (pulse)unsigned long last_increment = millis();enum states { BLANK, MUTED, BOOTING, SPEAKING, ERROR, MESSAGE, CALL,} state;void setup() { ring.begin(); ring.show(); ring.setBrightness(100); pinMode(ArtemisMIC, INPUT_PULLUP); state = BOOTING;}void loop() { switch (state) { case MUTED: fill(MUTE); break; case BOOTING: spinner(BOOTING_BG, BOOTING_FG, counter); break; case SPEAKING: pulse(SPEAK_MIN, SPEAK_MAX, ratio, up); break; case ERROR: pulse(ERR_MIN, ERR_MAX, ratio, up); break; case MESSAGE: pulse(MSG_MIN, MSG_MAX, ratio, up); break; case CALL: pulse(CALL_MIN, CALL_MAX, ratio, up); break; case BLANK: default: fill(MESSAGE); break; } ring.show(); if (millis() - last_increment > STEP_SIZE || millis() < last_increment) { // 0->SIZE per DURATION counter = ++counter % SIZE; if (counter == 0) { up = !up; // swap } last_increment = millis(); } // 0.0->1.0 per duration ratio = min(1.0, (double)(counter*STEP_SIZE + (millis() - last_increment)) / DURATION); // YOUR LOGIC (state transitions) // This example sketch shows the 'booting' animation, // and switches to the 'muted' animation as long as the S.R._mic behaviour changes
int MICresponse= digitalRead(ArtemisMIC); switch (state){case BOOTING: if(!MICresponse){state= MUTED;} break;case MUTED: if(MICresponse){state= BOOTING;} break; default: state= MESSAGE; break;} delay(1);}
_1x_bGT19vVAby.png?auto=compress%2Cformat&w=40&h=40&fit=fillmax&bg=fff&dpr=2){kind=logo}
_3u05Tpwasz.png?auto=compress%2Cformat&w=40&h=40&fit=fillmax&bg=fff&dpr=2)
Comments