This is an implementation on Arduino Nano 33 BLE Sense for the Tensorflow's example program: speech commands. The project is a course project for COMP554 at Rice University. The implementation process is detailed described in Chapter 7 and Chapter 8 of the book TinyML: Machine Learning with TensorFlow on Arduino and Ultra-Low Power Micro-Controllers.
Step 1: Neural Network Training
Luckily, TensorFlow Developers already provides a training script for us in Google Colab:
The python notebook includes shell and python codes. The train.py script under tensorflow/tensorflow/examples/speech_commands would automatically download the Speech Commands dataset which includes WAVE audio files for 35 different words:
However, due to our test, the Arduino Nano 33 BLE Sense board only supports classification between two words now. So, the only thing required to be modified is the two words in the first code cell:
# A comma-delimited list of the words you want to train for. # The options are: yes,no,up,down,left,right,on,off,stop,go # All other words will be used to train an "unknown" category. os.environ["WANTED_WORDS"] = "yes,no"
Replace "yes, no" by any two words we want to recognize, and run all cells to train the Neural Network Model.
The only support model is the tiny_conv model, which contains a convolution layer and an FC layer.
(fingerprint_input) v [Conv2D]<-(weights) v [BiasAdd]<-(bias) v [Relu] v [MatMul]<-(weights) v [BiasAdd]<-(bias) v
We cannot change or modify the model because the model on the device is defined in a static library and we do not have the code.
The script will generate a "tiny_conv.cc" as output.
Step 2: Training Result Upload
Connect the Arduino Nano BLE Sense Board to Computer and use Arduino as our IDE. At first, install Arduino Nano 33 BLE in the boards manager and install Arduino Tensorflow Lite in library manager from Tools. Then get an example called micro_speech in Arduino_TensorFlowLite from File. Use the generated tiny_conv.cc, the result of our step 1, to replace micro_features_tiny_conv_micro_features_model_data.cpp. Modify the variable kCategoryCount to be 4 in micro_features_micro_model_settings.h and replace values "yes" and "no" to be "right" and "left" in micro_features_micro_model_settings.cpp.
In arduino_commend_responder.cpp, modify the respond of each detected word, red light corresponds to the word "left" while green light corresponds to the word "right". Last, choose the correct port and device from Tools and upload it from Sketch.
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"micro_features_micro_features_generator.h"#include<cmath>#include<cstring>#include"micro_features_micro_model_settings.h"#include"tensorflow/lite/experimental/microfrontend/lib/frontend.h"#include"tensorflow/lite/experimental/microfrontend/lib/frontend_util.h"// Configure FFT to output 16 bit fixed point.#define FIXED_POINT 16namespace{FrontendStateg_micro_features_state;boolg_is_first_time=true;}// namespaceTfLiteStatusInitializeMicroFeatures(tflite::ErrorReporter*error_reporter){FrontendConfigconfig;config.window.size_ms=kFeatureSliceDurationMs;config.window.step_size_ms=kFeatureSliceStrideMs;config.noise_reduction.smoothing_bits=10;config.filterbank.num_channels=kFeatureSliceSize;config.filterbank.lower_band_limit=125.0;config.filterbank.upper_band_limit=7500.0;config.noise_reduction.smoothing_bits=10;config.noise_reduction.even_smoothing=0.025;config.noise_reduction.odd_smoothing=0.06;config.noise_reduction.min_signal_remaining=0.05;config.pcan_gain_control.enable_pcan=1;config.pcan_gain_control.strength=0.95;config.pcan_gain_control.offset=80.0;config.pcan_gain_control.gain_bits=21;config.log_scale.enable_log=1;config.log_scale.scale_shift=6;if(!FrontendPopulateState(&config,&g_micro_features_state,kAudioSampleFrequency)){error_reporter->Report("FrontendPopulateState() failed");returnkTfLiteError;}g_is_first_time=true;returnkTfLiteOk;}// This is not exposed in any header, and is only used for testing, to ensure// that the state is correctly set up before generating results.voidSetMicroFeaturesNoiseEstimates(constuint32_t*estimate_presets){for(inti=0;i<g_micro_features_state.filterbank.num_channels;++i){g_micro_features_state.noise_reduction.estimate[i]=estimate_presets[i];}}TfLiteStatusGenerateMicroFeatures(tflite::ErrorReporter*error_reporter,constint16_t*input,intinput_size,intoutput_size,uint8_t*output,size_t*num_samples_read){constint16_t*frontend_input;if(g_is_first_time){frontend_input=input;g_is_first_time=false;}else{frontend_input=input+160;}FrontendOutputfrontend_output=FrontendProcessSamples(&g_micro_features_state,frontend_input,input_size,num_samples_read);for(inti=0;i<frontend_output.size;++i){// These scaling values are derived from those used in input_data.py in the// training pipeline.constexprint32_tvalue_scale=(10*255);constexprint32_tvalue_div=(256*26);int32_tvalue=((frontend_output.values[i]*value_scale)+(value_div/2))/value_div;if(value<0){value=0;}if(value>255){value=255;}output[i]=value;}returnkTfLiteOk;}
micro_features_micro_features_generator.h
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_#include"tensorflow/lite/c/c_api_internal.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"// Sets up any resources needed for the feature generation pipeline.TfLiteStatusInitializeMicroFeatures(tflite::ErrorReporter*error_reporter);// Converts audio sample data into a more compact form that's appropriate for// feeding into a neural network.TfLiteStatusGenerateMicroFeatures(tflite::ErrorReporter*error_reporter,constint16_t*input,intinput_size,intoutput_size,uint8_t*output,size_t*num_samples_read);#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_FEATURES_GENERATOR_H_
micro_features_micro_model_settings.cpp
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"micro_features_micro_model_settings.h"constchar*kCategoryLabels[kCategoryCount]={"silence","unknown","left","right",//"up",//"down"};
micro_features_micro_model_settings.h
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_// Keeping these as constant expressions allow us to allocate fixed-sized arrays// on the stack for our working memory.// The size of the input time series data we pass to the FFT to produce the// frequency information. This has to be a power of two, and since we're dealing// with 30ms of 16KHz inputs, which means 480 samples, this is the next value.constexprintkMaxAudioSampleSize=512;constexprintkAudioSampleFrequency=16000;// All of these values are derived from the values used during model training,// if you change your model you'll need to update these constants.constexprintkFeatureSliceSize=40;constexprintkFeatureSliceCount=49;constexprintkFeatureElementCount=(kFeatureSliceSize*kFeatureSliceCount);constexprintkFeatureSliceStrideMs=20;constexprintkFeatureSliceDurationMs=30;constexprintkCategoryCount=4;constexprintkSilenceIndex=0;constexprintkUnknownIndex=1;externconstchar*kCategoryLabels[kCategoryCount];#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_MICRO_MODEL_SETTINGS_H_
micro_features_no_micro_features_data.cpp
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"micro_features_no_micro_features_data.h"/* File automatically created by * tensorflow/examples/speech_commands/wav_to_features.py \ * --sample_rate=16000 \ * --clip_duration_ms=1000 \ * --window_size_ms=30 \ * --window_stride_ms=20 \ * --feature_bin_count=40 \ * --quantize=1 \ * --preprocess="micro" \ * --input_wav="speech_commands_test_set_v0.02/no/f9643d42_nohash_4.wav" \ * --output_c_file="/tmp/no_micro_features_data.cc" \ */constintg_no_micro_f9643d42_nohash_4_width=40;constintg_no_micro_f9643d42_nohash_4_height=49;constunsignedcharg_no_micro_f9643d42_nohash_4_data[]={230,205,191,203,202,181,180,194,205,187,183,197,203,198,196,186,202,159,151,126,110,138,141,142,137,148,133,120,110,126,117,110,117,116,137,134,95,116,123,110,184,144,183,189,197,172,188,164,194,179,175,174,182,173,184,174,200,145,154,148,147,135,143,122,127,138,116,99,122,105,110,125,127,133,131,123,116,119,127,114,193,176,185,170,175,146,166,167,185,185,185,183,195,185,176,178,197,155,137,144,164,132,153,132,138,137,134,95,120,116,131,122,99,120,120,110,116,110,126,127,128,159,187,119,178,187,197,167,199,184,180,165,194,176,144,134,187,136,142,134,145,132,145,105,119,123,125,116,125,102,129,138,130,99,99,90,120,123,134,95,194,172,187,123,191,179,195,182,201,137,167,142,185,161,187,146,167,152,154,107,152,112,134,144,117,116,105,85,105,105,99,90,123,112,112,68,107,105,117,99,116,143,139,90,154,142,188,172,178,135,175,149,177,110,173,160,169,162,173,119,132,110,85,85,117,129,117,112,117,51,112,95,139,102,105,90,128,119,112,99,170,168,195,152,174,173,180,0,157,130,169,149,149,123,170,130,170,133,159,102,134,90,85,105,126,119,130,90,78,68,127,120,95,51,122,110,112,78,116,95,180,135,179,146,179,162,197,153,172,135,154,0,149,95,145,114,166,0,114,110,145,107,114,90,136,68,95,95,95,85,116,99,116,0,95,68,102,51,102,78,185,157,138,158,180,117,173,142,145,117,169,130,159,99,138,123,169,90,78,0,123,85,107,51,114,102,95,0,116,85,119,95,95,68,85,51,116,68,102,78,167,105,164,163,178,126,164,154,154,51,177,120,156,85,134,139,168,90,161,102,114,116,122,95,112,102,107,51,114,85,119,78,114,90,102,51,102,51,114,99,177,68,152,102,184,166,179,129,177,129,180,110,158,105,139,0,145,85,148,102,117,102,116,0,78,68,90,51,107,85,78,0,51,0,51,0,95,51,107,68,180,117,90,0,138,0,187,146,119,140,164,90,136,0,131,51,159,99,141,138,116,51,90,51,90,68,105,0,85,78,112,51,122,95,128,68,85,0,112,68,147,126,178,146,171,130,190,147,188,123,170,78,132,0,130,125,159,95,102,0,110,0,95,85,120,68,78,51,99,51,105,0,112,102,105,68,90,51,90,0,127,95,166,175,187,133,135,0,171,139,132,128,140,51,126,107,161,0,95,51,119,0,114,0,95,110,116,51,112,0,90,0,116,51,68,0,105,68,105,0,164,78,173,0,194,166,145,114,116,51,107,122,151,0,156,102,148,51,122,95,129,0,85,0,127,78,90,0,78,0,95,0,110,0,68,119,120,68,68,0,122,99,147,127,200,167,85,114,161,85,161,125,143,99,156,85,147,68,99,0,107,102,132,51,112,68,95,78,99,0,68,0,51,0,90,78,128,51,95,0,166,136,174,138,189,144,130,129,138,134,132,120,134,0,51,78,147,51,51,0,51,0,78,0,68,68,95,78,90,0,0,0,68,0,90,68,110,0,95,51,165,151,157,0,0,0,112,0,112,95,149,107,119,68,126,68,138,0,78,0,78,0,99,51,112,0,102,0,78,51,85,0,0,0,78,0,95,0,95,78,105,0,152,0,0,51,132,105,159,0,129,102,114,0,138,51,123,0,129,78,119,51,51,51,105,0,78,85,95,0,85,0,0,0,85,0,78,0,0,0,172,142,141,0,137,0,148,128,157,120,146,120,120,0,95,78,141,68,68,0,68,0,90,0,85,0,107,0,78,0,85,51,102,0,68,78,68,0,51,0,125,0,141,51,102,138,175,51,120,51,173,85,116,141,164,68,150,123,133,51,114,0,117,68,150,51,116,68,78,0,68,0,68,0,85,0,78,0,51,78,155,90,161,0,132,99,123,78,107,0,134,90,95,0,78,0,162,143,85,0,107,78,125,90,90,51,51,0,85,0,0,0,132,102,102,154,128,0,99,68,162,102,151,0,99,51,147,141,156,0,112,120,158,127,145,139,187,171,135,138,146,0,95,68,127,0,85,0,105,0,0,0,187,170,162,188,165,51,51,78,243,215,225,196,205,181,205,168,176,134,157,110,126,114,133,139,193,163,159,116,160,126,122,127,171,99,114,68,123,85,90,0,157,146,166,179,136,0,116,90,242,219,240,204,216,164,188,171,176,164,154,158,190,157,190,141,182,177,169,128,172,145,105,129,157,90,78,51,119,68,137,68,116,78,141,132,151,122,156,140,234,206,229,201,216,174,191,144,162,85,122,157,194,167,204,149,180,166,166,139,122,133,156,126,145,85,128,0,99,51,145,0,126,51,166,162,166,162,177,157,228,198,221,197,214,177,173,166,173,139,185,191,202,163,205,172,206,189,135,68,166,134,149,134,135,90,127,107,175,90,136,117,135,140,172,167,166,149,177,152,221,191,215,194,211,0,156,147,182,178,208,163,190,157,208,200,195,164,179,154,181,150,143,99,132,137,185,143,163,85,51,107,132,134,164,127,167,159,175,141,216,195,223,211,238,223,243,215,226,204,232,211,232,213,240,218,235,214,238,205,207,173,149,201,215,200,230,213,208,195,175,151,195,175,182,163,235,217,218,190,211,191,215,191,217,220,241,215,229,206,236,210,227,216,236,188,183,149,202,189,208,172,191,201,220,193,221,207,216,208,201,131,170,187,229,197,211,194,226,201,205,184,206,177,221,210,226,184,204,197,218,198,212,209,213,141,172,110,175,167,180,156,213,188,192,179,213,205,204,174,200,147,162,181,203,167,198,187,210,164,196,169,189,168,224,198,213,204,198,195,230,211,221,197,208,0,0,0,85,90,167,130,175,173,203,164,193,144,170,145,185,148,154,139,198,159,180,171,216,174,178,161,166,136,216,184,215,197,199,190,228,195,208,51,117,0,0,0,0,0,140,51,135,154,188,155,168,0,90,0,156,85,110,0,174,90,172,154,179,99,142,166,179,157,177,95,192,142,204,198,217,147,173,0,112,0,0,0,0,0,0,0,110,0,107,0,160,0,148,95,172,0,0,0,116,0,122,114,170,0,0,0,0,0,179,110,196,85,205,183,169,0,99,0,0,0,0,0,0,0,0,0,0,0,141,0,112,0,0,0,134,0,0,0,0,0,0,0,139,0,0,0,0,112,186,78,163,0,169,128,174,0,0,0,0,0,0,0,0,0,0,0,0,0,95,0,105,0,0,0,105,0,0,0,0,0,0,0,95,0,0,0,0,0,0,0,119,0,164,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,90,0,0,68,117,0,0,0,0,0,0,0,148,0,0,0,0,0,0,0,0,0,116,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,51,0,0,0,99,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,99,0,0,0,0,0,0,0,0,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,};
recognize_commands.h
C/C++
/* Copyright 2017 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_#include<cstdint>#include"tensorflow/lite/c/c_api_internal.h"#include"micro_features_micro_model_settings.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"// Partial implementation of std::dequeue, just providing the functionality// that's needed to keep a record of previous neural network results over a// short time period, so they can be averaged together to produce a more// accurate overall prediction. This doesn't use any dynamic memory allocation// so it's a better fit for microcontroller applications, but this does mean// there are hard limits on the number of results it can store.classPreviousResultsQueue{public:PreviousResultsQueue(tflite::ErrorReporter*error_reporter):error_reporter_(error_reporter),front_index_(0),size_(0){}// Data structure that holds an inference result, and the time when it// was recorded.structResult{Result():time_(0),scores_(){}Result(int32_ttime,uint8_t*scores):time_(time){for(inti=0;i<kCategoryCount;++i){scores_[i]=scores[i];}}int32_ttime_;uint8_tscores_[kCategoryCount];};intsize(){returnsize_;}boolempty(){returnsize_==0;}Result&front(){returnresults_[front_index_];}Result&back(){intback_index=front_index_+(size_-1);if(back_index>=kMaxResults){back_index-=kMaxResults;}returnresults_[back_index];}voidpush_back(constResult&entry){if(size()>=kMaxResults){error_reporter_->Report("Couldn't push_back latest result, too many already!");return;}size_+=1;back()=entry;}Resultpop_front(){if(size()<=0){error_reporter_->Report("Couldn't pop_front result, none present!");returnResult();}Resultresult=front();front_index_+=1;if(front_index_>=kMaxResults){front_index_=0;}size_-=1;returnresult;}// Most of the functions are duplicates of dequeue containers, but this// is a helper that makes it easy to iterate through the contents of the// queue.Result&from_front(intoffset){if((offset<0)||(offset>=size_)){error_reporter_->Report("Attempt to read beyond the end of the queue!");offset=size_-1;}intindex=front_index_+offset;if(index>=kMaxResults){index-=kMaxResults;}returnresults_[index];}private:tflite::ErrorReporter*error_reporter_;staticconstexprintkMaxResults=50;Resultresults_[kMaxResults];intfront_index_;intsize_;};// This class is designed to apply a very primitive decoding model on top of the// instantaneous results from running an audio recognition model on a single// window of samples. It applies smoothing over time so that noisy individual// label scores are averaged, increasing the confidence that apparent matches// are real.// To use it, you should create a class object with the configuration you// want, and then feed results from running a TensorFlow model into the// processing method. The timestamp for each subsequent call should be// increasing from the previous, since the class is designed to process a stream// of data over time.classRecognizeCommands{public:// labels should be a list of the strings associated with each one-hot score.// The window duration controls the smoothing. Longer durations will give a// higher confidence that the results are correct, but may miss some commands.// The detection threshold has a similar effect, with high values increasing// the precision at the cost of recall. The minimum count controls how many// results need to be in the averaging window before it's seen as a reliable// average. This prevents erroneous results when the averaging window is// initially being populated for example. The suppression argument disables// further recognitions for a set time after one has been triggered, which can// help reduce spurious recognitions.explicitRecognizeCommands(tflite::ErrorReporter*error_reporter,int32_taverage_window_duration_ms=1000,uint8_tdetection_threshold=200,int32_tsuppression_ms=1500,int32_tminimum_count=3);// Call this with the results of running a model on sample data.TfLiteStatusProcessLatestResults(constTfLiteTensor*latest_results,constint32_tcurrent_time_ms,constchar**found_command,uint8_t*score,bool*is_new_command);private:// Configurationtflite::ErrorReporter*error_reporter_;int32_taverage_window_duration_ms_;uint8_tdetection_threshold_;int32_tsuppression_ms_;int32_tminimum_count_;// Working variablesPreviousResultsQueueprevious_results_;constchar*previous_top_label_;int32_tprevious_top_label_time_;};#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_RECOGNIZE_COMMANDS_H_
recognize_commands.cpp
C/C++
/* Copyright 2017 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"recognize_commands.h"#include<limits>RecognizeCommands::RecognizeCommands(tflite::ErrorReporter*error_reporter,int32_taverage_window_duration_ms,uint8_tdetection_threshold,int32_tsuppression_ms,int32_tminimum_count):error_reporter_(error_reporter),average_window_duration_ms_(average_window_duration_ms),detection_threshold_(detection_threshold),suppression_ms_(suppression_ms),minimum_count_(minimum_count),previous_results_(error_reporter){previous_top_label_="silence";previous_top_label_time_=std::numeric_limits<int32_t>::min();}TfLiteStatusRecognizeCommands::ProcessLatestResults(constTfLiteTensor*latest_results,constint32_tcurrent_time_ms,constchar**found_command,uint8_t*score,bool*is_new_command){if((latest_results->dims->size!=2)||(latest_results->dims->data[0]!=1)||(latest_results->dims->data[1]!=kCategoryCount)){error_reporter_->Report("The results for recognition should contain %d elements, but there are ""%d in an %d-dimensional shape",kCategoryCount,latest_results->dims->data[1],latest_results->dims->size);returnkTfLiteError;}if(latest_results->type!=kTfLiteUInt8){error_reporter_->Report("The results for recognition should be uint8 elements, but are %d",latest_results->type);returnkTfLiteError;}if((!previous_results_.empty())&&(current_time_ms<previous_results_.front().time_)){error_reporter_->Report("Results must be fed in increasing time order, but received a ""timestamp of %d that was earlier than the previous one of %d",current_time_ms,previous_results_.front().time_);returnkTfLiteError;}// Add the latest results to the head of the queue.previous_results_.push_back({current_time_ms,latest_results->data.uint8});// Prune any earlier results that are too old for the averaging window.constint64_ttime_limit=current_time_ms-average_window_duration_ms_;while((!previous_results_.empty())&&previous_results_.front().time_<time_limit){previous_results_.pop_front();}// If there are too few results, assume the result will be unreliable and// bail.constint64_thow_many_results=previous_results_.size();constint64_tearliest_time=previous_results_.front().time_;constint64_tsamples_duration=current_time_ms-earliest_time;if((how_many_results<minimum_count_)||(samples_duration<(average_window_duration_ms_/4))){*found_command=previous_top_label_;*score=0;*is_new_command=false;returnkTfLiteOk;}// Calculate the average score across all the results in the window.int32_taverage_scores[kCategoryCount];for(intoffset=0;offset<previous_results_.size();++offset){PreviousResultsQueue::Resultprevious_result=previous_results_.from_front(offset);constuint8_t*scores=previous_result.scores_;for(inti=0;i<kCategoryCount;++i){if(offset==0){average_scores[i]=scores[i];}else{average_scores[i]+=scores[i];}}}for(inti=0;i<kCategoryCount;++i){average_scores[i]/=how_many_results;}// Find the current highest scoring category.intcurrent_top_index=0;int32_tcurrent_top_score=0;for(inti=0;i<kCategoryCount;++i){if(average_scores[i]>current_top_score){current_top_score=average_scores[i];current_top_index=i;}}constchar*current_top_label=kCategoryLabels[current_top_index];// If we've recently had another label trigger, assume one that occurs too// soon afterwards is a bad result.int64_ttime_since_last_top;if((previous_top_label_==kCategoryLabels[0])||(previous_top_label_time_==std::numeric_limits<int32_t>::min())){time_since_last_top=std::numeric_limits<int32_t>::max();}else{time_since_last_top=current_time_ms-previous_top_label_time_;}if((current_top_score>detection_threshold_)&&((current_top_label!=previous_top_label_)||(time_since_last_top>suppression_ms_))){previous_top_label_=current_top_label;previous_top_label_time_=current_time_ms;*is_new_command=true;}else{*is_new_command=false;}*found_command=current_top_label;*score=current_top_score;returnkTfLiteOk;}
micro_speech.ino
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include<TensorFlowLite.h>#include"main_functions.h"#include"audio_provider.h"#include"command_responder.h"#include"feature_provider.h"#include"micro_features_micro_model_settings.h"#include"micro_features_tiny_conv_micro_features_model_data.h"#include"recognize_commands.h"#include"tensorflow/lite/experimental/micro/kernels/micro_ops.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"#include"tensorflow/lite/experimental/micro/micro_interpreter.h"#include"tensorflow/lite/experimental/micro/micro_mutable_op_resolver.h"#include"tensorflow/lite/schema/schema_generated.h"#include"tensorflow/lite/version.h"// Globals, used for compatibility with Arduino-style sketches.namespace{tflite::ErrorReporter*error_reporter=nullptr;consttflite::Model*model=nullptr;tflite::MicroInterpreter*interpreter=nullptr;TfLiteTensor*model_input=nullptr;FeatureProvider*feature_provider=nullptr;RecognizeCommands*recognizer=nullptr;int32_tprevious_time=0;// Create an area of memory to use for input, output, and intermediate arrays.// The size of this will depend on the model you're using, and may need to be// determined by experimentation.constexprintkTensorArenaSize=10*1024;uint8_ttensor_arena[kTensorArenaSize];}// namespace// The name of this function is important for Arduino compatibility.voidsetup(){// Set up logging. Google style is to avoid globals or statics because of// lifetime uncertainty, but since this has a trivial destructor it's okay.// NOLINTNEXTLINE(runtime-global-variables)statictflite::MicroErrorReportermicro_error_reporter;error_reporter=µ_error_reporter;// Map the model into a usable data structure. This doesn't involve any// copying or parsing, it's a very lightweight operation.model=tflite::GetModel(g_tiny_conv_micro_features_model_data);if(model->version()!=TFLITE_SCHEMA_VERSION){error_reporter->Report("Model provided is schema version %d not equal ""to supported version %d.",model->version(),TFLITE_SCHEMA_VERSION);return;}// Pull in only the operation implementations we need.// This relies on a complete list of all the ops needed by this graph.// An easier approach is to just use the AllOpsResolver, but this will// incur some penalty in code space for op implementations that are not// needed by this graph.//// tflite::ops::micro::AllOpsResolver resolver;// NOLINTNEXTLINE(runtime-global-variables)statictflite::MicroMutableOpResolvermicro_mutable_op_resolver;micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_DEPTHWISE_CONV_2D,tflite::ops::micro::Register_DEPTHWISE_CONV_2D());micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_FULLY_CONNECTED,tflite::ops::micro::Register_FULLY_CONNECTED());micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,tflite::ops::micro::Register_SOFTMAX());// Build an interpreter to run the model with.statictflite::MicroInterpreterstatic_interpreter(model,micro_mutable_op_resolver,tensor_arena,kTensorArenaSize,error_reporter);interpreter=&static_interpreter;// Allocate memory from the tensor_arena for the model's tensors.TfLiteStatusallocate_status=interpreter->AllocateTensors();if(allocate_status!=kTfLiteOk){error_reporter->Report("AllocateTensors() failed");return;}// Get information about the memory area to use for the model's input.model_input=interpreter->input(0);if((model_input->dims->size!=4)||(model_input->dims->data[0]!=1)||(model_input->dims->data[1]!=kFeatureSliceCount)||(model_input->dims->data[2]!=kFeatureSliceSize)||(model_input->type!=kTfLiteUInt8)){error_reporter->Report("Bad input tensor parameters in model");return;}// Prepare to access the audio spectrograms from a microphone or other source// that will provide the inputs to the neural network.// NOLINTNEXTLINE(runtime-global-variables)staticFeatureProviderstatic_feature_provider(kFeatureElementCount,model_input->data.uint8);feature_provider=&static_feature_provider;staticRecognizeCommandsstatic_recognizer(error_reporter);recognizer=&static_recognizer;previous_time=0;}// The name of this function is important for Arduino compatibility.voidloop(){// Fetch the spectrogram for the current time.constint32_tcurrent_time=LatestAudioTimestamp();inthow_many_new_slices=0;TfLiteStatusfeature_status=feature_provider->PopulateFeatureData(error_reporter,previous_time,current_time,&how_many_new_slices);if(feature_status!=kTfLiteOk){error_reporter->Report("Feature generation failed");return;}previous_time=current_time;// If no new audio samples have been received since last time, don't bother// running the network model.if(how_many_new_slices==0){return;}// Run the model on the spectrogram input and make sure it succeeds.TfLiteStatusinvoke_status=interpreter->Invoke();if(invoke_status!=kTfLiteOk){error_reporter->Report("Invoke failed");return;}// Obtain a pointer to the output tensorTfLiteTensor*output=interpreter->output(0);// Determine whether a command was recognized based on the output of inferenceconstchar*found_command=nullptr;uint8_tscore=0;boolis_new_command=false;TfLiteStatusprocess_status=recognizer->ProcessLatestResults(output,current_time,&found_command,&score,&is_new_command);if(process_status!=kTfLiteOk){error_reporter->Report("RecognizeCommands::ProcessLatestResults() failed");return;}// Do something based on the recognized command. The default implementation// just prints to the error console, but you should replace this with your// own function for a real application.RespondToCommand(error_reporter,current_time,found_command,score,is_new_command);}
Our platform: Arduino Nano 33 BLE Sense
C/C++
No preview (download only).
micro_features_yes_micro_features_data.h
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_externconstintg_yes_micro_f2e59fea_nohash_1_width;externconstintg_yes_micro_f2e59fea_nohash_1_height;externconstunsignedcharg_yes_micro_f2e59fea_nohash_1_data[];#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_YES_MICRO_FEATURES_DATA_H_
arduino_audio_provider.cpp
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*//* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"audio_provider.h"#include"PDM.h"#include"micro_features_micro_model_settings.h"namespace{boolg_is_audio_initialized=false;// An internal buffer able to fit 16x our sample sizeconstexprintkAudioCaptureBufferSize=DEFAULT_PDM_BUFFER_SIZE*16;int16_tg_audio_capture_buffer[kAudioCaptureBufferSize];// A buffer that holds our outputint16_tg_audio_output_buffer[kMaxAudioSampleSize];// Mark as volatile so we can check in a while loop to see if// any samples have arrived yet.volatileint32_tg_latest_audio_timestamp=0;}// namespacevoidCaptureSamples(){// This is how many bytes of new data we have each time this is calledconstintnumber_of_samples=DEFAULT_PDM_BUFFER_SIZE;// Calculate what timestamp the last audio sample representsconstint32_ttime_in_ms=g_latest_audio_timestamp+(number_of_samples/(kAudioSampleFrequency/1000));// Determine the index, in the history of all samples, of the last sampleconstint32_tstart_sample_offset=g_latest_audio_timestamp*(kAudioSampleFrequency/1000);// Determine the index of this sample in our ring bufferconstintcapture_index=start_sample_offset%kAudioCaptureBufferSize;// Read the data to the correct place in our bufferPDM.read(g_audio_capture_buffer+capture_index,DEFAULT_PDM_BUFFER_SIZE);// This is how we let the outside world know that new audio data has arrived.g_latest_audio_timestamp=time_in_ms;}TfLiteStatusInitAudioRecording(tflite::ErrorReporter*error_reporter){// Hook up the callback that will be called with each samplePDM.onReceive(CaptureSamples);// Start listening for audio: MONO @ 16KHz with gain at 20PDM.begin(1,kAudioSampleFrequency);PDM.setGain(20);// Block until we have our first audio samplewhile(!g_latest_audio_timestamp){}returnkTfLiteOk;}TfLiteStatusGetAudioSamples(tflite::ErrorReporter*error_reporter,intstart_ms,intduration_ms,int*audio_samples_size,int16_t**audio_samples){// Set everything up to start receiving audioif(!g_is_audio_initialized){TfLiteStatusinit_status=InitAudioRecording(error_reporter);if(init_status!=kTfLiteOk){returninit_status;}g_is_audio_initialized=true;}// This next part should only be called when the main thread notices that the// latest audio sample data timestamp has changed, so that there's new data// in the capture ring buffer. The ring buffer will eventually wrap around and// overwrite the data, but the assumption is that the main thread is checking// often enough and the buffer is large enough that this call will be made// before that happens.// Determine the index, in the history of all samples, of the first// sample we wantconstintstart_offset=start_ms*(kAudioSampleFrequency/1000);// Determine how many samples we want in totalconstintduration_sample_count=duration_ms*(kAudioSampleFrequency/1000);for(inti=0;i<duration_sample_count;++i){// For each sample, transform its index in the history of all samples into// its index in g_audio_capture_bufferconstintcapture_index=(start_offset+i)%kAudioCaptureBufferSize;// Write the sample to the output bufferg_audio_output_buffer[i]=g_audio_capture_buffer[capture_index];}// Set pointers to provide access to the audio*audio_samples_size=kMaxAudioSampleSize;*audio_samples=g_audio_output_buffer;returnkTfLiteOk;}int32_tLatestAudioTimestamp(){returng_latest_audio_timestamp;}
micro_features_yes_micro_features_data.cpp
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"micro_features_yes_micro_features_data.h"/* File automatically created by * tensorflow/examples/speech_commands/wav_to_features.py \ * --sample_rate=16000 \ * --clip_duration_ms=1000 \ * --window_size_ms=30 \ * --window_stride_ms=20 \ * --feature_bin_count=40 \ * --quantize=1 \ * --preprocess="micro" \ * --input_wav="speech_commands_test_set_v0.02/yes/f2e59fea_nohash_1.wav" \ * --output_c_file="yes_micro_features_data.cc" \ */constintg_yes_micro_f2e59fea_nohash_1_width=40;constintg_yes_micro_f2e59fea_nohash_1_height=49;constunsignedcharg_yes_micro_f2e59fea_nohash_1_data[]={244,226,245,223,234,213,228,208,194,110,95,116,102,0,137,161,183,173,137,116,133,157,151,156,128,110,128,0,68,78,78,90,68,68,78,102,95,78,95,78,210,188,209,183,204,188,201,191,166,119,90,107,110,107,175,157,179,168,182,145,152,164,171,165,136,143,122,68,0,78,90,90,110,90,102,99,90,68,78,68,223,186,179,123,182,110,196,171,159,110,102,95,90,99,160,134,125,136,153,152,164,134,164,151,141,136,99,90,90,90,78,78,102,119,102,90,110,90,68,51,177,175,211,172,183,0,95,68,129,102,68,85,114,105,110,85,102,95,140,51,85,51,95,90,143,116,90,78,78,51,107,85,68,0,68,51,90,51,68,0,164,117,193,120,156,0,138,51,90,0,51,0,51,85,0,0,51,0,0,0,0,0,114,0,85,78,90,51,0,0,51,85,99,85,107,68,90,85,78,0,51,0,110,0,68,0,0,0,51,0,51,0,0,0,68,90,107,0,68,0,0,0,68,0,51,68,0,78,68,0,51,0,78,68,90,68,78,51,51,0,0,0,0,0,0,0,0,0,0,0,90,0,0,0,0,0,51,0,0,0,0,0,0,0,0,0,0,0,51,68,0,0,78,0,78,0,78,0,51,0,0,0,0,0,0,0,0,0,0,0,51,0,51,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,51,0,51,0,51,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,51,0,0,0,0,51,78,0,0,51,51,0,0,0,78,0,213,170,192,180,196,188,173,131,173,116,137,105,159,127,0,0,0,0,127,164,165,161,170,164,185,197,195,167,134,138,159,134,136,105,51,0,99,0,51,0,228,215,229,218,237,215,228,210,237,222,239,211,208,211,234,218,220,209,225,219,235,222,245,225,245,224,243,223,241,218,237,224,234,213,221,193,197,164,157,128,227,188,232,196,220,220,240,219,234,213,234,211,231,218,233,213,239,215,228,207,229,206,224,208,226,207,232,210,225,208,230,199,227,206,210,205,218,174,178,141,235,208,220,206,225,203,233,203,225,167,205,199,208,190,221,204,223,207,225,188,225,197,215,188,199,183,225,195,224,200,216,178,208,188,215,202,214,183,176,140,198,150,211,194,203,120,175,188,204,189,219,192,223,202,216,186,203,185,210,182,214,183,204,170,204,125,184,187,206,185,198,182,210,161,202,198,218,173,145,120,188,183,205,168,200,170,210,177,187,190,209,193,193,166,210,162,175,119,174,147,182,161,181,134,176,143,187,165,186,149,185,141,192,181,202,123,170,143,144,78,149,0,208,182,170,78,170,0,117,51,156,99,195,170,200,130,152,68,175,141,173,134,194,132,189,164,198,134,173,117,171,149,183,181,185,99,153,117,125,0,166,0,173,117,144,0,117,102,188,120,193,166,197,68,163,119,169,99,134,0,162,0,164,68,171,116,126,0,120,68,68,0,105,0,159,95,150,51,90,85,0,0,131,0,105,0,145,51,170,51,120,0,107,0,145,85,160,0,85,0,0,51,149,0,78,0,0,0,0,0,0,0,90,0,112,0,78,102,122,0,0,0,0,0,105,0,0,0,0,0,0,0,0,0,112,0,164,120,143,0,0,0,0,0,51,0,90,0,78,0,0,0,0,0,110,0,139,0,112,51,0,0,0,0,0,0,0,0,0,0,0,0,102,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,107,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,78,0,51,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,51,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,127,110,133,0,167,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,132,0,190,194,202,0,197,187,161,0,0,0,0,0,0,0,0,0,214,213,223,203,218,189,200,122,78,0,0,0,0,0,0,0,0,0,0,0,191,210,231,197,226,217,238,216,236,207,199,0,0,0,0,0,107,122,155,160,214,215,236,202,235,203,225,191,203,188,199,194,212,127,51,0,174,188,219,196,228,221,240,207,235,220,241,219,237,207,212,142,95,0,139,78,162,177,197,183,211,199,235,208,238,215,227,207,211,201,224,213,226,192,213,170,223,205,234,221,245,225,242,220,245,221,239,221,238,213,226,180,159,112,176,159,208,202,213,191,205,191,225,197,238,219,224,201,227,200,221,201,225,203,212,195,229,210,228,210,239,216,226,212,233,205,225,200,229,207,222,151,147,119,179,185,230,218,223,192,202,136,205,177,223,204,228,215,232,209,221,189,221,205,209,200,226,209,229,205,235,192,209,198,228,190,206,185,207,187,214,175,177,184,220,195,214,207,230,184,205,159,208,184,189,169,224,213,219,199,229,203,216,205,222,204,224,206,231,208,231,176,197,184,216,193,211,139,212,195,231,164,166,195,217,182,208,190,217,179,205,68,182,119,195,168,182,136,204,179,193,158,182,140,188,154,197,169,190,99,184,0,125,0,131,0,99,68,179,85,190,184,213,203,223,202,212,190,209,138,178,0,159,51,128,51,105,0,139,51,179,125,185,114,171,128,175,132,181,174,155,0,0,0,90,0,125,0,176,188,227,217,244,215,234,221,239,192,224,210,0,0,134,0,51,0,105,0,105,0,143,90,192,119,175,147,141,51,184,110,85,0,0,0,0,0,0,0,151,139,201,203,232,203,226,208,236,206,230,212,0,0,0,0,0,0,0,0,0,0,0,0,169,0,119,0,78,0,0,0,0,0,0,0,0,0,68,0,0,133,200,180,220,197,228,201,221,184,213,193,110,0,0,0,0,0,0,0,0,0,78,0,164,0,0,0,0,0,107,0,0,0,0,0,0,0,0,0,0,0,150,164,202,182,224,197,211,179,212,193,134,0,0,0,0,0,0,0,0,0,85,0,150,0,85,0,95,0,0,0,0,0,0,0,0,0,0,0,0,0,102,90,193,160,203,164,200,178,205,174,116,0,0,0,0,0,0,0,0,0,120,114,123,0,114,0,145,68,0,0,0,0,0,0,0,0,0,0,0,0,102,68,199,170,195,180,208,176,200,164,0,0,0,0,0,0,0,0,0,0,0,0,110,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,142,102,172,110,186,167,185,147,189,154,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,177,0,158,136,197,155,189,166,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,85,0,155,90,175,117,175,138,202,165,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,51,0,139,0,120,68,51,123,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,119,0,78,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,};
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/// This is a standard TensorFlow Lite model file that has been converted into a// C data array, so it can be easily compiled into a binary for devices that// don't have a file system. It was created using the command:// xxd -i tiny_conv.tflite > tiny_conv_simple_features_model_data.cc#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_externconstunsignedcharg_tiny_conv_micro_features_model_data[];externconstintg_tiny_conv_micro_features_model_data_len;#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MICRO_FEATURES_TINY_CONV_MICRO_FEATURES_MODEL_DATA_H_
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"command_responder.h"#include"Arduino.h"// Toggles the built-in LED every inference, and lights a colored LED depending// on which word was detected.voidRespondToCommand(tflite::ErrorReporter*error_reporter,int32_tcurrent_time,constchar*found_command,uint8_tscore,boolis_new_command){staticboolis_initialized=false;if(!is_initialized){pinMode(LED_BUILTIN,OUTPUT);// Pins for the built-in RGB LEDs on the Arduino Nano 33 BLE SensepinMode(LEDR,OUTPUT);pinMode(LEDG,OUTPUT);pinMode(LEDB,OUTPUT);is_initialized=true;}staticint32_tlast_command_time=0;staticintcount=0;staticintcertainty=220;if(is_new_command){error_reporter->Report("Heard %s (%d) @%dms",found_command,score,current_time);// If we hear a command, light up the appropriate LED.// Note: The RGB LEDs on the Arduino Nano 33 BLE// Sense are on when the pin is LOW, off when HIGH.if(found_command[0]=='l'){last_command_time=current_time;digitalWrite(LEDG,LOW);// Green for left}if(found_command[0]=='r'){last_command_time=current_time;digitalWrite(LEDR,LOW);// Red for right}if(found_command[0]=='u'&&found_command[1]=='n'){last_command_time=current_time;digitalWrite(LEDR,LOW);// Yellow for unknowndigitalWrite(LEDG,LOW);// }//if (found_command[0] == 'd') {// last_command_time = current_time;// digitalWrite(LEDR, LOW); // Violet for down// digitalWrite(LEDB, LOW); // //}// if (found_command[0] == 'u') {// last_command_time = current_time;// digitalWrite(LEDB, LOW); // Blue for unknown// }}// If last_command_time is non-zero but was >3 seconds ago, zero it// and switch off the LED.if(last_command_time!=0){if(last_command_time<(current_time-3000)){last_command_time=0;digitalWrite(LED_BUILTIN,LOW);digitalWrite(LEDR,HIGH);digitalWrite(LEDG,HIGH);digitalWrite(LEDB,HIGH);}// If it is non-zero but <3 seconds ago, do nothing.return;}// Otherwise, toggle the LED every time an inference is performed.++count;if(count&1){digitalWrite(LED_BUILTIN,HIGH);}else{digitalWrite(LED_BUILTIN,LOW);}}
arduino_main.cpp
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"main_functions.h"// Arduino automatically calls the setup() and loop() functions in a sketch, so// where other systems need their own main routine in this file, it can be left// empty.
audio_provider.h
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_#include"tensorflow/lite/c/c_api_internal.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"// This is an abstraction around an audio source like a microphone, and is// expected to return 16-bit PCM sample data for a given point in time. The// sample data itself should be used as quickly as possible by the caller, since// to allow memory optimizations there are no guarantees that the samples won't// be overwritten by new data in the future. In practice, implementations should// ensure that there's a reasonable time allowed for clients to access the data// before any reuse.// The reference implementation can have no platform-specific dependencies, so// it just returns an array filled with zeros. For real applications, you should// ensure there's a specialized implementation that accesses hardware APIs.TfLiteStatusGetAudioSamples(tflite::ErrorReporter*error_reporter,intstart_ms,intduration_ms,int*audio_samples_size,int16_t**audio_samples);// Returns the time that audio data was last captured in milliseconds. There's// no contract about what time zero represents, the accuracy, or the granularity// of the result. Subsequent calls will generally not return a lower value, but// even that's not guaranteed if there's an overflow wraparound.// The reference implementation of this function just returns a constantly// incrementing value for each call, since it would need a non-portable platform// call to access time information. For real applications, you'll need to write// your own platform-specific implementation.int32_tLatestAudioTimestamp();#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_AUDIO_PROVIDER_H_
command_responder.h
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/// Provides an interface to take an action based on an audio command.#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_#include"tensorflow/lite/c/c_api_internal.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"// Called every time the results of an audio recognition run are available. The// human-readable name of any recognized command is in the `found_command`// argument, `score` has the numerical confidence, and `is_new_command` is set// if the previous command was different to this one.voidRespondToCommand(tflite::ErrorReporter*error_reporter,int32_tcurrent_time,constchar*found_command,uint8_tscore,boolis_new_command);#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_COMMAND_RESPONDER_H_
feature_provider.cpp
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#include"feature_provider.h"#include"audio_provider.h"#include"micro_features_micro_features_generator.h"#include"micro_features_micro_model_settings.h"FeatureProvider::FeatureProvider(intfeature_size,uint8_t*feature_data):feature_size_(feature_size),feature_data_(feature_data),is_first_run_(true){// Initialize the feature data to default values.for(intn=0;n<feature_size_;++n){feature_data_[n]=0;}}FeatureProvider::~FeatureProvider(){}TfLiteStatusFeatureProvider::PopulateFeatureData(tflite::ErrorReporter*error_reporter,int32_tlast_time_in_ms,int32_ttime_in_ms,int*how_many_new_slices){if(feature_size_!=kFeatureElementCount){error_reporter->Report("Requested feature_data_ size %d doesn't match %d",feature_size_,kFeatureElementCount);returnkTfLiteError;}// Quantize the time into steps as long as each window stride, so we can// figure out which audio data we need to fetch.constintlast_step=(last_time_in_ms/kFeatureSliceStrideMs);constintcurrent_step=(time_in_ms/kFeatureSliceStrideMs);intslices_needed=current_step-last_step;// If this is the first call, make sure we don't use any cached information.if(is_first_run_){TfLiteStatusinit_status=InitializeMicroFeatures(error_reporter);if(init_status!=kTfLiteOk){returninit_status;}is_first_run_=false;slices_needed=kFeatureSliceCount;}if(slices_needed>kFeatureSliceCount){slices_needed=kFeatureSliceCount;}*how_many_new_slices=slices_needed;constintslices_to_keep=kFeatureSliceCount-slices_needed;constintslices_to_drop=kFeatureSliceCount-slices_to_keep;// If we can avoid recalculating some slices, just move the existing data// up in the spectrogram, to perform something like this:// last time = 80ms current time = 120ms// +-----------+ +-----------+// | data@20ms | --> | data@60ms |// +-----------+ -- +-----------+// | data@40ms | -- --> | data@80ms |// +-----------+ -- -- +-----------+// | data@60ms | -- -- | <empty> |// +-----------+ -- +-----------+// | data@80ms | -- | <empty> |// +-----------+ +-----------+if(slices_to_keep>0){for(intdest_slice=0;dest_slice<slices_to_keep;++dest_slice){uint8_t*dest_slice_data=feature_data_+(dest_slice*kFeatureSliceSize);constintsrc_slice=dest_slice+slices_to_drop;constuint8_t*src_slice_data=feature_data_+(src_slice*kFeatureSliceSize);for(inti=0;i<kFeatureSliceSize;++i){dest_slice_data[i]=src_slice_data[i];}}}// Any slices that need to be filled in with feature data have their// appropriate audio data pulled, and features calculated for that slice.if(slices_needed>0){for(intnew_slice=slices_to_keep;new_slice<kFeatureSliceCount;++new_slice){constintnew_step=(current_step-kFeatureSliceCount+1)+new_slice;constint32_tslice_start_ms=(new_step*kFeatureSliceStrideMs);int16_t*audio_samples=nullptr;intaudio_samples_size=0;// TODO(petewarden): Fix bug that leads to non-zero slice_start_msGetAudioSamples(error_reporter,(slice_start_ms>0?slice_start_ms:0),kFeatureSliceDurationMs,&audio_samples_size,&audio_samples);if(audio_samples_size<kMaxAudioSampleSize){error_reporter->Report("Audio data size %d too small, want %d",audio_samples_size,kMaxAudioSampleSize);returnkTfLiteError;}uint8_t*new_slice_data=feature_data_+(new_slice*kFeatureSliceSize);size_tnum_samples_read;TfLiteStatusgenerate_status=GenerateMicroFeatures(error_reporter,audio_samples,audio_samples_size,kFeatureSliceSize,new_slice_data,&num_samples_read);if(generate_status!=kTfLiteOk){returngenerate_status;}}}returnkTfLiteOk;}
feature_provider.h
C/C++
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_#include"tensorflow/lite/c/c_api_internal.h"#include"tensorflow/lite/experimental/micro/micro_error_reporter.h"// Binds itself to an area of memory intended to hold the input features for an// audio-recognition neural network model, and fills that data area with the// features representing the current audio input, for example from a microphone.// The audio features themselves are a two-dimensional array, made up of// horizontal slices representing the frequencies at one point in time, stacked// on top of each other to form a spectrogram showing how those frequencies// changed over time.classFeatureProvider{public:// Create the provider, and bind it to an area of memory. This memory should// remain accessible for the lifetime of the provider object, since subsequent// calls will fill it with feature data. The provider does no memory// management of this data.FeatureProvider(intfeature_size,uint8_t*feature_data);~FeatureProvider();// Fills the feature data with information from audio inputs, and returns how// many feature slices were updated.TfLiteStatusPopulateFeatureData(tflite::ErrorReporter*error_reporter,int32_tlast_time_in_ms,int32_ttime_in_ms,int*how_many_new_slices);private:intfeature_size_;uint8_t*feature_data_;// Make sure we don't try to use cached information if this is the first call// into the provider.boolis_first_run_;};#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_FEATURE_PROVIDER_H_
main_functions.h
C/C++
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.==============================================================================*/#ifndef TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_#define TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_// Initializes all data needed for the example. The name is important, and needs// to be setup() for Arduino compatibility.voidsetup();// Runs one iteration of data gathering and inference. This should be called// repeatedly from the application code. The name needs to be loop() for Arduino// compatibility.voidloop();#endif // TENSORFLOW_LITE_EXPERIMENTAL_MICRO_EXAMPLES_MICRO_SPEECH_MAIN_FUNCTIONS_H_
Comments