Beyond Believable(Intelligent nRF5340-DK Virtual Assistant)

Introduction:

The aim of this project is to show case how quickly someone can use proof-of-concepts to quickly prototypes case-studied or scenarios around the Nordic nRF5340 development kit. We are building resources around the nRF5340 board that we could use in the education settings including but not limited to artificial-intelligence, machine-learning, natural-language-understanding(nlp), edge-computing, Internet-of-Things and cellular inter-communication(LTE, LoraWan, etc…). We also hope to be able to showcase device-to-device communication between wearable devices that we have selected for this project.

About Nordic nRF5340 Kit:

The nRF5340 development kit is a dual-core Bluetooth 5.2 system-on-chip (SoC) device that supports Bluetooth Low Energy, Bluetooth mesh, NFC, Thread and Zigbee

[1]

. According to the data-sheet, the board features some of the followings:

An extensive range of wireless protocols.

Bluetooth Low Energy with:

• high-throughput 2 Mbps
• Advertising Extensions and
• Long Range
• Mesh protocols like Bluetooth mesh
• high-throughput 2 MbpsAdvertising Extensions andLong RangeMesh protocols like Bluetooth mesh
• Bluetooth Low Energy with:high-throughput 2 MbpsAdvertising Extensions andLong RangeMesh protocols like Bluetooth mesh
• An extensive range of wireless protocols.Bluetooth Low Energy with:high-throughput 2 MbpsAdvertising Extensions andLong RangeMesh protocols like Bluetooth mesh
• Thread and Zigbee can run concurrently with Bluetooth Low Energy,
• enabling smartphones to provision, commission, configure and control mesh nodes.
• NFC, ANT, 802.15.4 and 2.4 GHz proprietary protocols are also supported.

128 MHz Arm Cortex-M33 CPU with:

• 1 MB Flash + 512 KB RAM
• 64 MHz Arm Cortex-M33 CPU with
• 256 KB Flash + 64 KB RAM
• Bluetooth Low Energy
• Bluetooth Direction Finding
• Advanced security
• USB, QSPI, HS-SPI
• 105 °C extended operating temperature
• 1.7-5.5 V supply voltage range
• 1 MB Flash + 512 KB RAM64 MHz Arm Cortex-M33 CPU with256 KB Flash + 64 KB RAMBluetooth Low EnergyBluetooth Direction FindingAdvanced securityUSB, QSPI, HS-SPI105 °C extended operating temperature1.7-5.5 V supply voltage range

As you can see from the above datasheet extract, it could be pretty up in the bag with nRF5340 kit. The kit is features full that you could deploy it in many possible environments that spans from retails, homes, transport, medical, media/entertainment, sports, education and many more… You just need to do a little research, come up with a proof-of-concepts and to prototype your idea.

Unboxing nRF5340:

My nRF5340 kit came with some peripheral components that we will be using along with the board. But first what does the board actually look like?

There will more unboxing of other peripheral components that we are interfacing with nRF5340 kit. Please stay tune in

It is a good step and way to get familiar with the kit and its attachments. Having the manual, data-sheet, user-guide, programming manual beside you will make your life much more easier than predicted. Below is a labeled screenshot of nRF5340 board showing connection pins, power supply points(usb, vdd and li-Po) plus an on/off power-switch and a good range of power input from 1.75 to 5.5 volts the kit comes with lithium CR2032 coin cell battery.

Available four(4) user buttons, debugging points/connections, external memory, RF antena, NFC antena, and many more…

One noticeable feature of nRF5340 board is an onboard Segger Jlink programming/debugging chip. This makes this board versatile in terms of being programmable with OpenOCD, SWD, etc… couple with debug-in and debug-out connection, many RF shortwave measurements can be carried in suite

The board also feature an Arduino Uno Rev3 form factor making things possible to retrofit expansion shields and use them with this nRF5340-DK.

The back of the kit reveals various labeled connections and information that could be followed when customising the kit for a user-case or for domain-specific applications.

📷You need take time to study those labels for available GIO and functions available. Also take note of the word “short” to bridge/enable certain feature.

Your first stop on getting started with kit is the documentation website setup specifically for nRF52340-DK here: https://infocenter.nordicsemi.com/index.jsp?topic=%2Fug_nrf5340_dk%2FUG%2Fdk%2Fintro.html

Everything you need but afraid to ask is here, its the play ground, tinkering and how-tos for this development kit.

We will comeback to this again when we will look at the dual cortex-M33 application core and the network core processors, registers and pins assignments that came with the kit

Getting Started with nRF5340:

Resources for working with nRF5340 are on the product page here: https://www.nordicsemi.com/Software-and-tools/Development-Kits/nRF5340-DK

There is a GitHub repositories for hacking and tinkering with the kit:

https://github.com/NordicSemiconductor#start-of-content

https://github.com/nrfconnect

https://github.com/nRFCloud

https://github.com/nRFCloud/sdk-zephyr

https://github.com/nrfconnect/sdk-zephyr

and many more…. We will be using some of these repositories for our story-telling on how we explore this features laden kit.

Out-of-Box features:

What features/demos made available right out of the box?

1.) Connect and test:

the following five(5) quick steps to connect, power-up and test the board using pre-programmed code:

• We use a micro USB 2.0 cable to connect the nRF5340-DK to Linux box USB port. This will provide power to the board and supports target programming.
• Next we connect the NFC antenna to the board. The connector is located next to user button 4 and marked as "NFC". See our screenshot below here:

• We then turn on the power switch which is just by the board bottom left hand side close to the micro USB connector.

We can confirm that the board boots up as expected, LED 1 started to give a pulsating light. We play each of the four(4) user button and their

corresponding LED light showing pulsating light when each button is pressed.

End of our Connect and Test routines.

Our next test will be to run some of the samples that were already made available for nRF5340-DK. Running a precompiled programs will let us test the board working operation. This is a good step in the right direction as we have got only 1 board that we are tinkering and kicking around with.

The Heart Rate Demo: this is a precompiled demo program that is available on the board documentation site. What we did was as stated below:

We download the Heart Rate Monitor (HRM) demo application from https://www.nordicsemi.com/-/media/Software-and-other-downloads/Dev-Kits/nRF5340-DK/Peripheral_Heart_Rate_nRF5340_DK.zip

Next it was extracted it into our project folder. The folder contains hex files for both the application and the network core binaries

Now we install the nRF connect for Desktop from this link: https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Connect-for-desktop/Download#infotabs

Our package is for Linux environment. This was download and setup for Ubuntu 20.04 platform.

According to the website, the nRF Connect for Desktop is a cross-platform tool that enables testing and development with Nordic’s products including the nRF family e.g. nRF51, nRF52 and nRF53 products.

It is supposedly to make test and programming those kits a snap-on and easy going tool for Nordic development kits and dongles, and nRF91 (our Thingy:91 kit) Series development kits. nRF Connect will automatically detect which kit is connected to your computer and upload the needed firmware. Finger cross :)

Note: On macOS and Linux, you must download and install the https://www.segger.com/downloads/jlink/#J-LinkSoftwareAndDocumentationPack separately. Since I have been working with hardware always, I already have this installed.

Now open your terminal box, navigate to the downloaded folder and do the following to get nRf connect Desktop installed:

chmod a+x nrfconnect361x8664.AppImage

📷now issue the command:./nrfconnect361x8664.AppImage

Follow the onscreen installation information.

Click on the Accept button on the above screenshot to continue

You will get a pop-up dialog window with installation packages options. Make your own choice for what want you need.

As per my requirements, it seems that all of them have to be installed both for our nRF5340 and the peripheral (e.g. Thingy:91, Dongle) that we will attaching to the board later on in the project. So we are installing everything. Luckily we have more a terabyte left on our hard disk storage. You need to manually install each of these packages. I wish there is an install everything button implemented here.

Do not forget to create shortcuts for your installed apps. Click on the drop-down button to access to do. See below for a tip on this:

They will be placed on your Linux box desktop if you have one:

at this point in our exploration, the Toolchain, the Programmer and the getting started assistant tools will be our horizon. With the nRF connect Desktop, nRF conncet Toolchain and nRF connect Programmer

done, our next step is to begin testing with the Heart Rate Monitor and other sample programs.

Next access your computer desktop and double click on the nRF connect programmer to run it. Well the desktop shortcuts did not work so we have to run the nRF connect Desktop, add it to our taskbar.

So open the programmer for nRF connect Desktop window.

You will see a window like shown below here:

The programmer window shows an option bar to select your device.

Below the bar is main area to connect to display memory contents, beside this, is where you can drag and drop the hex file to be uploaded to your device. Other programming routines like adding, erasing, reset, read/write and protection of firmware are available as well.

📷📷

On the select a device, click on PCA10095. Note: the green dot next to the device name show that connection had been established.

the programmer will poke your device, query and load the device information on the log pane.

You will see information about the status of your device e.g. if it is protected or not as you can see below:

Let us load the Hear Rate Monitor hex file onto our nRF5340 board.

Navigate to folder and upload the application hex (peripheral_hr.hex).

There will be two HEX files in the folder, the one for the application core (peripheral_hr.hex). The other (hci_open_amp.hex) is for the network core.

Drap and drop the hex file onto the programmer in the direction of the arrow

Tick the auto read memory box

click on erase & write option to get the hex upload done.

At the end you will get an environment like the shown below:

We to reset the board.

We then need to get the mobile App install on our phone and connect with it with nRF5340 board.

For now we will use the ready made Android.apk but we will customising the one on GitHub for our project later on.

The link for the App is here: https://www.nordicsemi.com/Software-and-Tools/Development-Tools/nRF-Toolbox

There are iOS and Android there including their source code.

Note: that the.apk is also available on the GitHub repository.

See it here: https://github.com/NordicSemiconductor/Android-nRF-Toolbox/releases/download/v2.9.0/nRF.Toolbox.2.9.0.apk

screenshot here

After installing Android App, run it

Reset (press reset button) your nRF5340 board for it to load and run the application

Turn on your phone bluetooth

Open nRF Toolbox.

Tap HRM icon and select Connect.

Select Zephyr Heartrate Sensor.

A simulated heart rate and battery charge percentage is displayed.

Next Press the reset button.

The development kit will now restart and run the application. Note that while restarting, the JLINK drive will be disconnected.

Download and install the Nordic nRF Toolbox app from Google Play or App Store.

STP FOR HRM/BLUETOOTH

============================================

Our next step in this journey is to embark on getting Zephyr and MicroPython bundle together to work in concert and harmonised them and use them on nRF5340 board.

First there was Zephyr rtos:

in the project nordic toolchain folder, we create a folder named named ncs (short name for nordic connect sdk). Here we are using the nordic zephyr rtos version from the GitHub.

next we run the command:

west init -m https://github.com/nrfconnect/sdk-nrf

inside the folder, this will download four GitHub repositories for us including the zephyr rtos version for nordic from the nrfconnect repositories.

Now issue the following commands:

west update

west zephyr-export

to get zephyr rtos environment variables initialised for use.

This will take some time so be patient while west tool is doing all those work for your. It will clone the zephyr, mcuboot, nrfxlib, several modules like openThread, mbedtls, alexa, civetweb, lvgl and many modules. It just pretty packed with bells and whistles….

When completed, you get console message about the export environment variable:

next from inside the ncs directory,

cd nrf

now do: => git checkout v1.5.1

next run west update for the new branch. Once again wait for the steps to run to finish. We are almost there!

Some commits will be left behind but don’t worry about them for now.

It was noted that we will need additional python 3 tools, so still inside ncs folder, we run the following commands:

pip3 install --user -r zephyr/scripts/requirements.txt

pip3 install --user -r nrf/scripts/requirements.txt

pip3 install --user -r bootloader/mcuboot/scripts/requirements.txt

After installing Segger Embedded Studio, Arm Toolchain (9-2019-q4-major of the GNU ARM), I ran emStudio from the Embedded Studio bin folder to finally set up the rest of the toolchain including zephyr but less Micropython which will be our next and final step before we start to build various application and learning platforms that we will need. Stay tuned in.

See below screenshot for Segger Embedded Studio settings:

Let us do the final setting for our nRF5340 board.

From the Board Name drop-down:

select nrf5340dk_nrf5340_cpuappns

project: select zephyr => hello_world

click the OK button in the above window.

Few more settings before we build our first hello-world application:

Click Tools > Options and select the nRF Connect tab.

Next specify zephyr folder in SES, you must specify the following information:

• Zephyr Base - the location of your cloned Zephyr repository (e.g. /home/sanyaade/Compilers/nordic/ncs/zephyr)
• GNU ARM Embedded Toolchain Directory- the location of your GNU ARM Embedded Toolchain installation (for example, /home/sanyaade/Compilers/ArmGcc/gcc-arm-none-eabi-9-2019-q4-major)

To build our hello-world application, from the build menu, select build zephyr merged hex or hit the F7 key on your keyboard:

You should a build zephyr hex file located in the project folder as shown below:

We will then go ahead and flash it with our board, get putty to see the output from putty terminal.

With the Segger Embedded Studio and nRF Connect Desktop added to our belt of tools, we now have to build environment at hand. You can compile, build and load zephyr firmware from any of these environments

from embedded studio, select tool

Next run putty with serial, buadrate: 115200, in my case I could get serial output from dev/ttyACM2 as shown below:

as you can see from above pressing the reset button shows that our nrf5340dk reboots and loads our just built app which is a non secure App. Image is loaded at 0x8000 and the Hello World application runs. Huh!

This is a big sign of relief and we can build few more applications.

Our next incarnation is to now build zephyr plus MicroPython. Its midnight in London but I am working through the night. Soda water, toffee, biscuits and coffee with make it till dawn here….

Building MicroPython Port to Zephyr RTOS for nRF5340-DK

First we need the current release of MicroPython downloaded for our use. As of this project, the release was tag v1.15.

Fork or clone? Since this will be a continuous integration (CI) and continuous development cycle (CD) couple with my interest in zephyr for my teaching and research then a fork of MicroPython is the right direction to follow. Also creation of micropython libraries with ports from other MicroPython clones like AdaFruits CircuitPython, OpenMV, EduBlock, etc…. Also to use this project as a base for ports for other nRF development kits.

NOTE: MicroPython can be made to run on nRF5340-DK without zephyr-RTOS but since we want to take advantages of all nRF kits that zephyr-rtos already has, possible expansions, enhancements and maintainability, it is sensible to go zephyr-rtos way. Also bindings for various Zephyr subsystems will be added

Navigate to the ncs folder that we set-up for this porting project. The one we are using above.

On the terminal, run the following commands:

git clone https://github.com/micropython/micropython.git

Next switch to MicroPython v1.15 tag:

cd micropython

git checkout v1.15

since there is no MicroPython port for nRF5340-DK yet we have to clone nRF9160 board information as a starting point for porting.

NRF5340-DK, nRF9160-DK, nRF52840-DK are hardware compatible with arduino uno rev 3 compatible which means our development kit should work at software interface level. Thank you Nordic!

We clone the MicroPython board info for nRF9160, rename it to nRF5340

1.) rename:

2.) rename:

3.) rename:

4.) edit the followings board header, mk, pins files:

Edit those three files customise them for PCA10095 which is the hardware reference name for the nRF5340-DK.

Do same for:

NRF9160 => NRF5340

you should end up with:

#define PCA10095

#define MICROPY_HW_BOARD_NAME "PCA10095"

#define MICROPY_HW_MCU_NAME "NRF5340"

#define MICROPY_PY_SYS_PLATFORM "NRF5340-DK"

Next modify mpconfigboard.mk as followed:

MCU_SERIES = m33

MCU_VARIANT = nrf53

MCU_SUB_VARIANT = nrf5340

LD_FILES += boards/nrf5340_1M_512k.ld

NRF_DEFINES += -DNRF5340_XXAA -DNRF_TRUSTZONE_NONSECURE

We may need to change the pin.csv file but this means that we need to compare the two development kits together for similarities and differences.

For now the pin.csv looks like shown below:

P0, P0

P1, P1

P2, P2

P3, P3

P4, P4

P5, P5

P6, P6

P7, P7

P8, P8

P9, P9

P10, P10

P11, P11

P12, P12

P13, P13

P14, P14

P15, P15

P16, P16

P17, P17

P18, P18

P19, P19

P20, P20

P21, P21

P22, P22

P23, P23

P24, P24

P25, P25

P26, P26

P27, P27

P28, P28

P29, P29

P30, P30

P31, P31

=======================================================================