Autonomous Tour Guide

Visitors always feel lost whenever they come for the first time to any large organization with many outdoor buildings. This organization could be anything such as a big company, university, museum or even a hospital.

The solution I propose for this problem is an autonomous tour guide using Sony's Spresense and the DonkeyCar. Existing solutions to this problem can be mobile apps or old-fashioned maps. They work, but they aren't as 'flashy' as an automated tour guide on wheels. It will be useful for the obvious reason that it will guide visitors in an organization, and also because it will increase the popularity of an organization using such a system. Think of it this way, a university which has an automated robot as a tour guide speaks for itself; it will make a great impression on the visitors (which will most likely be prospective students), and the same can be said for a tourist attraction such as a museum.

The tour guide using Spresense and DonkeyCar has two parts. The first part is the DonkeyCar platform, which uses the Raspberry Pi and machine learning to drive an RC car autonomously; the DonkeyCar will be trained to run on an outdoor track going past all the buildings of the organization. The second is Sony’s Spresense development board, which is connected to the Raspberry Pi, and uses geofencing to determine if the DonkeyCar is in range of a specific building, and then using the Spresense’s audio output, it will also give information to the visitors, who will follow. So to summarize, the tour guide is a DonkeyCar which has Sony’s Spresense development board mounted on it along with a speaker, to guide visitors in an organization.

The DonkeyCar is an amazing project but still in it's very early stages, so I would advise you at the start to not have very high expectations and to think of it realistically. It has a great, growing community and their Slack channel is also very informative. If you get stuck anywhere or even if you have just gotten to know this, I would recommend you to join that.

You can find more details about all the parts needed to build the DonkeyCar, and also where you can find a complete kit from this page.

How the Tour Guide works

This is a summary of how the tour guide works:

• You will need to decide upon an open, outdoor track (as GPS will not work indoors) for the tour guide (Spresense board on top of DonkeyCar).
• Then decide what points (in "Longitude, Latitude" format) you will need as beacons in that track. These are the buildings at which the tour guide will stop and inform visitors about the place.
• The Spresense will be mounted on top of the DonkeyCar (which will be moving on it's outdoor track) and it will constantly get its current position (in "Longitude, Latitude" format) using GPS. The Spresense board will also have an SD card and speakers connected it. The SD card will have audio files (.mp3) present on it, each describing a certain building or beacon.
• Think of an imaginary circle with a specified radius around each beacon. Whenever the Spresense board is inside this imaginary circle (or geofence), it will automatically stop the DonkeyCar and play the file for that beacon, and once the file is played, it will start moving again until it reaches the next beacon, and so on.
• Therefore visitors will follow the tour guide and gain knowledge about the organization and it's buildings.
• Once everything is working properly, the DonkeyCar's neural network can be trained to run on the outdoor track, making it an autonomous tour guide.

This should give you an idea of how the tour guide works:

How the DonkeyCar works

You must know by now, that the DonkeyCar uses the Raspberry Pi as the brain for the project, it runs the Donkey library and you will have to first assemble the DonkeyCar using this guide here. When assembled, you will need to install the software for it on the Raspberry Pi, as well as on another host PC using this guide here. The host PC is needed because the Raspberry Pi is not powerful enough to train an autopilot.

Note: I simply used the pre-built disk image for the Raspberry Pi which you can get fromhere.

Then you will need to calibrate your DonkeyCar's steering and throttle so as to make it work consistently. To do that, you need to find optimal PWM values for throttle and steering, and this process is described here. After calibration, you should test your DonkeyCar and try to get used to driving it. I recommend using a physical joystick rather than the web controller. I used a PS4 joystick through the browser and it worked without any issues.

Once you are confident enough to drive, you need to start to record data (without any mistakes as it would affect the autopilot). And once you get data recorded, (it is stored in the tubs folder in the mycar directory on the Raspberry Pi), you will need to transfer it back to your host PC to train.

Note: I am using Windows, so I had to use Cygwin with rsync and ssh installed. If you use Linux, you should have these installed depending on the distro being used.

The training is done using Tensorflow and Keras, and once you do it, you will get an autopilot file which you will need to transfer back to the Rapberry Pi. You can use this autopilot file to drive in two modes; one in which the DonkeyCar controls the steering only and throttle is controlled by the user (called Local Angle), and the other in which the DonkeyCar controls both steering and throttle (called Local Pilot). Everything about training an autopilot is described here.

The way the Donkey library works is that you can add your own sensors and actuators by making custom 'parts'. More details on adding your own parts can be found on these excellent YouTube videos here and here.

How the geofence works

First off, I must admit that I am not an expert with geographic co-ordinate systems and this is the first time I tried to something like this.

I wanted a way for the Spresense board to know when it enters a geographical area and after some research, I found out that the Spresense board supports geofencing but in the NuttX SDK only (described here). There's currently no support for that in the Arduino IDE (which I will be using).

A workaround for this problem, as indicated by Karl Komierowski from Sony, was to enable geofencing in the NuttX SDK and then do system calls directly from the Arduino code, but it would complicate things a lot so I didn't do it this way.

Instead I found this article on Adafruit which uses the Adafruit IO and FONA808, and describes how to create a geofence. It does that using a function called the haversine, which is used because it takes in account the fact that the earth is spherical. Anyone can say that it won't make a difference because the distances in this use case are pretty small to have an effect, but I used this just to be on the safe side.

This is the haversine function which will calculate the distance between two co-ordinates in "Latitude, Longitude" format (with one being the beacon's co-ordinate and one being the current location), and returns 1 if the current location is in the geofence, 0 if not:

//Check if point is in geofence. Returns 'true' if yes and 'false' if not
bool isPointInGeofence(float flat1, float flon1, float flat2, float flon2) {
 bool pointIn = false;
 float dist_calc = 0;
 float dist_calc2 = 0;
 float diflat = 0;
 float diflon = 0;
 diflat  = radians(flat2 - flat1);
 flat1 = radians(flat1);
 flat2 = radians(flat2);
 diflon = radians((flon2) - (flon1));
 dist_calc = (sin(diflat / 2.0) * sin(diflat / 2.0));
 dist_calc2 = cos(flat1);
 dist_calc2 *= cos(flat2);
 dist_calc2 *= sin(diflon / 2.0);
 dist_calc2 *= sin(diflon / 2.0);
 dist_calc += dist_calc2;
 dist_calc = (2 * atan2(sqrt(dist_calc), sqrt(1.0 - dist_calc)));
 dist_calc *= 6371000.0;
 if (dist_calc <= maxDistance) {
   pointIn = true;
 }
 return pointIn;
}

Setting up the DonkeyCar

You should build your car from the instructions at this page, to the point where you are able to drive, and to train and run an autopilot on your car. I will not go in the details, just follow the link above and you will find everything on how to do that.

There were a few issues I got into when installing the software, but googling your problem will give you an idea of how to solve it, so I won't be going into those details. If you have anything, feel free to comment.

Note: I will be using Windows on my host PC for running the DonkeyCar software.

A pinout of the Raspberry Pi for your reference:

This is my assembled DonkeyCar:

Choosing an outdoor track:

Once you are able to train and drive an autopilot, you need to choose the outdoor track; choose an open area (where you can get GPS signals) and something which the DonkeyCar can be trained upon. We'll be doing the actual training later; for now you only need to choose a suitable outdoor track.

Also, I will not go on about this because once you train an auto pilot on the DonkeyCar you will know what sort of track is good for it.

Adding the Spresense 'part' to the DonkeyCar Library:

Once you're confident with using the DonkeyCar and can drive, train and run an autopilot then you should add the Spresense 'part' to the DonkeyCar library. You can read more about DonkeyCar library 'parts' here.

To access your Raspberry Pi remotely, you will need an SSH client (I am using PuTTY), and to know the IP address of your Raspberry Pi (simply get that from your router’s settings).

Using PuTTY or any other SSH client, connect to the DonkeyCar's Raspberry Pi.

First of all, you need to know the serial port the Spresense will connect to on the Raspberry Pi. Without plugging in the Spresense, type this and hit enter on the Raspberry Pi:

ls /dev/tty* 

It should show all the available serial ports, and now plug in the Spresense to the Raspberry Pi, type the same command and hit enter. The new port is the Spresense's serial port, and you should note that down.

Now navigate to the 'mycar' folder (or whatever you called it) and create a python file at the root called "Spresense.py" with the following contents:

import RPi.GPIO as GPIO
import serial
import time
  
ser = serial.Serial('/dev/ttyUSB0', 115200)
oePin = 17
  
class SpresenseSerial:
   def update(self):
       print('Starting Spresense serial connection...')
       GPIO.setmode(GPIO.BCM)
       GPIO.setwarnings(False)
       GPIO.setup(oePin,GPIO.OUT)
   def run_threaded(self):
       if(ser.in_waiting >0):
           line = ser.readline()
           command = int(line)
           if(command == 1):
               GPIO.output(oePin,GPIO.HIGH)
               print('Stopping DonkeyCar')
           if(command == 0):
               GPIO.output(oePin,GPIO.LOW)
               print('DonkeyCar free to move')

Note: Remember to change the serial port you noted earlier and save it at the root of the mycar folder.

Also delete the "manage.py" file, and create a new one (also called "manage.py") with the following contents:

#!/usr/bin/env python3
"""
Scripts to drive a donkey 2 car and train a model for it.
Usage:
   manage.py (drive) [--model=<model>] [--js] [--chaos]
   manage.py (train) [--tub=<tub1,tub2,..tubn>]  (--model=<model>) [--base_model=<base_model>] [--no_cache]
Options:
   -h --help        Show this screen.
   --tub TUBPATHS   List of paths to tubs. Comma separated. Use quotes to use wildcards. ie "~/tubs/*"
   --js             Use physical joystick.
   --chaos          Add periodic random steering when manually driving
"""
import os
from docopt import docopt
import donkeycar as dk
#import parts
from donkeycar.parts.camera import PiCamera
from donkeycar.parts.transform import Lambda
from donkeycar.parts.keras import KerasCategorical
from donkeycar.parts.actuator import PCA9685, PWMSteering, PWMThrottle
from donkeycar.parts.datastore import TubGroup, TubWriter
from donkeycar.parts.controller import LocalWebController, JoystickController
from donkeycar.parts.clock import Timestamp
from Spresense import SpresenseSerial
def drive(cfg, model_path=None, use_joystick=False, use_chaos=False):
   """
   Construct a working robotic vehicle from many parts.
   Each part runs as a job in the Vehicle loop, calling either
   it's run or run_threaded method depending on the constructor flag `threaded`.
   All parts are updated one after another at the framerate given in
   cfg.DRIVE_LOOP_HZ assuming each part finishes processing in a timely manner.
   Parts may have named outputs and inputs. The framework handles passing named outputs
   to parts requesting the same named input.
   """
   V = dk.vehicle.Vehicle()
   spresense = SpresenseSerial()
   V.add(spresense, inputs=[], outputs=[], threaded=True)
   clock = Timestamp()
   V.add(clock, outputs='timestamp')
   cam = PiCamera(resolution=cfg.CAMERA_RESOLUTION)
   V.add(cam, outputs=['cam/image_array'], threaded=True)
   if use_joystick or cfg.USE_JOYSTICK_AS_DEFAULT:
       ctr = JoystickController(max_throttle=cfg.JOYSTICK_MAX_THROTTLE,
                                steering_scale=cfg.JOYSTICK_STEERING_SCALE,
                                auto_record_on_throttle=cfg.AUTO_RECORD_ON_THROTTLE)
   else:
       # This web controller will create a web server that is capable
       # of managing steering, throttle, and modes, and more.
       ctr = LocalWebController(use_chaos=use_chaos)
   V.add(ctr,
         inputs=['cam/image_array'],
         outputs=['user/angle', 'user/throttle', 'user/mode', 'recording'],
         threaded=True)
   # See if we should even run the pilot module.
   # This is only needed because the part run_condition only accepts boolean
   def pilot_condition(mode):
       if mode == 'user':
           return False
       else:
           return True
   pilot_condition_part = Lambda(pilot_condition)
   V.add(pilot_condition_part, inputs=['user/mode'],
                               outputs=['run_pilot'])
   # Run the pilot if the mode is not user.
   kl = KerasCategorical()
   if model_path:
       kl.load(model_path)
   V.add(kl, inputs=['cam/image_array'],
             outputs=['pilot/angle', 'pilot/throttle'],
             run_condition='run_pilot')
   # Choose what inputs should change the car.
   def drive_mode(mode,
                  user_angle, user_throttle,
                  pilot_angle, pilot_throttle):
       if mode == 'user':
           return user_angle, user_throttle
       elif mode == 'local_angle':
           return pilot_angle, user_throttle
       else:
           return pilot_angle, pilot_throttle
   drive_mode_part = Lambda(drive_mode)
   V.add(drive_mode_part,
         inputs=['user/mode', 'user/angle', 'user/throttle',
                 'pilot/angle', 'pilot/throttle'],
         outputs=['angle', 'throttle'])
   steering_controller = PCA9685(cfg.STEERING_CHANNEL)
   steering = PWMSteering(controller=steering_controller,
                          left_pulse=cfg.STEERING_LEFT_PWM,
                          right_pulse=cfg.STEERING_RIGHT_PWM)
   throttle_controller = PCA9685(cfg.THROTTLE_CHANNEL)
   throttle = PWMThrottle(controller=throttle_controller,
                          max_pulse=cfg.THROTTLE_FORWARD_PWM,
                          zero_pulse=cfg.THROTTLE_STOPPED_PWM,
                          min_pulse=cfg.THROTTLE_REVERSE_PWM)
   V.add(steering, inputs=['angle'])
   V.add(throttle, inputs=['throttle'])
   # add tub to save data
   inputs = ['cam/image_array', 'user/angle', 'user/throttle', 'user/mode', 'timestamp']
   types = ['image_array', 'float', 'float',  'str', 'str']
   #multiple tubs
   #th = TubHandler(path=cfg.DATA_PATH)
   #tub = th.new_tub_writer(inputs=inputs, types=types)
   # single tub
   tub = TubWriter(path=cfg.TUB_PATH, inputs=inputs, types=types)
   V.add(tub, inputs=inputs, run_condition='recording')
   # run the vehicle
   V.start(rate_hz=cfg.DRIVE_LOOP_HZ,
           max_loop_count=cfg.MAX_LOOPS)
def train(cfg, tub_names, new_model_path, base_model_path=None ):
   """
   use the specified data in tub_names to train an artifical neural network
   saves the output trained model as model_name
   """
   X_keys = ['cam/image_array']
   y_keys = ['user/angle', 'user/throttle']
   def train_record_transform(record):
       """ convert categorical steering to linear and apply image augmentations """
       record['user/angle'] = dk.util.data.linear_bin(record['user/angle'])
       # TODO add augmentation that doesn't use opencv
       return record
   def val_record_transform(record):
       """ convert categorical steering to linear """
       record['user/angle'] = dk.util.data.linear_bin(record['user/angle'])
       return record
   new_model_path = os.path.expanduser(new_model_path)
   kl = KerasCategorical()
   if base_model_path is not None:
       base_model_path = os.path.expanduser(base_model_path)
       kl.load(base_model_path)
   print('tub_names', tub_names)
   if not tub_names:
       tub_names = os.path.join(cfg.DATA_PATH, '*')
   tubgroup = TubGroup(tub_names)
   train_gen, val_gen = tubgroup.get_train_val_gen(X_keys, y_keys,
                                                   train_record_transform=train_record_transform,
                                                   val_record_transform=val_record_transform,
                                                   batch_size=cfg.BATCH_SIZE,
                                                   train_frac=cfg.TRAIN_TEST_SPLIT)
   total_records = len(tubgroup.df)
   total_train = int(total_records * cfg.TRAIN_TEST_SPLIT)
   total_val = total_records - total_train
   print('train: %d, validation: %d' % (total_train, total_val))
   steps_per_epoch = total_train // cfg.BATCH_SIZE
   print('steps_per_epoch', steps_per_epoch)
   kl.train(train_gen,
            val_gen,
            saved_model_path=new_model_path,
            steps=steps_per_epoch,
            train_split=cfg.TRAIN_TEST_SPLIT)
if __name__ == '__main__':
   args = docopt(__doc__)
   cfg = dk.load_config()
   if args['drive']:
       drive(cfg, model_path = args['--model'], use_joystick=args['--js'], use_chaos=args['--chaos'])
   elif args['train']:
       tub = args['--tub']
       new_model_path = args['--model']
       base_model_path = args['--base_model']
       cache = not args['--no_cache']
       train(cfg, tub, new_model_path, base_model_path) 

Make sure that the files are present at the root of the 'mycar' folder:

Once you do these modifications, you have added the Spresense 'part' to the DonkeyCar library and you can move on to setting up the Spresense board.

Note: Be sure to add the Spresense 'part' after you are done with testing and driving the DonkeyCar because after modifying the files, if you try running the DonkeyCar without connecting the Spresense (with the proper code) it may give you errors.

A small (but very important!) modification:

As the title says, this is step is very important, but very simple as well.

This is the PCA9685 servo driver which you will find on your DonkeyCar:

If you look closely, you will see the OE pin, which stands for output enable. This pin when LOW, enables the output and disables it when LOW. In the picture above, the OE pin is below the GND pin.

Because we need the Tour Guide to stop when it is near a beacon, you need to connect it to GPIO17 on your Raspberry Pi. I just used a female to female jumper wire to connect it. This is what it looks like after connecting everything:

Note: If you need to change the pin numberfor OE, youcan easilydo that from the Spresense.py file.

Setting up the Spresense board

In this section, I will talk about how to set up the Spresense board so that we can later connect it to the Raspberry Pi on the DonkeyCar.

This will have two parts, the first is the GPS and the second is the SD card and audio, and once you're done with those, then we need to upload and test if it is actually working.

Note: In this section I will only be setting up and testing the Spresense part of the project. Once this is confirmed to work then we move on to using it with the DonkeyCar.

These are the Spresense boards (main board plugged into extension board):

Before you go on, make sure that you have completed the getting started guide for the Spresense board from here.

Setting up the GPS:

Note: You will need to understand exactly how the geofence works so that you can decide upon suitable beacons. To do that, make sure that you read the "How the geofence works" section above.

Once you're done with that, the rest is pretty simple.

Open Google Maps from here and navigate to where you chose your outdoor track for the DonkeyCar.

Decide upon a suitable beacon in the DonkeyCar's path, such as a building or something similar. Now think of an imaginary circle with the center as the point you decided upon earlier with a radius (in meters). Decide the radius so that it should intersect with the DonkeyCar's path, and whenever the tour guide (DonkeyCar + Spresense) enters this imaginary circle or geofence, it will start to play the audio file for that specific beacon.

Note: You can use a different radius for a different beacon by modifying the code, but I would suggest you not to do this as it couldcomplicate things. Instead use one radius for all the geofences. In my opinion anything from 50 to 100 meters is reasonable, so decide your points taking this into account.

Once you are confident with your selection, click on that point once and it will show you the co-ordinate in "Latitude, Longitude" format, which you should note down.

Repeat the above for the number of beacons you need.

Note: For the code to work properly, you must use at least two beacons. I will be using two, but if you need more simply get more co-ordinates.

So when you're done, you will have noted a number of co-ordinates (depending on the number of beacons), and also the radius for the geofence. You will need them later when uploading the code.

Setting up the SD Card and Audio:

Earlier, you noted down the beacons, but now you need to get the audio files to be played when the tour guide enters a beacon.

For testing, you can just use any audio file, but if you want to actually implement this, you have two options to get the files. They are:

• Use a text-to-speech (or TTS) engine. There are many websites that provide this service online, and one I found is called "Text to MP3", which you can access here. As the name says, you can provide it with text and it will generate speech in a number of different voices.
• Record your voice using the Spresense board. It supports up to 8 mic inputs and can record audio in a high resolution. I didn't try this myself, but it is not that hard. You will have to connect a microphone and then upload the example sketch for microphone recording. Once you get the mp3 file, you can proceed further.

For testing, I would recommend the first option; just open this and put in some text and generate audio files (one for each beacon).

Once you have the files, connect an SD card (which will be used in the Spresense board) to your PC and format it as FAT:

Then connect the Spresense to your PC, open the Arduino IDE, click 'File' and navigate to Examples ==> Audio ==> dsp_installer. Here you will find two sketches (mp3_dec_installer / mp3_enc_installer) which you need to upload on your Spresense one by one, and follow the on-screen instructions in the Serial Monitor.

This basically installs the audio codecs on your Spresense board itself or on the SD card (I've done it on the SD card). Once you are done, you will find this on your SD card:

Note: This step is very important for the audio part to work.

Now copy over the audio files to the SD card and rename them. For example, the file you want to be played at the first beacon should be called "1.mp3" (with "1" being the filename and ".mp3" the file format), and so on.

So this is how my SD card looks like when I have two beacons (so two audio files, and the audio codecs folder):

After copying and renaming the files, remove the SD card and place it in the Spresense SD card slot on the extension board. Also make sure to connect the main board to the extension board (if you haven't done that already).

Now you need a way to play the files. Again you have two options for that:

• Using headphones or earphones connected to the 3.5mm jack on the Spresense extension board. Once everything is confirmed to be working, we can replace them with a speaker (connecting the AUX to the 3.5mm jack) and if the speaker is USB powered (most probably), it can be connected to the Raspberry Pi (of the DonkeyCar). I will be using this setup.
• The other option is to connect external speakers directly to the Spresense board. You can read more on this here, but it requires hardware modifications to the Spresense board, which I didn't want to play with. But if you want to go this way, all the details are given on the link above.

After you've put the SD card containing the audio files on the Spresense, and also attached the earphones to the 3.5mm jack, now we need to modify and upload the code.

Modifying and uploading the code:

You will need the co-ordinates you noted earlier, which are in "Latitude, Longitude format" and also the radius of the geofence (in meters).

First download the attached code for the Spresense and do the following changes:

In the arrays beaconLatitudes[] and beaconLongitudes[], add the latitudes and longitudes, so that at beaconLatitudes[0] is the latitude for the first beacon and at beaconLongitudes[0] is the longitude for the first beacon, and so on for the rest.

Note: The number of elements in beaconLatitudes[] and beaconLongitudes[] should be the same otherwise it would mean that the co-ordinates you provided for the beacons are invalid, and the code will not run.

If needed, change the value for maxDistance (in meters). This is the radius of the geofence, and set it to whatever you decided earlier. Anything between 50 to 100 meters seems reasonable to me for this purpose.

If you want debug output in the serial monitor, in the line #define mySerial you can change Serial2 to Serial but I would suggest you leave that as it is for now, and return try to change later if you're still having issues.

Once you're done just upload the code to the Spresense board.

Testing the code:

Note: The SD card with the audio files must be present in the Spresense board's SD card slot, otherwise it will not work. Earphones or headphones must also be connected to your Spresense board so that you can test if the audio files actually start playing whenyou enter a geofence.

To test if the code is working, you need to have a way to power the Spresense (maybe using your laptop, phone or powerbank) while you follow the path you decided upon earlier for the DonkeyCar and you must pass through the beacons. If everything is working you must hear the audio file for each beacon play as you go through.

Note: The Spresense will always check for beacons in the order you placed them in the arrays, (beaconLatitudes[0] and beaconLongitudes[0] will be beacon 1 and so on) and will check for one beacon at a time until it returns to the first. Feel free to play with that if you want, I have tried to keep it as simple as possible.

The Spresense main board has 4 LEDs which is very convenient, and so I have set:

• LED0 to turn on only when the Spresense is receiving values for the GPS
• LED3 to turn on only when the Spresense has detected that it it near a beacon and is playing the audio file for that beacon.

I am sure you will find the LEDs helpful enough in confirming everything works, but if you still feel there is an issue, you should change Serial2 to Serial in the line #define mySerial. Once you resolve the issue make sure to change it back to Serial2 and re-upload the code, otherwise you will have issues later.

This is a screenshot of a serial terminal application with the Spresense connected to my phone (via OTG). In the example below I tested the code with 3 beacons:

With everything being setup and configured, you can now move on to making the tour guide itself (DonkeyCar + Spresense combo).

The Tour Guide

This section is focused on connecting everything up, modifying the DonkeyCar as the Tour Guide and testing. By the end of this section, you will have connected both the DonkeyCar and the Spresense board to make the tour guide (and later we make it an 'autonomous tour guide'!).

ModifyingtheDonkeyCar:

This is the connection diagram:

And below are pictures of me setting up the DonkeyCar to work as an Autonomous Tour Guide.

You need to place the Spresense board (after uploading the modified code) on the DonkeyCar. I placed it on the small cardboard piece I mentioned earlier.

Connecting a USB cable between the Spresense and Raspberry Pi:

And now attaching the power bank, if you haven't already:

These are the speakers I will be using. They are powered by USB, as you can see below:

The speakers need to be placed on the DonkeyCar, and to do that, I used a spare piece of acrylic and used double-sided tape to attach it on top of the frame:

Placing the speakers on this acrylic sheet:

Finally:

With this, you have made your own Tour Guide and it's time to test it.

TestingtheTourGuide:

Now take it for a ride; I used a PS4 joystick as I found it to be more responsive and easier to control. As you drive it through the track, the Tour Guide must automatically stop at the beacons and play the audio files for that beacon, and then should move on once the file is played.

Once you're sure it's working, you can move on to making it an autonomous tour guide. To do that, you will need to train an autopilot on the same outdoor track, going through all the beacons.

Note: I have not talked about the actual track where you'll be training upon. Normally with the DonkeyCar, something like tape or ribbons are used which work, but I also have had some luck with an open track. Therefore I leave you to experiment with this. From what I have tested, an open track works with the condition that there is no noticeable change in the environment to the DonkeyCar's camera.

Drive a few laps, and once you're confident, delete the tub folder (to remove all previous data) and then drive carefully without hitting anything.

Collect enough good data and then transfer it to your PC with rsync:

rsync -r pi@<your_pi_ip_address>:~/mycar/data/  ~/mycar/data/

Note: I am using Windows as my host PC, so I use rsync with Cygwin.

Once you get the images on your PC, you need to train an autopilot, using:

python ~/mycar/manage.py train --tub <tub folder names comma separated> --model ./models/mypilot

And then move it back to your Raspberry Pi with:

rsync -r ~/mycar/models/ pi@<your_ip_address>:~/mycar/models/

Make sure the Spresense is connected to the Raspberry Pi with a USB cable and then run the DonkeyCar software with your autopilot with:

python manage.py drive --model ~/mycar/models/mypilot

It should turn on give you output same as it did before, but this time, we're using an autopilot.

Test the autopilot by going to the web interface on your host PC, and set it to Local Angle and limit the throttle to something less (30% works great for me). Then drive and it should steer automatically. Once the steering is good and it can navigate through the track without hitting, set the throttle mode to constant (the least value possible), and see the car navigate by itself.

Note: Local Pilot doesn't work for me for some reason; the car just sits there without doing anything, but be sure to check it out, maybe it does work for you. It says in the DonkeyCar documentation that as for now, the Local Pilot mode is not very reliable.

Hopefully it should work as intended and now, congratulations, you have built an Autonomous Tour Guide using the DonkeyCar and the Spresense board.

Explainingthecode:

The code is pretty simple and self explanatory.

The Spresense Arduino code is a combination and modification of the GNSS and audio player example codes. The Spresense board will store GPS co-ordinates of beacons, check whether it is located near a beacon, and if yes, it will send the integer value 1 through the serial port and start playing the audio file for that beacon. Once played, it will send an integer 0 through the serial port. One of the changes that I have done is to set all Serial output to be on the hardware serial port (Serial2) except sending the 1 and 0 for the DonkeyCar stopping and running; but you can modify it to get debug serial output if needed, as mentioned previously. The reason I did this change is because the Spresense communicates to the Raspberry Pi through it's serial port and I didn't want any interference in that.

On the Raspberry Pi side, we added the "Spresense.py" part in the DonkeyCar drive loop which waits for available serial data and once it receives the integer value 1 (when the Spresense is about to play the audio file) it will set the OE pin on the PCA9685 module as HIGH which will disable output on the module, effectively stopping the DonkeyCar. The DonkeyCar remains stopped until the Raspberry Pi receives an integer value 0 and it will set the OE pin on the PCA9685 back LOW, enabling the outputs and the DonkeyCar is free to move again.

Further Improvements

There can be a some improvements for this:

• Using a solar panel to charge the Tour Guide because it is outdoor there is a great potential for this. It can be a self sustained system, getting charged as it moves on it's track.
• Upgrading to a LiPo battery for increased drive time for the DonkeyCar.
• Recording audio from the Spresense board and playing those instead of the robotic sound generated by using online services.
• Using a display connected to the Spresense which can show text describing each beacon.
• There can also be many other applications for this, other than the DonkeyCar Tour Guide; for example, you can use it for safety applications related to GPS such as trackers.

Thanks for reading, and hope you liked my project. Feel free to give me your opinions and ideas about the project.