CityGuide

Overview

Have you ever went away on holidays and decided to take a tour of the city to see all the highlights that the town provides. If you, you must be familiar to the speed at which the tour goes; slowly detailing all landmarks on the track. Through rain and sun, in a cluster of people and umbrellas.

It is 2019, so it is time for a change. CityGuide aims to detach all the boredom of sightseeing allowing you to get through the landmarks at your pace, create your own course through the sights and choose the ones you want to visit. The guide is transformed into a device, that you just put in your pocket or strap to your belt.

The device will automatically detect where you are (so you won't have to press on the audio track number like in a museum) and what landmark you arrived at, telling you all you need to know about it. Just plug in the headphones and go.

Video

Image

Functionality

The device is easy to use by the client and easy to set up by the tourism company, in only a few minutes, the device is up and ready to run. During rain or sun shine, CityGuide will guide anyone through the city.

The device operates quite simply; using the Sony Spresense's onboard GNSS module, it retrieves the current location of the individual and processes it in order to see if they are at a certain landmark on the map. If the user is at a marked location; the appropriate recording for the landmark will be played, briefly informing the client about the background to what they are seeing. Below is a diagram displaying the functionality overview.

Setup Overview

The supplier, in our case the tourism company has to set up the device before it is utilised (a step-by-step guide into setting up the project is found in the Constructing the Project section). They have to plan a map of landmarks that the user could visit and record the tracks that will be played when the user arrives at the selected landmarks. All of this data then has to be placed onto an SD card and then inputted into the device.

Device Startup

The device will need to have an SD card inputted with all the data required at startup. Here, the device will read the given data, extract the location of each landmark, as well as the name and mp3 file to play when the user is at the given location and save the data to variables.

The device will then create a grid around all of the locations provided, the grid will be used to compare the current geolocation of the user to the locations of the landmarks in order to see if the user is at a landmark.

Playing the File

If the device can confirm that the user is at a landmark, it plays the file for that landmark and outputs the audio into the headphone jack. If two grids share the same locations, the device will play the file for the first landmark it hits.

Code Overview

Below is a diagram illustrating the project's code overview.

• Get File Data retrieves the data from the SD file provided by the manager of the device and processes it.
• Get GPS reads the current geolocation from the GNSS module.
• If(GPS) checks if new GPS data has been received from connected satellites and if there is a fix on the satellites.
• Compare with Locations compares the geolocation received with the geolocation of landmarks inputted by the user.
• If(atLandmark) triggers if the current geolocation is within a grid generated around an inputted geolocation.
• Play Audio plays the appropriate audio file for the landmark at which the client is.

Debug LEDs

The Spresense board is equipped with 4 debug LEDs of which 3 are used by this application for debug purposes.

• LED 1 is the first LED from the top of the picture above. It is used to indicate the current status of the device. It blinks when the device is setting up and then remains on while the algorithm is running.
• LED 2 is the GNSS status LED, it is used to indicate the current GNSS state. The LED blinks when the device is searching for satellites and is not yet receiving GPS data, it remains on when the device is locked onto satellites and is receiving reliable geolocation data and the LED turns off if the device is not using GNSS.
• LED 3 indicates if an mp3 file is currently playing. It will blink while the playback is setting up and will remain on when a file is playing, turning off when nothing is being played.

Playback Overview

The device has to stop the GNSS module when an audio file is running as both GNSS and Audio libraries cannot operate together, GNSS is resumed after the file has been played with all the satellites connected.

The user has the capacity to toggle the volume of the device with the provided potentiometer.

The Device in Action

Below are images showing the device in action, for a better look at the device's performance, check out the video above. But scan through these images too as they are quite good.

The device is designed in such a way that one location can only be visited once. Once the location has been reached, the file is played and the location is marked as seen, this prevents the file from playing the second time the client reaches the same location.

Benefits

The company switching to CityGuide will benefit in:

• A greater autonomy and ease of setting up
• Cut down on costs
• Rapid deployment
• A professional way to attract customers

The Individual using CityGuide will benefit in:

• A faster way to see the city
• Sightsee at their speed
• Choose the landmarks to visit
• Visit as a collective or alone

Constructing the Project

Step 1: Required Apparatus

This project does not require a lot of components as most of the needed sensors and modules are built into the Spresense board, the list of materials is below.

• 1, Sony Spresense Board
• 1, Potentiometer
• 1, Breadboard
• 1, SD Card
• Headphones/Earphones
• Jumper Wires

Step 2: Connecting the Circuit

All that has to be done is to connect the potentiometer to the Spresense board, an image of the schematics are below.

Soldering the Potentiometer

As the user is not going to carry a breadboard around with them, the potentiometer will have to be soldered to the Spresense board. Some images of the soldering below.

Step 3: Installing the Arduino Spresense Libraries

To start off, we need to include the libraries necessary to operate the Spresense board using the Arduino IDE. This is done easily using the board manager. Follow the steps below to guide you through the steps for setting up the environment.

Step 4: Installing the Port Utility

The guide below will display the steps needed to download and install the required Port Utility to send commands vice-versa from and to your Spresense. This guide has been designed for Mac Users, use this link for help installing the Port Utility on your PC.

To get started, download the appropriate Utility for your OS.

• Windows here
• Mac OSX here

Note that Arduino boards cannot operate with the utility installed, it is crucial to uninstall it to program and use Arduino or similar boards on your computer.

In order to uninstall the VPC driver. locate the uninstaller.sh shell script in the.dmg file and run it using Terminal on your Mac. To do so; open Terminal, then type ssh and drag and drop the uninstaller.sh file onto the Terminal window. You may be requested to input your password.

Step 5: Updating the Spresense's Firmware

In order to operate the project on your Spresense board; it needs to be updated to the latest firmware. The guide below illustrates the steps required to update the device's firmware.

Step 6: Installing the MP3 Decoder

In order to play MP3 files on the Spresense board, an MP3 decoder is required on the device. This decoder can be installed wither on the device's SPI Flash or on an SD card. We will install it on the Flash so we will not need a specific SD card in order to run the project. The steps below illustrate the steps to install the decoder.

Step 7: Setting Up the SD Card

In order to use the SD card with your project, it has to be appropriately formatted as FAT. The steps below show how this is done on a Mac.

Step 8: Preparing the Files

Now we have to set up the files on the SD card, these will be used in order to identify the landmarks for the device and supply it with the sound files to play at certain landmarks. The following files are needed on the SD card.

• Datalog.txt is the text file onto which the landmarks and related data is inputted by the manager
• mp3files used as the files that the device will play back when it arrives at the appropriate landmark.

The datalog.txt file is the heart of the device, all of the data that is needed for the project's operation is included in this file. The manager can add as many landmarks as they wish on the map, all they have to do is include a line for that landmark in the file. The following image illustrates the way a line dedicated to a landmark has to be formatted.

• GPO is the name of the landmark represented on the line. It can be anything but cannot contain spaces.
• 53.349430,-6.260235 is the geolocation of the landmark, getting this location is described in the guide below. The Latitude and Longitude are separated by a semi-colon and no space.
• fileA.mp3 is the file that is played when the user is at the specific location, in our case the GPO. Note that no spaces are permitted in the name, the .mp3 must be included at the end and the name cannot contain numbers.
• A space is included between each section.

With the diagram above in mind, we can start customising the datalog.txt file. The images below will guide you through the process.

Step 9: Acknowledging the Code

There are 3 main sections to the code, each one of these sections is quite big.

• Get SD Setup
• Get and Process GPS
• Play Audio File

They are all detailed below.

Get SD Setup

void getData()
{
 Serial.println("Getting Data");
 Serial.println("________________________________________");
 Serial.println("Locating File");
 Serial.println("  OK - Preparing Variables");
 int space = 0;
 int comma = 0;
 String localName[10];
 String localSound[10];
 String localRawLocation[10];
 String localLatitude[10];
 String localLongitude[10];
 Serial.println("  OK - Opening File 'datalog.txt'");
 myFile = SD.open("datalog.txt");
 Serial.println("  OK - Verifying Presence");
 if(myFile)
 {
   Serial.println("  Success - File Loaded");
   Serial.println("");
   Serial.println("Extracting Data");
   Serial.println("  OK - Beginning Final Extraction");
   while(myFile.available())
   {
     char c = myFile.read();
     if(c == '\n')
     {
       newLine++;
     }
     else if(c == ' ')
     {
       space++;
     }
     else
     {
       if(space == 0)
       {
         localName[newLine] += c;
       }
       else if(space == 1)
       {
         if(c == ',')
         {
           comma++;
         }
         else if(comma == 0)
         {
           localLatitude[newLine] += c;
         }
         else
         {
           localLongitude[newLine] += c;
         }
         localRawLocation[newLine] += c;
       }
       else if(space == 2)
       {
         localSound[newLine] += c;
         if(c == '3')
         {
           space = 0;
           comma = 0;
         }
       }
     }
   }
   // parse the local data into the struct
   for(int i = 0; i < (newLine + 1); i++)
   {
     landmark[i].name = localName[i];
     landmark[i].sound = localSound[i];
     landmark[i].rawLocation = localRawLocation[i];
     landmark[i].latitude = localLatitude[i].toFloat();
     landmark[i].longitude = localLongitude[i].toFloat();
   }
 }
 else
 {
   Serial.println("  Error - File not Present");
   Serial.println("  OK - Terminating Algorithm");
   Serial.println("________________________________________");
   Serial.println("");
   terminateLEDS();
   while(1) {};
 }
 myFile.close();
 Serial.println("  Success - Data Loaded Locally");
 Serial.println("");
 feedback();
 drawGrid();
}
  
void feedback()
{
 Serial.println("Data Feedback");
 Serial.println("  OK - Dumping All Data");
 Serial.println("");
 for(int i = 0; i < (newLine + 1); i++)
 {
   Serial.print("Struct "); Serial.println(i);
   Serial.print("  location name   "); Serial.println(landmark[i].name);
   Serial.print("  sound file      "); Serial.println(landmark[i].sound);
   Serial.print("  raw location    "); Serial.println(landmark[i].rawLocation);
   Serial.print("  latitude        "); Serial.println(landmark[i].latitude, 4);
   Serial.print("  longitude       "); Serial.println(landmark[i].longitude, 4);
 }
 Serial.println("");
 Serial.println("  Success - Data Dumped");
 Serial.println("________________________________________");
 Serial.println("");
}
  
void drawGrid()
{
 Serial.println("Generating Grids");
 Serial.println("________________________________________");
 Serial.println("Mapping Grid Around Co-ordinates");
 Serial.println("  OK - Looping through landmarks");
 for(int i = 0; i < (newLine + 1); i++)
 {
   landmark[i].maxLat = landmark[i].latitude + 0.001;
   landmark[i].minLat = landmark[i].latitude - 0.001;
   landmark[i].maxLng = landmark[i].longitude + 0.001;
   landmark[i].minLng = landmark[i].longitude - 0.001;
 }
 Serial.println("  Success - Grids Generated");
 Serial.println("");
 Serial.println("Dumping Grid Data");
 Serial.println("  OK - Dumping Grid Data");
 Serial.println("");
 for(int i = 0; i < (newLine + 1); i++)
 {
   Serial.print("Struct "); Serial.println(i);
   Serial.print("  max lat   "); Serial.println(landmark[i].maxLat, 4);
   Serial.print("  min lat   "); Serial.println(landmark[i].minLat, 4);
   Serial.print("  max lng   "); Serial.println(landmark[i].maxLng, 4);
   Serial.print("  min lng   "); Serial.println(landmark[i].minLng, 4);
 }
 Serial.println("");
 Serial.println("  Success - Data Dump Complete");
 Serial.println("________________________________________");
 Serial.println("");
}

This section of the code is composed of 3 loops. Overall, this section reads the data from datalog.txt and processes it.

• getData() reads the file and extracts the data from it, it figures the number of locations inputted and the details attached to all of these.
• feedback() prints the data received to the Serial Monitor to allow the user to debug the data file
• drawGrid() draws a grid around all of the inputted geolocations, if the client is inside one of these grids, the device plays back the appropriate sound file.

Get and Process GPS

void startGPS(bool hot)
{
 Serial.println("");
 digitalWrite(LED0, HIGH);
 Serial.println("Initialising GNSS");
 Serial.println("  OK - Setting Debug");
 gnss.setDebugMode(PrintInfo); // set the mode to print info
 Serial.println("Initialising Module");
 if(gnss.begin() != 0)
 {
   Serial.println("  Error - Module Failed to Initialise");
   digitalWrite(LED0, LOW);
   while(1) {};
 }
 else
 {
   Serial.println("  OK - Setting Elements");
   gnss.select(QZ_L1CA);
   gnss.select(QZ_L1S);
   Serial.println("  OK - Starting Positioning");
   if(hot)
   {
     if(gnss.start(HOT_START) != 0)
     {
       Serial.println("  Error - Start Failed");
       digitalWrite(LED0, LOW);
       while(1) {};
     }
     else
     {
       Serial.println("  Success - GNSS Setup Complete");
     }
   }
   else
   {
     if(gnss.start(COLD_START) != 0)
     {
       Serial.println("  Error - Start Failed");
       digitalWrite(LED0, LOW);
       while(1) {};
     }
     else
     {
       Serial.println("  Success - GNSS Setup Complete");
     }
   }
 }
 digitalWrite(LED0, HIGH);
 delay(500);
}
  
void processGPS(SpNavData *pNavData)
{
 char dataBuffer[STRING_BUFFER_SIZE];
 // print number of satellites
 snprintf(dataBuffer, STRING_BUFFER_SIZE, "numSat:%2d, ", pNavData->numSatellites);
 Serial.print(dataBuffer);
 // print the location data
 Serial.print(" ");
 if(pNavData->posFixMode == FixInvalid)
 {
   Serial.print("NO FIX  ");
 }
 else
 {
   Serial.print("FIX     ");
 }
 if(pNavData->posDataExist == 0)
 {
   Serial.println("No Geolocation");
 }
 else
 {
   gpsLatitude = pNavData->latitude;
   gpsLongitude = pNavData->longitude;
   Serial.print(gpsLatitude, 6); Serial.print(","); Serial.println(gpsLongitude, 6);
   checkLocation();
 }
}
  
void checkLocation()
{
 for(int i = 0; i < (newLine + 1); i++)
 {
   if(gpsLatitude <= landmark[i].maxLat && gpsLatitude >= landmark[i].minLat &&
      gpsLongitude <= landmark[i].maxLng && gpsLongitude >= landmark[i].minLng )
   {
     if(landmark[i].visited)
     {
       Serial.println("  [1/2] Already Visited");
       delay(500);
     }
     else
     {
       Serial.println("  [1/2] First Visit");
       delay(1000);
       Serial.println("  OK - Checking Data");
       Serial.print("  landmark name       "); Serial.println(landmark[i].name);
       Serial.print("  landmark sound file "); Serial.println(landmark[i].sound);
       Serial.println("  [2/2] Calling 'playFile' with audio file");
       Serial.println("  OK - Calling Function");
       Serial.println("");
       attachFile(landmark[i].sound);
       landmark[i].visited = true;
       break;
     }
   }
 }
}

This section is also composed of 3 loops. It is in charge of setting up and enabling the GNSS module and connecting and retrieving the data from satellites.

• startGPS() initialises the GNSS module, this is not done in the setup() loop as the GNSS module is disabled before playing back the Audio and has to be enabled again.
• processGPS() processes the data returned by the GNSS module after a read was made in the void() loop and extracts the geolocation.
• checkLocation() checks the geolocation received by the GNSS module against the grids generated earlier to check if the client is at a landmark.

Play Audio File

void attachFile(String fileName)
{
 Serial.println("");
 Serial.println("Setting Up File");
 Serial.println("________________________________________");
 Serial.println("Finalising Setup");
 digitalWrite(LED2, HIGH);
 delay(500);
 digitalWrite(LED2, LOW);
 Serial.println("");
 Serial.println("Locating File");
 Serial.println("  OK - Opening File");
 Serial.print("  OK - Locating "); Serial.println(fileName);
 soundFile = SD.open(fileName);
 if(!soundFile)
 {
   Serial.println("  Fatal Error - File Not Present");
   Serial.println("________________________________________");
   Serial.println("");
   delay(500);
   return;
 }
 else
 {
   Serial.println("  Success - File Located");
 }
 Serial.println("");
 Serial.println("Analysing Format");
 Serial.println("  OK - Getting Frames to Analyse");
 int err = theAudio->writeFrames(AudioClass::Player0, soundFile);
 Serial.println("  OK - Analysing Data");
 if(err != AUDIOLIB_ECODE_OK)
 {
   Serial.println("  Fatal Error - File Formatation is Bad");
   soundFile.close();
   Serial.println("________________________________________");
 }
 else
 {
   Serial.println("  Success - Formatation is Good");
 }
 Serial.println("________________________________________");
 Serial.println("");
 playFile(fileName);
}
void playFile(String fileName)
{
 Serial.println("Playing File");
 Serial.println("________________________________________");
 Serial.println("  OK - Pausing GNSS");
 gnss.stop();
 Serial.print("Playing "); Serial.println(fileName);
 theAudio->startPlayer(AudioClass::Player0);
 digitalWrite(LED2, HIGH);
 while(1) // main playback loop
 {
   Serial.print(".");
   int rawPot = analogRead(A0);
   int volume = map(rawPot, 0, 1024, -700, 0);
   theAudio->setVolume(volume); // set the volume to the position of the potentiometer
   int err = theAudio->writeFrames(AudioClass::Player0, soundFile);
   if(err == AUDIOLIB_ECODE_FILEEND) // end of file record
   {
     Serial.println("");
     break;
   }
   if(err)
   {
     Serial.println("");
     Serial.println("  Error - Playback Error");
     Serial.print("  OK - Error ID "); Serial.println(err);
     break;
   }
   usleep(40000);
 }
 theAudio->stopPlayer(AudioClass::Player0);
 soundFile.close();
 digitalWrite(LED2, LOW);
 Serial.println("  OK - Playback Terminated");
 delay(500);
 Serial.println("  OK - Restarting GNSS");
 startGPS(true);
 Serial.println("________________________________________");
 Serial.println("");
 delay(500);
}

This section of the code controls the mp3 file playback, it sets up the file and then plays it.

• attachFile() sets up the playback, it starts the playback procedure, checks for the presence of the file and verifies to ensure that the mp3 file is formatted correctly and the frames are readable.
• playFile() starts off by disabling the GNSS module to start the playback, finalises setup and then actually plays the file until it is done, the user can control the volume of the playback using the potentiometer while the file is playing.

Setting Up the Variables

Now that the code has been described, some variables have to be personalised in order to get the best experience out of the project. The first variable to edit is proDebug, this variable controls the serial prints. If it is enabled (set to true), the device requires to be connected to the computer and the Serial Monitor enabled. It is set to false by default as it has to be set to false to operate on the field.

The second variable does not need to actually be edited, but the index must be adjusted. A structure called Landmark is used to keep all the landmark related data, a code snippet with the structure is included below.

struct Landmark
{
 // strings
 String name;
 String sound;
 String rawLocation;
 // actual location
 float latitude;
 float longitude;
 // grid co-ordinates
 float maxLat;
 float minLat;
 float maxLng;
 float minLng;
 // extras
 bool visited;
};
  
Landmark landmark[10];

As seen above, everything from the name to if the location has already been visited is included. What has to be changed is the declaration below, which creates an array of instances of the structure. At the moment, there I an index of 10 landmarks, the manager must change this number to the number of landmarks they have included on the file. If there are less than 10, it can be left as it is. And that is it.

Libraries

• SDHCI - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain
• GNSS - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain
• Audio - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain

Final

The last step is to plug the device into the computer and upload the sketch. Power consumption wise, I have connected the Spresense board to a power bank, It should last a few trips before it needs to be recharged.

And then the last step is to enclose the project to make it portable and nice. I designed some quick schematics for the project's enclosure, I have attached them below.

I got the enclosure made and finished nicely. I ended up with a box that looked like this.

So after I got the raw box done, it was time to edit it and drill some holes for the potentiometer and to access the onboard SD Card module and the onboard headphone jack. The steps bellow show how to finalise the enclosure and fit the device into it.

So now, you are ready to go with the device on the field. Note that at a cold start, it takes about 10 minutes for the device to lock onto enough satellites to collect geolocation data. And that is it!

Background

I spent a while thinking about a great project using the Spresense board. The problem was that you could make anything with the Spresense board, so it was quite hard to come up with the perfect idea. But I think that I achieved my idea of a great project. A total change in the way tourism is done. Building on that device you get in a museum, that device with earphones that tells you about the thing you are looking at when you insert the ID of the thing you are looking at.

It strips that idea apart, replacing the manual button selection with an automatic system that plays back when the client arrives at a location. Creating the sightseeing of the future, at your pace.