MuSa, the experience manager

Presentation of the project

MuSa is a project developed by Silvia Del Piano,Stefano Foti, and Gianmarco Zizzo for the Internet of Things 2020 course, at Sapienza, University of Rome.

The aim of this project, thought at first for "Museo dell'Arte Classica Sapienza", is to provide personalized tours and to give information to the users through their smartphones.

How MuSa comes to life - the problem

We projected MuSa thinking about many commons issues that a visitor has when he visits a museum, such as:

• limited time to take a full visit
• meeting on his path of some pieces of arts that do not match with his interests
• the desire of having more details about an opera, but at the same time he does not like to have a physical or audio guide for several reasons

At the same time, we thought about the curators:

• the need to have some statistics about the most appreciated artworks and about the visitor's paths, in addition to the fact of offering some services that tempt the visitors to come back another time

Since the basic concept of MuSa is to put the user at the center of the experience, we started from these points for building something that could satisfy the user needs.

Previous experiences of this topic

The topic of personalized tours has been already experienced in smart museums, but we hardly found something modern and fully developed.For example, most tour modalities contain a fixed list of artworks, which is the same for everyone or for visitors from the same predefined user groups (e.g. groups of tourists, students, experts). The second problem is the lack of connection between online tours and on-site/multimedia tours, which are usually separated into two tours without any connections. These two problems became the main challenges in building personalized tours.The interaction with artworks and the use of beacons in smart museums has been, obviously, already experienced too; anyway, there are a lot of modalities to give life to these two aspects.We made a deep analysis of previous experiences in our documentation, with related references.

MuSa solution - overview

MuSa collects and analyzes information about visitors and their tours and proposes custom journeys, to offer a new way to enjoy the art.At the very beginning, every user should fill a short survey to collect information about his personality. Then MuSa will ask him if he wants to enjoy a customized tour or just help with data collecting. Of course, the user may also choose to not use the system at all. When a user wants just to contribute with data collecting he will start traveling on his own inside the museum with his smartphone collecting data in a non-invasive way inside his pocket. When a user wants to enjoy the tour with MuSa, indeed, he will just start exploring the museum, following the customized suggestions that MuSa will show him.

Architecture

In this section, you can find a list of all the main hardware and software that are present in our project.

• Network diagram
  Before analyzing the single hardware and software components, it is necessary to give a look at the whole network diagram.Near each piece of art or cluster of artworks, there is an ESP32 board that catches beacons from the user's smartphone, through the Android App. The ESP32 boards, in a variable time, send the beacons to the Raspberry Pi gateway; the gateway, every five seconds, sends the pre-processed data to the cloud, that hosts the backend, which at the end will answer to the app requests.

Now we will analyze the components of the system one by one:

• ESP32
  ESP32 is a series of low-cost, low-power system on a chip microcontrollers with integrated Wi-Fi and dual-mode Bluetooth. It is capable of functioning reliably in industrial environments and is powered by advanced calibration circuitries, that allow it to dynamically remove external circuit imperfections and adapt to changes in external conditions. Engineered for mobile devices, wearable electronics, and IoT applications, ESP32 achieves ultra-low power consumption. It can perform as a complete standalone system and it can interface with other systems to provide Wi-Fi and Bluetooth functionality.In our project, the ESP32 has a key role. We need an ESP32 near each artwork firstly for receiving beacons from users' smartphones and tracking them; then, this board sends every five seconds a message with the collected data to the main gateway (a Raspberry Pi).

• Raspberry Pi
  The Raspberry Pi is a series of small single-board computers. Its hardware has evolved through several versions that feature variations in the type of the central processing unit, amount of memory capacity, networking support, and peripheral-device support. Anyway, it is very powerful. In our project, the Raspberry Pi serves as a gateway between the ESP32s and Microsoft Azure; it pre-processes the received data, doing some intense Edge computing (further information in the "IoT aspects" section), and sends them to the Cloud, through the Azure IoT Hub.

• LedRGB
  We used a common RGB led and, of course, there are a lot of other solutions that could substitute it. In our project, the RGB led is turned on for green color by the gateway during data processing, using MQTT messages. The red color is reserved for issues, startup phase, and status, so it is managed directly by the board.
• Microsoft Azure Cloud Service
  Azure is a cloud computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centers. It provides software as a service (SaaS), platform as a service (PaaS) and infrastructure as a service (IaaS) and supports many different programming languages, tools, and frameworks, including both Microsoft-specific and third-party software and systems. For our project, the choice to use Azure as Cloud service is the result of our four assignments for the same IoT course, which allowed us to experiment with several Cloud services and compare them. We think that Azure offers more useful functionalities than ThingsBoard, and we think that is easier to use than Amazon Web Services.

• MQTT/MQTT-SN
  MQTT is a lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks.In our project, we use MQTT-SN for the communications between the ESP32 boards and the Raspberry Pi, and MQTT for between the Raspberry Pi and the Cloud.
• Bluetooth Low Energy (BLE)
  Bluetooth Low Energy is a wireless personal area network technology aimed at novel applications in the healthcare, fitness, beacons, security, and home entertainment industries.In our project, we used BLE for the beacons exchange between the devices, aimed at the indoor positioning. We did some extra additions for trying to improve the precision of the tracking, and we will discuss them later.

• Android App
  MuSa's services are provided through an Android App developed in Java. When a visitor wants to use MuSa, first of all, the app profiles the user through a welcome questionnaire, then suggests him a customized tour, and if the user accepts it, he will receive information and details for each artwork present in his tour, when he is near to it.

IoT elements

Since this project takes life from an Internet of Things course, it is filled by IoT elements and topics.

• Data collection and data analysis are two of the main goals of the IoT.With MuSa, we collect information about visitors and users for several reasons like to provide personalized tours and improve those tours through visitor's behavior. Even if it is not planned in our project, the curators may use the collected data also for several other aspects, like reorganize the museum's artwork positioning, simply analyzing those data.
• Edge computing is a widespread computing paradigm used in IoT; we pre-process the data collected to send them to the cloud in a better-structured way and to save on the number of messages sent, to better exploit the Cloud's free-plan.When one of the ESP32 catches, every second, users' beacons stores them temporarily and in a variable time, that depends on how many users there are in the museum, until to a maximum of five seconds, sends them to the gateway. This is already a sort of pre-processing since that if this data were sent directly to the Cloud every time, the free-plan would be gone much earlier. We did further calculations about the message's saving in our documentation.When the messages arrive at the Raspberry Pi, it will do another pre-processing of the data, since it applies a Kalman Filter those data for avoiding noise or distortions. We gave further explanations about the Kalman Filter in our documentation.
• We use two different types of Boards: a Raspberry Pi and an ESP32.As explained before, the Raspberry Pi serves as a gateway between the ESP32s and the Cloud, and the ESP32s, placed near each artwork, receive the beacons from the users. Further information about the boards' characteristics are available in the Architecture section, while deeper information about the boards' role are available in our documentation.
• The use of a Cloud service: Microsoft Azure.Microsoft Azure is a crowded place: here we can find the IoT Hub, the Event Hub, our application code, the backend, with its database and the (not implemented) machine learning algorithm.In a real deployment, the database contains information about the pieces of art and when the application needs to present the user a tour or details about an artwork, it will make a query to it. In our demo, the information are stored in the DbContext in the backend, which takes care also to provide them to the application.
• The use of MQTT: a famous lightweight messaging protocol in IoT for small sensors and mobile devices. It use is planned between the boards and the gateway, and also for communicating with the Led RGB.
• We use Bluetooth Low Energy (BLE) technology.We need to use BLE to understand the pieces of art the user is nearest to. So, what we really need to implement is proximity identification more than a complete indoor localization. To achieve this goal we could also use other technologies, therefore we decided to compare the different solutions before making a definite choice. In particular, we considered the most used technologies for these types of problems, which are BLE (we found documentation about the particular iBeacon protocol, but the following considerations can be generalized), NFC, RFID, and GPS. The cooperation table is available in our documentation.The BLE technology met all our requirements: it's not expensive, compatible with mobile phones, has a wide enough range, can be used to do proximity identification. The only issue is the accuracy, but as explained we used techniques to mitigate this disadvantage (RSSI and Kalman Filter).
• The use of Devices sending beacons, that in our case are smartphones.The user's smartphone enables beacon advertising, the artwork boards catch them periodically. Further information about beacon's format and user tracking is available in our documentation.
• A led Actuator, that shows through a green light the ten artworks most appreciated in the current day.The RGB led is turned on for green color by the gateway during data processing, using MQTT messages on topics /Musa/ARTWORK_ID/led and payload G_ON/G_OFF to turn on and off the diode as needed. The red color is reserved for issues, startup phase, and status, so it is managed directly by the board.

Evaluation and performances

We evaluated with different methods the user experience and the technologies used. In particular, the quality of the hardware and software components will be measured taking into account their peculiarities, but also the quality of the whole system.However, keep in mind that this is a project developed for an IoT course, so we will cover this section to provide a comprehensive analysis, but it will not be at the enterprise level.

• The user experience
  Since that users are the core of MuSa, one of the main aspects to evaluate is the user experience. We thought about two evaluation methods: a Moment method and an Episode method.For what concerns the Episode method, that is a method that gives a feedback about the whole experience, at the end of the visit the user will be asked to fill a short survey about his experience. This questionnaire can be implemented using also the AttrakDiff.For the Moment method, that is a method that gives feedback about single moments, through the interaction with MuSa we can understand the visitor's mood (for example, if he is not following the proposed tour we can assume he probably is not enjoying it). Furthermore, for this purpose, we suggest to use also a commercial tool like PrEmo that allows MuSa to know the user's feeling in real-time, but we did not implement it for our demo because is a commercial tool.We presented to a group of people our demo and asked them to imagine the full product deployed in a museum. Considering this, we asked them to fill the survey the MuSa presents the user at the end of their customized tour. Here are the answers collected:

• Overall evaluation of the system
  Software quality is defined by a set of regulations and guidelines by ISO/IEC 9126-1. We used a criteria-based evaluation which gives a measurement of quality in several areas, including understandability, documentation, and portability (the original document that we used as a model is this one). Of course, we did not use the criteria we did not need, so we produced this lighter customized version: MuSa Criteria.You can find the result of our analysis in this document.
• Power consumption
  Doing some measurements through an amperemeter, we discovered that each ESP32 board needs around 110mA at 5V to work, so it consumes 550 mW, which is a good value keeping in mind that we are using both Wi-Fi and BLE.

• Complexity and responsiveness
  We need our system to be responsive to follow the user in real-time during his visit, but at the same time we also don't want to flood our network with messages, keeping the complexity low, also to save the board's power. We need to find a trade-off between these two needs. Therefore, we did a brief analysis of the system and decided to use a Raspberry Pi board as a gateway (more details in our documentation). This also affects the cost: we want to push the limits of our free plan to exploit it to the fullest (the limit of messages we can send to Azure is 8000 per day) so that we can pay as little as possible.This can be achieved by taking advantage of edge computing: we do data pre-processing in the gateway to aggregate messages of different boards and send to Azure a single report. The Raspberry also acts as a sort of bridge, taking MQTT-SN messages and transforming them into MQTT ones. Let's do a short evaluation in terms of complexity and responsiveness of this system, let's consider the following scenario:- we have 30 ESP32 boards that send 1 message per second- raw data is sent to Azure: this means that there are around 30 messages * 60 s = 1800 messages each minute, which means that in 8000 / 1800 = 4, 4 minutes the free plan would expire.- Edge computing: instead of sending the messages directly to Azure, they are collected by the gateway, which instead sends to the cloud a report every 5 seconds. So we have 60 / 5 = 12 messages per minute, which means that the free plan would last for 8000 / 12 = 666, 66 minutes, which is 11, 11 hours.So we have 5 minutes vs 11 hours a day, the advantages of the edge computing architecture are immense! We could have a museum with 30 boards use our system for free each day (unless it stays open for more than 11 hours).
• Maximum number of users
  We made some calculations and a stress test about the maximum number of simultaneous users with our current hardware and technologies.As we saw our boards use many different technologies, like WiFi, BLE, MQTT with JSON body. Those technologies require libraries, and libraries require space; our ESP32 has 4MB of space, we have been able to ad hoc partition it and get enough space for all that we needed, both for code and dynamic partition. But how much free space do we have? Not so much. In this free space, we have to fit the data structures for building the JSON, put temporary data read from BLE, and so on. So some of the data structure can be statically predefined, while others live in the heap. We empirically found a good compromise for predefined sizes, around 5500 bytes for collected data, some more byte as network max message (due to headers), and so on. With this value, in case of single iteration, due to a maximum size for Beacon identifier/RSSI value of 58 bytes, we have space for around 94 iteration. During our stress tests with beacon simulator applications, we got more or less this result: catching 93 beacons out of 100. Let's comment on it. If we have a big room with 20 artworks and 20 boards, if every board catches at most 93 beacons and every board catches beacons of who is very far from the board, probably we will not be able to serve all the visitors, since 93 looks not to be such a big number. What is our solution? Transmit the beacon with lower power. It's easy to set the transmission power of the beacon (refer to our Android application to have an example). If we transmit the beacons with lower power, only the boards close to the visitor will catch his beacons, so we will be able to manage 93 visitors in a very small place around fewer artworks, but probably a very bigger number in the whole museum.
• Backend
  Our backend code quality has been tested with CodeCity. It is a very simple tool that allows us to check the most common software metrics in a new way in which software systems are visualized as interactive, navigable 3D cities. The classes are represented as buildings in the city, while the packages are depicted as the districts in which the buildings reside. The more the city is well structured, the higher the code quality is.We provide deeper details about CodeCity in our documentation. These are our results: there are no "pathological" patters: no building is too high, and they're quite well distributed among the landscape. The code is modular and "healthy".

• Frontend
  The Android application works well, and it doesn't exhibit delays or abnormal behavior. Unfortunately, there is currently no free tool to generate the MSE statistics from an Android Project. We checked the most common metrics for software evaluation. These are the results:

• Price
  - Hardware
  We need as many boards as the number of clusters of artworks and single important pieces displayed in the museum plus one for the gateway; each one costs around 10 euros. We need also the same number of RGB LEDs, that are very cheap, costing around 0, 50 euros each. Of course, there are a lot of other solutions that could substitute the LED.
  The most expensive component would be the two Raspberry Pi boards, which can cost from around 20 € to much more.
  
  - Cloud
  Azure is a commercial product offering a minority of free services for small projects. Using the Microsoft Pricing Calculator we checked for the cheapest possible solution for study purposes.
  The Azure IoT hub is free until 8000 messages/day. Exploiting our architecture choice to use a main gateway board as a bridge for the whole sensor network we can save a big amount of messages.
  The Azure Database is free for the previous Azure Database generation (the 4th), with few GB of space, but the fact you can not have any backup possibility is a major drawback.
  The Azure App Service Plan offers for free a complete solution to deploy a full-stack application with both FE and BE. The free solution offers 1GB of storage, 1GB of RAM, and a shared CPU. Please note that in such a way you can not keep your application always running
  .Unfortunately, the Azure Machine Learning does not offer any free plan, and it starts at about 4 USD/month, but again, for our final delivery, it is not implemented. We attach a short excel report to better visualize all the plans.It is very easy to scale this plan and evaluate how this affects costs using the Azure Pricing Calculator.

How Musa works - user's point of view

Let's have now a complete explanation of how the whole system works from the user's point of view!

• First of all, when a visitor comes to the museum, he has to decide if he wants to have an experience with MuSa or he wants only to help the data collecting: in both cases, he needs to download the MuSa Android App. Obviously, for this explanation, we won't consider the visitors that aren't interested.
• After that a user downloads the App, he fills the profiling survey and decides officially the use of MuSa he wants to do.

• If a user wants only to help the data collecting, his work with the app is done until his visit reaches the end. The app itself will collect data about the visitor's path. At the end of the tour, the visitor will press the End button.

• If a user wants to have a customized experience with MuSa, after he completes the starting survey, he will receive a suggested tour. Then, along his tour, he will receive information about the artworks like the author, the year of creation, and some other details.

• At the end of the tour, the user will be asked to fill the feedback survey, helping to evaluate and to improve MuSa

How Musa works - technical point of view

Let's have now a complete explanation of how the whole system works from a technical point of view!

• When a new user arrives at the museum and downloads the app, his smartphone starts sending beacons
• When the user is next to a piece of art, the ESP32 board receives the beacon
• All the ESP32 boards send their report to the Raspberry Pi gateway, in a variable time, until to a maximum of five seconds, that depends on how many users there are in the museum
• Every five seconds, the Raspberry Pi gateway sends the message to Azure IoT Hub
• Through the Event Hub, the message reaches the backend, that stands in the cloud too
• The backend provides the responses to the received requests and sends them to the app, going backward in the IoT pyramid.

NOTICE THAT
If you want to try MuSa by yourself, here you can find the full release and install notes!

Future plans

The main idea that we did not implement is the Machine Learning algorithm that should have to analyze the users’ behavior for improving the customized tours. The reason is that machine learning is not one of the principal topics of this course and it would have asked some specific knowledge bases.The second idea that we did not implement is a crowd-sensing logic in the Android App, through which we could have to improve the BLE positioning: even if it was very good for our purposes, we chose to follow the RSSI and Kalman Filter way.
Other minor improvements that we thought about are the use of Azure database for storing the information about the artworks and the tours in a robust way and the Moment method for evaluating the user experience during his use of the app, for which is suggested a commercial tool.

Conclusions

This is our project from the birth to the final delivery. If anything is not clear, please check the documentation when it is linked. If you are interested in MuSa, or in anything that could collaborate with MuSa, do not hesitate to contact us here or on our LinkedIn profiles. In the next section, you can find some useful links. Thanks for the attention!

Useful links

• Video Demonstration
• GitHub repository
• Documentation
• LinkedIn profiles: Silvia del Piano, Stefano Foti, Gianmarco Zizzo
• Slides and PDF presentation
• Release and install notes