Build2gether 2.0 – VISUAL IMPAIRMENTS Track 2 Entry Project

Introduction

This document describes a project to submit for the Build2Gether 2.0 challenge, focusing on assisting individuals with visual impairments in indoor activities. The project aims to create a device that helps visually impaired swimmers navigate a pool safely by detecting walls and lane lines. The device will have two modes: Learn Mode, where it maps the pool environment, and Swim Mode, where it guides the swimmer using audio feedback based on the learned map of the pool.

The document outlines the potential use of a SUPERBOX of hardware supplied by the challenge and builds upon a previous B2G1.0: project that addressed similar challenges for visually impaired swimmers B2G 1-- Swimming Theme - Making Waves with Water IoT - Hackster.io. I will refer to this project as B2G1.0 from the remainder of this document.

The main objectives of the project are:

• To create a device that helps visually impaired swimmers navigate a pool safely.
• To develop a device that can detect walls and lane lines in a pool.
• To incorporate two modes of operation: Learn Mode for mapping the pool environment and Swim Mode for guiding the swimmer using audio feedback.

The SUPERBOX Contents

The Superbox contains plenty of hardware to aid in the implementation of my idea.

The Superbox contains:

However, I need to figure out what hardware to choose. I only need certain hardware components. For this project.

I’ll continue to use the Blues hardware that I used in the B2G1.0 Project. At the conclusion of that project I had attained a working knowledge of the Blues hardware/software and had plans of implementing Impact-Detection firmware using the guidance of a Blues Accelerator example roadway-impact-detection on the Blues hardware.

Unfortunately I was unable to finish the implementation for that project. My plan is to pick where I left off and implement the firmware and Enhance the idea based on Feedback from the Contest Masters.

PRIVATE FEEDBACK FROM CONTEST MASTERS

For this contest challenge the organizers used Contest Masters to provide tailored feedback to my idea proposal survey that I filled out at the beginning of the contest. I received the feedback via an email. FEEDBACK SUMMARY report from the Contest Masters.

My idea proposed was the same Idea for my B2G1.0 project. Basically the feedback suggested re-evaluating the location technology strategy and considering practicality and user benefit.

Reassessing the location technology strategy

In Reassessing the location technology strategy feedback, I asked myself the following questions.

What sensors could be used to detect walls and lane lines in the pool, besides using GPS?

The possibility of using cameras, lidar, or ultrasonic sensors to detect walls and lane lines in the pool.

What are the advantages and disadvantages of using each type of sensor for this application?

Here's a breakdown of the potential advantages and disadvantages of using cameras, lidar, and ultrasonic sensors for detecting walls and lane lines in a pool that I came up with.

Cameras:

• Advantages:
• High resolution: Can potentially provide detailed information about the pool environment, including the shape of walls and lane lines.
• Versatility: Can be used to detect a variety of visual features, not just walls and lane lines.
• Disadvantages
• Complexity: Requires image processing algorithms to extract relevant information from the camera feed, which can be computationally intensive.
• Lighting conditions: Performance can be affected by lighting conditions in the pool, such as glare or low light.
• Privacy concerns: May raise privacy concerns if swimmers are identifiable in the camera images.

Lidar:

• Advantages:
• Accurate distance measurement: Provides precise distance measurements to objects, making it suitable for detecting walls and lane lines.
• Unaffected by lighting: Performance is not affected by lighting conditions, making it reliable in various environments.
• Disadvantages
• Cost: Lidar sensors can be more expensive than other options.
• Limited field of view: May require multiple sensors or a scanning mechanism to cover the entire pool area
• Complexity: Requires sophisticated algorithms to process lidar data and interpret the environment.

Ultrasonic sensor

• Advantages:
• Simplicity: Relatively simple to use and integrate into a system.
• Cost-effective: Ultrasonic sensors are generally more affordable than lid
• Disadvantages:
• Limited range: May have a limited detection range, which could be problematic in larger pools.
• Sensitivity to environmental factors: Performance can be affected by factors like water turbulence and temperature variations.
• Lower resolution: Provides less detailed information about the environment compared to cameras or lidar

How could the device use the camera for machine learning to learn the pool environment during a "dry run" in Learn Mode

• Dry Run: A person would manually guide the device along the intended swimming path, likely using the boogie board setup described in the proposal.
• Data Collection: During the dry run, the device's sensors (which could include cameras, lidar, or ultrasonic sensors) would collect data about the pool environment. This data might include distances to walls and lane lines, as well as changes in the sensor readings as the device moves.
• Machine Learning: The collected data would be used to train a machine learning model. The model would learn to identify patterns in the sensor data that correspond to the presence of walls and lane lines. It would also learn to associate these patterns with specific locations in the pool.
• Once the model is trained, it could be used in Swim Mode to guide the swimmer. The device's sensors would continuously collect data, and the model would use this data to estimate the swimmer's position relative to the walls and lane lines. The device could then provide audio feedback to help the swimmer stay on course.

What about using 3-axis gyroscopes and accelerometers?

3-axis gyroscopes and accelerometers together can be advantageous for tracking a swimmer's movement and orientation in a pool. Here's a summary of the benefits:

• Sensor Fusion: Combining data from both sensor types allows for the use of algorithms that leverage the strengths of each, overcoming individual limitations. For instance, accelerometers measure linear acceleration while gyroscopes measure angular velocity. Fusing this data provides a more accurate estimate of overall motion.
• Complementary Filtering: This technique merges the high-frequency response of accelerometers with the low-frequency response of gyroscopes. This helps filter out noise and drift, leading to more stable and reliable orientation measurements.
• Kalman Filtering: This advanced algorithm utilizes a model of system dynamics to estimate the swimmer's state (position, velocity, orientation) based on sensor data. Kalman filters are particularly effective in noisy environments like pools, where factors like water turbulence can impact sensor readings.

Some potential drawbacks of relying solely on these sensors:

• Determining Distance: While gyroscopes and accelerometers excel at tracking direction and orientation, they may not be as reliable for determining the precise distance traveled. Additional sensors or techniques might be needed for accurate distance measurement.
• Gyroscope Drift: Gyroscope readings can accumulate errors over time, affecting orientation accuracy. This drift can worsen due to factors like temperature fluctuations and vibrations common in pool setting
• Limited Linear Acceleration Measurement: Gyroscopes, while adept at measuring rotational motion, may not accurately measure linear acceleration, which is crucial for determining changes in speed and direction.

Therefore, while the combination of 3-axis gyroscopes and accelerometers offers advantages, they might need to be supplemented with other sensors or methods to ensure a comprehensive and reliable navigation solution for visually impaired swimmers.

While the feedback from the Contest Masters raised concerns about the reliability of using 3-axis solid-state gyroscopes for determining distance traveled, they can still be utilized effectively for determining the direction of travel. Here's how:

• Measuring Angular Velocity: Gyroscopes excel at measuring angular velocity, which is the rate of change in orientation. By integrating this data over time, you can track changes in the swimmer's heading (i.e., the direction they are facing).
• Complementing Other Sensors: Gyroscope data can be combined with data from other sensors, such as accelerometers or magnetometers, to improve the overall accuracy and reliability of the system. This sensor fusion approach can help compensate for the limitations of individual sensors and provide a more robust estimate of the swimmer's movement.
• Detecting Turns: Gyroscopes are particularly useful for detecting turns, as they can measure the rate of rotation around each axis. This information can be used to identify when a swimmer is initiating a turn at the end of a lap or changing direction during their swi

However, it's important to note that gyroscopes alone may not be sufficient for precise navigation in a pool environment. Their accuracy can drift over time, and they may not be able to reliably determine the distance traveled. Therefore, they are best used in conjunction with oter sensors and technologies to provide a more comprehensive and reliable navigation solution.

How can accelerometers be used with 3-axis solid-state gyroscopes?

Accelerometers can be used in conjunction with 3-axis solid-state gyroscopes to create a more robust and accurate system for tracking a swimmer's movement and orientation in the pool. Here's how they can be combined:

• Sensor Fusion: By combining data from both accelerometers and gyroscopes, you can employ sensor fusion algorithms. These algorithms use the strengths of each sensor to overcome their individual limitations. For example, accelerometers can provide information about linear acceleration, while gyroscopes measure angular velocity. By fusing this data, you can get a more accurate estimate of the swimmer's overall motion.
• Complementary Filtering: This technique combines the high-frequency response of the accelerometer with the low-frequency response of the gyroscope. This helps to filter out noise and drift, resulting in a more stable and reliable measurement of the swimmer's orientation.
• Kalman Filtering: This is a more sophisticated algorithm that uses a model of the system's dynamics to estimate the swimmer's state (position, velocity, orientation) based on sensor measurements. Kalman filters can be particularly effective in noisy environments, such as a pool where water turbulence and other factors can affect sensor readings.

Here's a breakdown of how accelerometers and gyroscopes complement each other:

By combining these two sensors and using appropriate sensor fusion algorithms, you can create a system that is more accurate, reliable, and robust than either sensor used alone. This can be particularly beneficial for tracking the complex movements of a swimmer in a pool environment.

What are the disadvantages of using low energy Bluetooth or 3-axis solid-state gyros for direction of travel ?

The disadvantages are that it may be challenging to determine reliable distance traveled using these technologies. Additionally, low energy Bluetooth's reception in wet environments is uncertain.

Reassessing the technology strategy

For reassessing the overall technology strategy, I addressed the possible problematic events, practicality and benefits of the proposed project by asking myself the following questions.

What specific use cases could be explored to better understand the needs of visually impaired swimmers in a pool environment?

Here are some specific use cases that could be explored to better understand the needs of visually impaired swimmers in a pool environment:

Navigating to the Wall: Explore how swimmers orient themselves to reach the pool wall safely and efficiently. This could involve understanding their preferred techniques, such as counting strokes or using audible cues from lane lines.

Avoiding Collisions with Other Swimmers: Examine how swimmers avoid collisions with other swimmers in the pool, especially in crowded lanes. This could involve understanding their strategies for detecting the presence of other swimmers and adjusting their course.

Identifying Lane Lines: Investigate how swimmers identify and stay within their designated lane. This could involve understanding their use of tactile cues on lane lines or their reliance on audible feedback.

Turning at the End of a Lap: Analyze how swimmers execute turns at the end of a lap without colliding with the wall. This could involve understanding their techniques for judging distance, adjusting speed, and initiating the turn.

Exiting the Pool Safely: Observe how swimmers navigate from the pool to the exit area. This could involve understanding their use of landmarks, guidance from sighted assistants, or other navigational aids.

By exploring these use cases, it becomes possible to identify the specific challenges faced by visually impaired swimmers and tailor the solution to address their unique needs. This user-centered approach can lead to a more practical, effective, and beneficial device.

How would the device handle changes in the pool environment, such as lane lines being moved or objects placed in the pool?

To handle changes in the pool environment, the device could incorporate the following strategies:

• Periodic Updates: The device could periodically re-enter Learn Mode to update its map of the pool environment. This could be done manually by the user or automatically based on a preset schedule.
• Dynamic Obstacle Detection: The device could use its sensors to continuously scan for new obstacles in the pool, even while in Swim Mode. If a new obstacle is detected, the device could alert the swimmer and update its internal map accordingly.
• User Feedback: The device could allow the user to manually mark changes in the pool environment, such as moved lane lines or new obstacles. This could be done through a simple interface, such as tactile buttons or voice commands.

Machine Learning Adaptation: The device's machine learning algorithms could be designed to adapt to changes in the pool environment over time. By continuously analyzing sensor data, the algorithms could learn to recognize new patterns that correspond to changes in the environment

By combining these strategies, the device can effectively handle unexpected changes in the pool environment, ensuring that the swimmer receives accurate guidance and remains safe even when the pool layout is modified.

What design considerations should be made to ensure the device is comfortable and easy to wear for visually impaired swimmers?

To ensure the device is comfortable and easy to wear for visually impaired swimmers, several design considerations should be taken into account:

• Ergonomics and Fit: The device should be designed to fit securely and comfortably on the swimmer's body without hindering their movement or causing discomfort. This could involve using adjustable straps, soft materials, and a streamlined shape to minimize drag and maximize comfort.
• Lightweight and Compact: The device should be as lightweight and compact as possible to avoid adding unnecessary burden or bulk to the swimmer. This could involve using lightweight materials and optimizing the placement of components within the device.
• Intuitive Controls: The controls for the device should be easy to locate and operate, even for individuals with limited dexterity or tactile sensitivity. This could involve using large, tactile buttons, voice commands, or other intuitive interfaces.
• Quick Release Mechanism: The device should have a quick-release mechanism to allow swimmers to easily remove it in case of an emergency or discomfort. This could be a simple buckle, clasp, or magnetic attachment
• Durability and Water Resistance: The device must be durable and water-resistant to withstand the pool environment and regular use. It should be able to tolerate exposure to chlorine and other pool chemicals, and it should be protected against accidental impacts.
• Customizability: The device should offer options for users to customize the fit and attachment according to their individual preferences and needs. This could involve providing different strap lengths, materials, or attachment points.

By carefully considering these design aspects, the device can be made comfortable, user-friendly, and easy to wear for visually impaired swimmers, promoting a positive and enjoyable swimming experience.

These findings were used to construct a Functional and Technical specification to evaluate the practicality and benefit of the proposed project

FEEDBACK SUMMARY

Problem: The primary challenge is ensuring the safety of visually impaired swimmers in unfamiliar indoor pools, specifically preventing collisions with walls and lane lines. The underlying causes include the limitations of GPS in indoor pool environments and the need for a reliable and practical solution that benefits users.

Feedback's Impact on Problem Understanding: The feedback highlighted the inadequacy of GPS for precise indoor positioning and the importance of considering alternative technologies. This prompted a deeper exploration of sensor technologies suitable for pool environments and a greater focus on user-centered design.

Solution: The revised solution involves a device equipped with sensors for proximity detection, an IMU for tracking the swimmer's orientation and movement, and a camera to gather and detect the environment.

The device will operate in two modes:

Learn Mode: A sighted assistant will guide the device along the pool walls and lane lines, creating a vision machine Learning Model of the pool environment.

Swim Mode: The device will use sensors, the Machine Learning Model and a camera to detect obstacles and provide audio feedback to the swimmer, guiding them along the mapped path and alerting them to potential collisions.

Feedback's Impact on Solution Development: The feedback emphasized the need for a practical and user-friendly solution. This led to the adoption of sensors for their simplicity and cost-effectiveness, as well as the incorporation of a two-mode system to address the challenges of pool mapping and real-time guidance. The revised solution prioritizes user safety, comfort, and ease of use while leveraging sensor technologies that are well-suited to the pool environment.

Feedback Question CONCLUSIONS

This is a summary of the steps I’ve taken to utilize the feedback from the Challenge Masters and my own research to answer the questions I had about meeting the feedback suggestions. Here is a summary of the steps taken to utilize the feedback

• Reassessment of the location technology strategy: The feedback received suggested that GPS may not be the best technology for this use case. So, alternative technologies such as cameras, accelerometer and gyroscope, along with collision detection sensors are now considered
• Use a camera module: A camera module will be used in conjunction with machine learning to map the pool environment in Learn Mode.
• Inertial Measurement Unit (IMU): An IMU containing an accelerometer and gyroscope will be used to track the device's orientation and movement in the water.
• Use collision detection of the Blues note carrier: The Blues note carrier will be used for collision detect
• Still use a waterproof enclosure attached to a Boogie board: The hardware will be protected in the water using a waterproof enclosure attached to a Boogie board.
• Two-mode system with a trained model: The device will have two modes: Learn Mode, where a pool staff member will train the machine-learning model, and Swim Mode, where the swimmer will use the trained model.
• Do not use GPS: GPS will not be used due to its limitations in indoor pool environments
• Audio Output: The device will provide audio feedback to the swimmer using a speaker or buzzer.
• Power Supply: A rechargeable battery will be used to power the device
• Determine which microcontroller to use from the SuperBox: The microcontroller from the SuperBox that best fits the project needs will be selected.
• Reassessment of the overall technology strategy: I addressed the possible problematic events, practicality and benefits of the proposed project

Specification

These functional and technical specifications serve as a blueprint for the development and evaluation of the proposed solution, ensuring that it meets the desired criteria for practicality, effectiveness, and user benefit. The specification includes some points from the B2G1.0 project.

Functional Specification

The functional specifications outlines the following:

• Sensor Requirements: The types of sensors to be used (e.g., IMU, camera), their specifications (e.g., range, accuracy, field of view), and how they will be integrated into the device.
• Operating Modes: Detailed descriptions of the Learn Mode and Swim Mode, including the user interactions required in each mode and the expected device behavior.
• Audio Feedback: The types of audio cues to be used (e.g., beeps, voice prompts), their timing and frequency, and how they will convey information to the swimmer effectively.
• User Interface: The design of a user interface for configuring the device, selecting modes, and providing feedback to the user.
• Power Management: Strategies for optimizing power consumption to ensure long battery life during use.

How could the device be designed to ensure the safety of visually impaired swimmers in the pool?

By incorporating these design features, the device can effectively guide visually impaired swimmers, helping them avoid collisions and stay on course, ultimately promoting a safer and more enjoyable swimming experience.

The device should incorporate several design features:

• Reliable Detection: The device must accurately and consistently detect walls and lane lines, even in varying pool conditions (e.g., lighting, water clarity). This could involve using a combination of sensors and machine learning algorithms to improve detection accuracy and reduce false positives/negatives
• Clear and Timely Feedback: The device should provide clear and timely audio feedback to the swimmer, indicating the proximity of walls and lane lines. The feedback should be easy to understand and interpret, allowing the swimmer to adjust their course accordingly
• Comfortable and Secure Attachment: The device should be comfortably and securely attached to the swimmer, either through a handheld boogie board or a wearable device. The attachment mechanism should be adjustable to accommodate different body sizes and swimming styles, and it should not interfere with the swimmer's movement
• Intuitive Operation: The device should be easy to operate, with simple controls for switching between modes and adjusting settings. The user interface should be accessible to visually impaired individuals, potentially using tactile buttons or audio cues
• Long Battery Life: The device should have a long battery life to support extended swim sessions without needing frequent recharging. This could involve using energy-efficient components and optimizing power consumption.
• Durability and Water Resistance: The device must be durable and water-resistant to withstand the pool environment. It should be able to tolerate exposure to chlorine and other pool chemicals, and it should be protected against accidental impacts.

What specific hardware components would be required to implement the device's functionality? Discuss the role of each component in the device's operation.

To implement the device's functionality, several hardware components would be required

• Microcontroller: The microcontroller acts as the brain of the device, processing sensor data, executing algorithms, and controlling the output. It could be a suitable model from the SUPERBOX, with onboard sensors.
• Swan + notecarrier-a with Wi-Fi notecard
• Sensors:
• Inertial Measurement Unit (IMU): An IMU containing an accelerometer and gyroscope could track the device's orientation and movement in the water, helping to determine the swimmer's position and direction..
• Camera: A camera module will be used in conjunction with machine learning to map the pool environment in Learn Mod
• The Blues note carrier will be used for collision detection
• Audio Output: A speaker or buzzer would be needed to provide audio feedback to the swimmer, indicating the proximity of walls and lane lines or signaling when the swimmer is off course.
• Power Supply: A rechargeable battery would be necessary to power the device. The battery capacity should be sufficient for extended swim sessions. The Notecarrier-A has a battery pack
• Waterproof Enclosure: A waterproof enclosure would protect the electronics from water damage, ensuring the device's durability and reliability in the pool environment.

The role of each component in the device's operation would be as follows:

• The microcontroller would collect data from the sensors, process this data using machine learning algorithms, and determine the swimmer's position relative to the pool walls and lane lines.
• The distance sensors would provide information about the distance to obstacles, while the IMU would track the device's movement and orientation.
• The microcontroller would use this information to generate audio feedback, which would be played through the buzzer, guiding the swimmer and helping them avoid collisions.
• The power supply would ensure continuous operation of the device, and
• The waterproof enclosure would protect the electronics from water damage.

What are some of the challenges that visually impaired swimmers face when navigating a pool, and how could the device address these challenges?

Visually impaired swimmers face several challenges when navigating a po

• Collision with walls: The most significant danger is colliding with the pool walls, which can cause injury. The device addresses this by detecting walls and providing audio feedback to alert the swimmer before they reach the wall.
• Staying on course: It can be difficult for visually impaired swimmers to stay on a straight course and within their lane. The device helps with this by tracking the swimmer's position and providing audio cues to guide them along the correct path
• Locating the pool exit: Finding the exit ladder or steps can be challenging. The device could potentially address this by detecting when the swimmer is near the pool edge and providing an audio signal to guide them towards the exit.
• Unfamiliarity with new pools: Visually impaired swimmers may struggle to adapt to new pool environments. The device's Learn Mode allows it to map the pool beforehand, providing the swimmer with a familiar reference point regardless of the pool they are in.

The device aims to address these challenges by providing a combination of obstacle detection, audio guidance, and pool mapping capabilities, ultimately making swimming a safer and more accessible activity for visually impaired individuals.

To ensure the audio feedback is effective in guiding visually impaired swimmers, several factors need to be considered:

• Clarity and Simplicity: The audio cues should be distinct and easy to understand, avoiding complex or confusing sounds.
• Volume and Frequency: The volume should be adjustable to accommodate different levels of hearing ability, and the frequency of the sounds should be easily distinguishable from background noise in the pool environment
• Directionality: The audio feedback should provide directional cues, indicating whether the swimmer needs to adjust their course to the left or right. This could be achieved through stereo sound or varying audio patterns
• Urgency: The audio cues should convey a sense of urgency when the swimmer is approaching an obstacle, prompting them to take immediate action to avoid a collision
• Customizability: The device should allow swimmers to personalize the audio feedback according to their preferences, such as choosing different sounds or adjusting the volume and frequency

By carefully considering these factors, the device's audio feedback can be designed to be effective, informative, and user-friendly, ultimately enhancing the swimming experience for visually impaired individuals.

The device proposes two modes of operation: Learn Mode and Swim Mode. Here's a breakdown of the advantages and disadvantages of each mode:

Learn Mode:

• Advantages:
• Maps the pool environment: This mode allows the device to learn the specific dimensions and layout of the pool, including the location of walls and lane lines.
• Customizable to different pools: The device can be adapted to different pool environments, making it versatile for swimmers who use various facilities.

Disadvantages:

• Requires initial setup time: The user needs to manually guide the device around the pool to map it, which takes some initial time and effort
• May need periodic updates: If the pool layout changes (e.g., lane lines are moved), the map may need to be updated.

Swim Mode:

• Advantages
• Provides real-time guidance: Once the pool is mapped, the device can provide audio feedback to guide the swimmer, helping them avoid obstacles and stay on course..
• Hands-free operation: The swimmer can focus on their swimming technique without needing to manually control the device
• Disadvantages:
• Relies on accurate mapping: The effectiveness of Swim Mode depends on the accuracy of the map created in Learn Mode. Errors in the map could lead to incorrect guidance.
• May not adapt to dynamic changes: If the pool environment changes unexpectedly (e.g., an object is placed in the pool), the device may not be able to detect it and could provide incorrect guidance

Overall, both Learn Mode and Swim Mode offer distinct advantages and disadvantages. Learn Mode is essential for initial setup and customization, while Swim Mode provides real-time guidance during swimming. The device's effectiveness relies on the accurate mapping of the pool environment in Learn Mode and the ability to handle unexpected changes in Swim Mode.

To minimize sensory overload and enhance the effectiveness of the audio feedback for visually impaired swimmers, the device can incorporate the following design considerations:

• Adjustable Volume: The device should allow users to personalize the volume of the audio feedback, ensuring it is neither too loud nor too soft, and can be easily heard over the background noise of the pool.
• Distinct Audio Cues: The audio cues used to signal proximity to walls, lane lines, or other obstacles should be distinct and easily distinguishable from one another. This can be achieved by using different tones, pitches, or rhythms for each type of cue.
• Spatial Audio: Implementing spatial audio techniques can provide directional information to the swimmer, indicating the direction of the obstacle or the required course correction. This can be achieved using stereo sound or binaural audio.
• Frequency Modulation: Varying the frequency or pitch of the audio cues can convey information about the distance to the obstacle, with higher frequencies indicating closer proximity.
• Haptic Feedback: In addition to audio cues, the device could incorporate haptic feedback, such as vibrations, to provide another channel of information to the swimmer. This can be particularly useful in noisy pool environments or for individuals with hearing impairments.
• User Customization: The device should offer options for users to customize the audio feedback according to their preferences and needs. This could include selecting different audio cues, adjusting the volume, frequency, or haptic feedback intensity.

By carefully considering these design aspects, the device can effectively communicate information to visually impaired swimmers without overwhelming them, promoting a safer and more enjoyable swimming experience

Technical Specification

The technical specifications includes:

• Hardware Components: The specific models of microcontrollers, sensors, and other components to be used, along with their technical specifications.
• Camera, collision detection, accelerometers and gyros
• Software Design: The software architecture, algorithms for sensor data processing and decision-making, and implementation details.
• Communication Protocols: The protocols used for communication between the device's components and with external devices (if applicable).
• Performance Metrics: The expected performance of the device in terms of accuracy, response time, and power consumption.
• Testing and Validation: Procedures for testing and validating the device's functionality and performance under different conditions.

Hardware Components:

Blues'(Swan v3, Notecarrier A, Notecard Cellular NBGL, Notecard WiFi v1) from previous B2G 1-- Swimming Theme - Making Waves with Water IoT

Swan v3 MCU

Swan is a low-cost embeddable STM32L4+-based microcontroller board designed to accelerate the development and deployment of battery-powered IoT solutions. It is especially useful for applications requiring large memory or a high degree of I/O expandability at an affordable cost, such as edge-based inferencing and remote monitoring. Acting in this role, Swan is designed to be the ideal companion to Blues's Notecard.

Notecarrier A

Notecarriers are companion development boards designed to make it easy to prototype and deploy IoT solutions with the Notecard.

The Notecarrier A is designed for building battery-powered wireless applications, exposes all of the pins of the Notecard via the provided headers, and has onboard cellular and GPS/GNSS antenna.

Notecarriers require a Notecard to function. The Notecarrier A is compatible with the Notecard Cellular, Notecard Cellular (Legacy), Notecard WiFi, and Notecard LoRa.

Notecard WiFi v1

The Notecard is a device-to-cloud data pump that reduces the complexity of building connected solutions with Wi-Fi.

The Notecard WiFi is a lower-cost version of the cellular-based Notecards for 2.4 GHz Wi-Fi only, and is API-compatible with the cellular-based Notecards. The Notecard WiFi also supports location tracking via Wi-Fi Triangulation.

With just two lines of code you can send data to the cloud in minutes, with no complex device registration or provisioning required.

With a powerful JSON-based API, you can program the Notecard over USB, or control it from your preferred microcontroller or single-board computer using one of our open-source firmware libraries.

Connect from your preferred host to the Notecard using Serial or I2C.

Utilizes the ESP32-S3-WROOM-1-N8R2 for connecting to 2.4 GHz Wi-Fi networks.

JSON- and SoftAP-based provisioning support.

Onboard ultra-low power ESP32 MCU with 8MB flash memory.

High-performance 3-axis accelerometer and temperature sensor.

Seeed Studio Grove - Vision AI Module

a thumb-sized AI camera with pre-installed ML algorithms for face recognition and people detection & supports customized models

Seeed Studio Raspberry Pi Camera Module

Raspberry Pi Camera v2 is the new official camera board released by the Raspberry Pi Foundation.

Grove Buzzer Module

The Grove Buzzer module is a simple electronic component that features a piezoelectric buzzer as its main element.

Use the Swan as the Arduino.

Directly connect the Grove Buzzer to the Arduino as follows:

Connect the red wire to the 5V pin on the Arduino.

Connect the black wire to the GND pin on the Arduino.

Connect the yellow wire to digital pin 6 on the Arduino

MAX32666FTHR BOARD

The MAX32666FTHR board is a rapid development platform to help engineers quickly implement battery optimized Bluetooth® 5 solutions with the MAX32666 Arm® Cortex®-M4 processor with FPU. The board also includes the MAX1555 1-Cell Li+ battery charger for battery management. The board also includes a variety of peripherals, such as a micro SD card connector, 6-axis accelerometer/gyro, RGB indicator LED, and pushbutton. Of special interest for this project is the 6-Axis Accelerometer/Gyro and the MAX1555 battery charger for battery management. Here is the full list of features:

• MAX1555 1-Cell Li+ Battery Charger
• Charge from USB
• On-Chip Thermal Limiting
• Charge Status Indicator
• Expansion Connections
• Breadboard Compatible Headers
• 10-Pin Cortex Debug Header
• Micro USB Connector
• Micro SD Card Connector
• Integrated Peripherals
• RGB Indicator LED
• User Pushbutton
• 6-Axis Accelerometer/Gyro
• Bluetooth Surface Mount Antenna

Software Design:

1. Software Architecture:

• Modular Design: The software will be divided into modules for sensor data acquisition, data processing, decision-making, audio feedback, and user interface.
• Real-time Operating System (RTOS): Consider using an RTOS (e.g., FreeRTOS) to manage tasks and ensure timely response to sensor inputs.
• Machine Learning Framework: Utilize a suitable ML framework (e.g., TensorFlow Lite Micro) for implementing and running the trained machine learning model on the microcontroller.

2. Sensor Data Acquisition:

• Camera:

Capture images at a suitable frame rate (consider power consumption).

Implement image pre-processing (e.g., resizing, color space conversion) to optimize for the ML model.

Interface with the camera module using appropriate libraries (e.g., OpenCV).

• IMU:

Read accelerometer and gyroscope data at a high enough frequency to capture swimmer movements accurately.

Implement sensor fusion algorithms (e.g., Kalman filtering) to combine IMU data for robust orientation and movement tracking.

• Collision Detection:

Utilize the Blues Notecarrier's impact detection capabilities.

Set appropriate thresholds for impact detection based on testing and experimentation.

3. Data Processing and Decision-Making:

• Machine Learning Model:

Deploy the trained ML model on the microcontroller.

Optimize the model for memory and computational constraints of the microcontroller.

Implement a mechanism to periodically update the model with new data (e.g., during Learn Mode).

• Sensor Data Fusion:

Combine data from the camera, IMU, and collision detection sensors to create a comprehensive understanding of the swimmer's position and environment.

Implement algorithms to filter out noise and handle sensor inconsistencies.

• Decision-Making Logic:

Develop algorithms to interpret sensor data and make real-time decisions about audio feedback and guidance.

Consider factors like distance to obstacles, swimmer's orientation, and pool dimensions.

4. Audio Feedback:

• Audio Cues:

Design a set of distinct audio cues for different scenarios (e.g., approaching wall, off course, reached end of lane).

Implement spatial audio techniques to provide directional information.

Allow for user customization of audio cues and volume.

Buzzer Control:

Interface with the buzzer module using appropriate libraries and control signals.

Implement timing and frequency control for audio cues.

5. User Interface:

• Mode Selection:

Implement a mechanism for the user to switch between Learn Mode and Swim Mode (e.g., button press, voice command).

• Feedback and Configuration:

Provide audio or haptic feedback to the user for mode changes and settings adjustments.

Allow for user configuration of parameters like audio volume and cue preferences.

6. Communication Protocols:

• Sensor Communication:

Use appropriate protocols (e.g., I2C, SPI) to communicate with the IMU, camera, and other sensors.

• Wireless Communication:

Utilize the Blues Notecard's wireless capabilities for data transmission and potential remote monitoring features.

7. Performance Metrics:

• Accuracy: Define metrics for evaluating the accuracy of obstacle detection, position tracking, and guidance.
• Response Time: Measure the time taken for the device to respond to sensor inputs and provide audio feedback.
• Power Consumption: Monitor and optimize power usage to ensure long battery life.

8. Testing and Validation:

• Test Cases: Develop a comprehensive set of test cases covering various pool scenarios, swimmer movements, and potential edge cases.
• Validation Methods: Use both simulated and real-world testing to validate the device's performance and reliability.
• User Feedback: Collect feedback from visually impaired swimmers to assess usability and identify areas for improvement

Algorithms for sensor data processing and decision-making

• Impact detection

I’ll continue to use the Blues hardware that I used in the B2G1.0 Project. At the conclusion of that project I had attained a working knowledge of the Blues hardware/software and had plans of implementing Impact-Detection firmware using the guidance of a Blues Accelerator example roadway-impact-detection on the Blues hardware.

This was of interest to me because of the relation to my idea of detecting the walls of a pool. It uses the Notecard attached to a Notecarrier-A to detect a collision and sends the impact to Notehub.

The GitHub page is HERE . The card.motion API is HERE.

• Buzzer

an Arduino example. The Grove Buzzer will emit a tone for one second, followed by a one-second pause, repeating this cycle continuously

void setup() {
pinMode(6, OUTPUT); // Set pin 6 as an output
}
void loop() {
digitalWrite(6, HIGH); // Turn the buzzer on
delay(1000);           // Wait for 1 second
digitalWrite(6, LOW);  // Turn the buzzer off
delay(1000);           // Wait for 1 second
}

BUILD

This section will attempt to describe the Mechanical and Electronic components of the prototype.

Here's a build for the electronics and assembly, aiming for a balance of functionality, ease of assembly, and potential for future enhancements:

Core Components:

Microcontroller: Swan v3 (STM32L4+): Offers a good balance of power, memory, and peripherals.

Wireless Communication: Blues Notecard WiFi: Provides necessary connectivity for data logging and potential future features.

Sensor Board: Notecarrier-A: Conveniently houses the Notecard and provides additional features like the accelerometer/gyro.

Camera: Seeed Studio Grove - Vision AI Module: Compact and offers pre-trained models for object detection, potentially simplifying development.

Audio Feedback: Grove Buzzer Module: Simple and effective for providing audio cues.

Power Supply:

Battery: Utilize the Notecarrier-A's battery holder for a rechargeable LiPo battery. This provides a compact and integrated power solution.

Enclosure:

Waterproof Enclosure: Hammond 1554H2GYCL: Offers necessary protection for the electronics. Consider adding custom cutouts for camera lens, buzzer, and any necessary buttons/switches.

Assembly:

Mount components: Securely attach the Swan v3, Notecarrier-A, and buzzer module within the waterproof enclosure using appropriate mounting hardware or adhesive.

Connect camera: Connect the Vision AI Module to the Swan v3 using the Grove connector.

Wiring: Carefully connect the buzzer module to the Swan v3's GPIO pins, ensuring correct polarity.

Battery: Insert the rechargeable LiPo battery into the Notecarrier-A's battery holder.

Enclosure: Seal the enclosure, ensuring all openings are properly closed and waterproof.

Integration with Boogie Board:

Attachment: Use a strong and adjustable strap or mounting mechanism to securely attach the enclosure to the boogie board. Consider the swimmer's comfort and ease of use.

Future Enhancements:

Additional Sensors: The modular design allows for easy integration of additional sensors in the future, such as a water quality sensor or a more advanced IMU.

Advanced Audio: Consider using a small waterproof speaker for more nuanced audio feedback, including voice prompts or directional cues.

User Interface: Add tactile buttons or a small OLED display to the enclosure for mode selection, configuration, and user feedback.

Wireless Charging: Explore wireless charging options for the LiPo battery to further improve user convenience.

Additional Considerations:

Thorough Testing: Conduct extensive testing in both dry and wet conditions to ensure the reliability and accuracy of the system.

User Feedback: Gather feedback from visually impaired swimmers throughout the development process to ensure the device meets their specific needs and preferences.

Waterproof Flotation Device Described

This section will describe the build of the flotation device that will be contracted for this prototype. It is basically the same design as the B2G1.0 project. However it will house different Electronics.

A Waterproof Enclosure

HAMMOND 1554H2GYCL Plastic Enclosure, Watertight,

The Boogie board + bungee cord + Waterproof Enclosure

Assembled

ELECTRONICS

This section will describe the electronics that will be contained in the waterproof enclosure pictured in the previous section.

Note this section will be expanded as time goes on and I experiment with connecting the components described in the Technical specification.

The Power source + The Hardware components + waterproof enclosure

Assembled and ready to attach to the boogie board

The whole thing is put together and ready for the Water.

IMPLEMENTATION

RUNNING THE SYSTEM

Attach the Waterproof Enclosure to the Boogie Board

Gather the necessary materials: waterproof enclosure, boogie board, and mounting hardware.

Clean and dry the surface of the boogie board where the enclosure will be attached.

Align the enclosure with the mounting holes on the boogie board.

Secure the enclosure to the boogie board using the mounting hardware.

Ensure that the enclosure is securely fastened and watertight.

Put the System in LEARN Mode

Press and hold the power button on the system for 3 seconds to turn it on.

Select the "LEARN" mode from the system's menu.

The system will enter LEARN mode and begin to record your swimming data.

Train the Model

Swim one lap of your pool, paying attention to the system's Off Course Beep.

If you hear the Off Course Beep, adjust your swimming direction to stay in the center of the lane.

Continue swimming until you reach the wall and hear the Wall Beep.

Stop swimming and wait for the system to process your data.

Reverse Direction and Swim Back to the Wall

After hearing the Wall Beep, reverse your swimming direction and swim back to the wall.

Pay attention to the system's Off Course Beep and compensate if necessary.

When you reach the wall, stop swimming and wait for the system to process your data.

Put the System in SWIM Mode

Select the "SWIM" mode from the system's menu.

The system will enter SWIM mode and begin to track your swimming performance.

Start Your First Lap

Start swimming your first lap and listen for the Wall Beep.

If you hear the Off Course Beep, adjust your swimming direction to stay in the center of the lane.

Continue swimming until you reach the wall and hear the Wall Beep.

Reverse Direction and Swim Subsequent Laps

After hearing the Wall Beep, reverse your swimming direction and swim back to the wall.

Repeat steps 6 and 7 for each subsequent lap

CONCLUSIONS

So, the Design Phase took longer than I thought, and the contest deadline is September 4, 2014, which is creeping up fast. I'm not quite finished with the project yet, but I'm working hard to wrap it up. I'm really excited about my project and I'm confident that it will be a valuable resource for others. I plan to document every step of the way, so people can see how I did it and maybe even replicate it themselves. Once I'm finished, I'll be showcasing a fully functional prototype, along with software and instructions. This will give people a chance to see what the project can do and how it works. I'm also going to be seeking feedback from users and experts to help me refine and improve the prototype. I want to make sure it's the best it can be and meets the highest standards of quality and functionality. I'm really excited to share this project with the world and I hope it will make a positive contribution to the field.

ENHANCEMENTS

This section contains the results of my research on how the system could be enhanced to make it better.

What additional safety features could be incorporated into the device to enhance the protection of visually impaired swimmers in the pool?

In addition to the safety features already discussed, here are some other ideas that could be incorporated to further enhance the protection of visually impaired swimmers:

• Emergency Shutoff: A feature that allows the swimmer to quickly and easily deactivate the device in case of an emergency or if they feel uncomfortable. This could be a physical button or a voice command.
• Remote Monitoring: The device could transmit data to a remote device or application, allowing a coach, lifeguard, or caregiver to monitor the swimmer's progress and location in real-time. This could provide an additional layer of safety and support.
• Collision Avoidance: More advanced algorithms could be developed to not only detect obstacles but also predict potential collisions and guide the swimmer to proactively avoid them.
• Pool Exit Detection: The device could detect when the swimmer is approaching the edge of the pool and provide an alert to help them safely exit the water.
• Water Chemistry Monitoring: Sensors could be included to monitor the pool's water chemistry, alerting the swimmer to potentially unsafe conditions such as high chlorine levels or imbalanced pH.
• Social Features: The device could incorporate features that allow swimmers to connect with each other, providing a sense of community and support. This could include audio communication channels or proximity-based alerts.

These additional safety features could significantly improve the device's ability to protect visually impaired swimmers and provide them with a more confident and enjoyable swimming experience.

What additional features or enhancements could be considered to make the device even more inclusive and accessible for visually impaired swimmers with diverse needs?

To make the device even more inclusive and accessible for visually impaired swimmers with diverse needs, the following additional features or enhancements could be considered:

• Multiple Languages and Accents: The audio feedback could be offered in multiple languages and with diverse accents to cater to a wider range of users.
• Customizable Audio Profiles: The device could allow users to create personalized audio profiles, adjusting the volume, pitch, speed, and types of audio cues to suit their individual preferences and hearing abilities.
• Text-to-Speech Conversion: The device could incorporate text-to-speech technology, enabling it to convert written instructions or pool information into spoken feedback. This could be useful for providing additional context or guidance to swimmers.
• Braille Interface: A Braille interface could be added to the device, allowing users who are proficient in Braille to receive tactile feedback and control the device's settings.
• Integration with Assistive Technologies: The device could be designed to integrate with other assistive technologies, such as screen readers or smart glasses, to provide a more seamless and comprehensive user experience.
• Open-Source Platform: Making the device's software and hardware design open-source could encourage community-driven development and customization, leading to more innovative and accessible solutions.
• User Testing and Feedback: Involving visually impaired swimmers in the design and testing process is crucial to ensure the device meets their specific needs and preferences. Their feedback can help identify areas for improvement and guide the development of future iterations.

By incorporating these additional features and enhancements, the device can be made more inclusive and accessible, catering to the diverse needs of visually impaired swimmers and empowering them to enjoy swimming with greater confidence and independence.

How could the accuracy and reliability of the device's obstacle detection be enhanced?

To enhance the accuracy and reliability of the device's obstacle detection, consider these strategies:

• Sensor Fusion: Combine data from multiple sensors (e.g., ultrasonic and IMU) to improve accuracy and reduce false detections. This can help differentiate between actual obstacles and environmental factors like water turbulence.
• Machine Learning Refinement: Continuously train and refine the machine learning model with real-world data collected during Learn Mode and Swim Mode. This will help the model adapt to different pool environments and improve its ability to generalize.
• Environmental Adaptation: Develop algorithms that can account for environmental factors like water temperature, lighting conditions, and pool acoustics. This can help maintain detection accuracy even in varying conditions.
• Redundancy: Implement redundant sensors or algorithms to cross-check detection results and reduce the risk of false positives or negatives.
• User Feedback: Incorporate a mechanism for users to provide feedback on detection accuracy. This can help identify edge cases or scenarios where the device may not perform optimally.

By implementing these strategies, the device can achieve more robust and reliable obstacle detection, ensuring the safety of visually impaired swimmers in diverse pool environments.

What specific features or enhancements could be considered to make the device even more inclusive and accessible for visually impaired swimmers with diverse needs?

To make the device even more inclusive and accessible for visually impaired swimmers with diverse needs, the following additional features or enhancements could be considered:

• Multiple Languages and Accents: The audio feedback could be offered in multiple languages and with diverse accents to cater to a wider range of users.
• Customizable Audio Profiles: The device could allow users to create personalized audio profiles, adjusting the volume, pitch, speed, and types of audio cues to suit their individual preferences and hearing abilities.
• Text-to-Speech Conversion: The device could incorporate text-to-speech technology, enabling it to convert written instructions or pool information into spoken feedback. This could be useful for providing additional context or guidance to swimmers.
• Braille Interface: A Braille interface could be added to the device, allowing users who are proficient in Braille to receive tactile feedback and control the device's settings.
• Integration with Assistive Technologies: The device could be designed to integrate with other assistive technologies, such as screen readers or smart glasses, to provide a more seamless and comprehensive user experience.
• Open-Source Platform: Making the device's software and hardware design open-source could encourage community-driven development and customization, leading to more innovative and accessible solutions.
• User Testing and Feedback: Involving visually impaired swimmers in the design and testing process is crucial to ensure the device meets their specific needs and preferences. Their feedback can help identify areas for improvement and guide the development of future iterations.

By incorporating these additional features and enhancements, the device can be made more inclusive and accessible, catering to the diverse needs of visually impaired swimmers and empowering them to enjoy swimming with greater confidence and independence.

Thank You

I genuinely appreciate you taking the time to explore my project. It's my hope that you now better understand how technology can transform the pool into a safer and more independent space for them. Your valuable feedback and genuine interest in this project mean the world to me. They serve as a source of motivation, encouraging me to strive for continuous improvement and refinement of this solution.