It is extremely challenging for visually impaired people to develop software using development tools that are not meant for them. The new AI code assistants are only available today in the mainstream development environments. We wanted to explore the feasibility of developing a code editor that did not rely on external screen readers and provide easy integration with generative AI to create code. Following feedback from the contest masters validated our approach. "Software development is particularly tricky for the blind, as there seems to be less focus on making solutions that are more user-friendly. Those with visual impairment can use screenreaders to understand what’s going on the screen, as well as use keyboard shortcuts to operate the computers. Those with some functional vision can utilize vision enhancement tools to make the visuals on the screen easier to see.However, the current landscape has shown that screenreader integration with IDEs may not always provide a seamless experience. Vision enhancement tools can help those with limited vision, but there still exists a time penalty. While reading lines of code with a screen reader, some crucial information may not be evident, such as upper/lower case characters or symbols. Screenreader users can choose to read the lines character by character, but this is slow.A Braille display can mitigate some of these issues, but they are expensive.When running code, the results may not always be accessible to the user, especially if graphical emulators are involved."
Python was selected as the language for developing the code editor and to call the API's to ChatGPT, Speech-to-Text and Text-to-Speech. Although this product was developed and tested on Windows, the code will run on multiple platforms. The concepts explored here are applicable to code development by visually impaired programmers as well as to software development by programmers without impairment. Therefore, the market for this product can be very large.
The editor is controlled through set of keyboard command shortcuts. The prompt to ChatGPT such as "I need code to send data from ESP32S3 to Notecard" can be either spoken directly or typed into a dialog box. The code generated by ChatGPT can be optionally read back as it comes in. There are short-cuts for documenting code functions through ChatGPT requests. We have uploaded a video to show the editor in action.
Coding can generate a lot of stress, and it is important to monitor the health in real-time. The IoT-based monitoring solution was implemented using hardware from three sponsors, Blues.io hardware for NB-IoT telemetry, Espressif ESP32S3 Sense as 1) the MCU for driving the Blues.io notecard, 2) to act as the BLE Central/Client and 3) collect environmental data using the BME680 sensor, and the Nordic NRF52840DK as BLE Peripheral/Server to transmit heart rate (simulated) measurements via Bluetooth.
This project addresses a critical need for the visually impaired programmers and perhaps will be shaped further by them as they use it to write code to improve the code editor itself.
Comments