3D Soundscape with Infineon PSoC technology

Spacial Audio and Ambisonics - Context

Spacial Audio has a history of 140 years. Nowadays spacial audio sound technology enables immersive sound experiences for areas such as gaming, or portable sound devices. Due to the growing urbanization, experts see the need to design and plan our environment from both, the visual and the listening perspective. Therefore the application-areas for spacial audio steadily increase.

With spacial audio technology the listener is placed virtually in a room while the sound sources/ loudspeakers are located (virtually) in multiple dimensions around him (i.e. 3D-Sound. or higher dimensional sound domes). To achieve the impression of a multi-dimensional soundscape, the input sound is manipulated algorithmically and distributed to the loudspeakers in a way that it gives the impression to the listener that he is part of a spacial sound scene. Technically it is either possible to consume positional sound experiences with a speaker array. However, it is also possible render multidimensional sound experiences in a way that they can be consumed with headphones, while the impression of the dimensionality is preserved.

Ambisonics is a formal and extensible hierarchical system that is used to represent sound fields. Ambisonics consists of the stages „encoding“, „transmission“ and „decoding“, and it is suitable to create sound fields of very high dimensional order. For more information about Ambisonics and positional sound, see the following document published by Stanford Universities’ Artificial Intelligence Center SRI International https://ccrma.stanford.edu/courses/222/lectures/14/Ambi-Decoders-Music222-2021.pdf.

In order to experience soundscapes dynamically, the listener wants to move freely within a given virtual audio space, while the resulting sound shall constantly adapt to his listening location. If a listener for example moves his head, the positional characteristics of the sound that hits his ears should adapt to this movement.

Problem to Solve

To achieve positional audio experiences , the listener usually connects a kind of positional sensor to his sound processor. This sensor provide the positional information. He then uses a sound renderer that is able to dynamically adapt the audio to the measured position. This adapted sound is send to the speakers or the headphones.

Infineon‘s PSOC 4100S is able to track positions and distances and is therefore a suitable 3D-positional sensor for such a scenario.

Audio Software Libraries - Installation

Several software libraries exist that support the real-time manipulation of spacial audio. In the above-mentioned way. To mention just a few, the „OpenAL-framework“ is an open source framework, which supports the creation and reproduction of three dimensional positional sound (see https://de.wikipedia.org/wiki/OpenAL). The "ICST Ambisonics Plugins" (https://ambisonics.ch/page/icst-ambisonics-plugins) enable ambisonics in the Reaper DAW.

Extensive tests show that the "Sound Scape Renderer (SSR)" is the most suitable tool for the idea current project. The SSR is a highly integrated tool for the real-time reproduction of spacial audio, and it is open source.

• Step 1: In order to follow-on with the current project, install the SSR on either your Mac OS-Computer or onto your Linux-Machine. For more information about the tool, and its installation on Mac or Debian, see the following link http://spatialaudio.net/ssr/.

Integrating PSOC 4100S and the Soundscape Renderer - Design

The SSR offers two main types of user interfaces. The interaction with the SSR is either possible with an own Graphical User Interface, and a mouse. Or it is possible to connect external positional sensors to the renderer by means of a network interface, which is based on TCP/IP sockets. The latter solutions follow the conventions of the Virtual-Reality Peripheral Network (VRPN) https://github.com/vrpn/vrpn/wiki, while the first offers a common graphical UI, as it is known from other software packages as well. Refer to the homepage at http://spatialaudio.net/ssr/ to find demonstrations of the available user interfaces.

Several commercial standard positional sensors exist that interface with SSR using the TPC/IP sockets. In addition, a low cost DIY head tracking device is available, named Razor AHRS (https://github.com/Razor-AHRS/razor-9dof-ahrs/wiki/Tutorial#writing-your-own-code-to-read-from-the-tracker).

Technically Infineon‘s PSOC 4100S can either be integrated with the SSR via the USB protocol and the Graphical User Interface, or it is possible to adapt the Razor AHRS libraries in a way that it can be connected via the TPC/IP sockets.

As the Infineon PSOC 4100S is able to mimic a conventional USB pointing device (mouse) but offers much more capabilities and programability than a usual mouse, the most simple solution is to interface it as a USB device, and to extend the behavior of that mouse, in a way that it is possible to use gestures or proximity.

Infineon PSOC 4100S - Basic Integration

Infineon’s PSOC 4100S Kit comes with several test programs, which allow you to test the different application areas for this device. The program „CE16892 USB-HID Trackpad“ allows you to configure the Kit to act like an USB Mouse. The source code for this program offers several parameters, which you can use to change the behavior of this device. Therefore you can make your pointing device much more capable that a usual mouse.

• Step 2: On the product homepage for the kit will you find the needed tools and documents in order to get you started (see https://www.infineon.com/cms/en/product/evaluation-boards/cy8ckit-041-41xx/#!parametrics)
• Step 3: In order to program the kit, and to flash firmware onto the device, you use the PSOC creator tool (Infineon’s new ModusToolbox™ Software is not yet adapted to the PSOC 4100S Kit). At current, the PSOC creator requires a Windows PC. Follow the mentioned user guide in order to install the PSOC creator tool on your Windows PC. To do so, attach the PSOC kit to your computer, and create/ open the project titled CE16892 USB-HID Trackpad, which is part of the standard delivery. In this project will you find all design, header, and source files that are available to program and adapt the PSOC kit in order to act like a USB device.
• Step 4: In order to program the PSOC 4100S Kit to act as a pointing device, you need to press the SW3 switch during boot up of the device. This makes the device ready to let you flash your own boot-loader of the device. Follow the project documentation of the mentioned project (CE16892 USB-HID Trackpad.pdf) for step-by step instructions how to flash your device accordingly. Once finished successfully, the PSOC 4100S Kit works as a two-button mouse that is controlled by a trackpad. The device communicates with the host computer as a mouse using the USB HID protocol. As not only Windows devices use the USB HID protocol, it is also possible to interface the PSOC Device to systems on MacOS or Linux, once it is properly flashed.
• Step 5: Once you have flashed your PSOC 4100S Kit with the standard software for a USB device, you can remove the device from the PC, and use it as a pointing device on other computers as well. Connect the device to the computer, which runs your SSR instance. With your new pointing device, you can now use the Graphical User Interface of the SSR in order to realize your own spacial sound scenarios.

As described in the mentioned project documentation for the CE16892 USB-HID Trackpad, you can alter the complete behavior of trackpad, leftClick, and rightClick sensors, or the ganged sensor, and adapt the Capsense area to your needs before you flash it to your device. For a basic proof of concept, it is sufficient to use the standard configuration.

Infineon PSOC 4100S - Extended Integration

As described above, you can manage the SSR either by means of a mouse and the standard GUI, or you can use a network device, which is for example equipped with positional sensors. Using networked devices you can for example realize a head tracking device that is capable to change the sound depending on the movement of the listener‘s hat. However, such devices are either expensive, or they require a larger implementation effort.

A much simpler solution is a mouse-like device that works in 3 dimensions so that you can simply move your fingers in space in order to manipulate the sound experience.

You can implement such a device by means of the PSOC 4100S Kit, and the previously installed program

Refer to the mentioned guide to get a better understanding about the available parameters.