Google’s NSynth Super Uses Machine Learning to Generate Sounds for Electronic Music

Traditional instruments get their unique sound from their construction and tuning. A guitar, for example, gets its tonal quality from the thickness of the string, how much tension is on that string (tuning), the length of the string (modified by pushing a fret), and also the wood the neck and body are made of. A synthesizer creates unique sounds by generating specific wave forms—which have grown more complex as the underlying technology has progressed.

NSynth, which was developed by Google’s Magenta team, combines the two with machine learning to produce entirely new sounds. On the most basic level, NSynth takes two or more recorded sounds from instruments, and then merges them into an entirely new sound. It’s important to note, however, that this isn’t as crude as just overlaying the sounds and mixing them — which would be trivial with any hardware mixer or audio editor.

Instead, the machine learning capability of NSynth is used to extract the various tonal properties of a recording to distill the essence of a particular instrument. That is then merged with another sound to make something entirely new that couldn’t be created with traditional instruments, synthesizers, or mixers. Using just 16 original source recordings, NSynth was able to generate over 100,000 completely new sounds.

The Magenta team then built NSynth Super, which is the physical device that allows musician to use NSynth. It has four dials to select source sounds, and then a touchscreen which forms a 2D spectrum between those sounds. The musician can select an arbitrary point on that spectrum, which will give them a new sound that’s a combination of the weights of the four original sounds. The new sounds can then be used with a traditional MIDI instrument to make music.

The electronics are built around a Raspberry Pi 3 running Raspbian Linux (note that the new Raspberry Pi 3B+ is not yet tested with these instructions), and a custom PCB used to read the inputs and control the outputs. A microcontroller on the PCB manages the physical inputs: there are four rotary encoders on the four corners for instrument selection; six potentiometers below the interface to control the position, envelope, and volume settings; and a capacitive grid on the surface of the PCB (exposed through the top layer of the case) used to select the mixing point of the four instruments.

If you want to build your own NSynth Super, the files are available over on GitHub.