Let’s-a Go, TensorFlow!

Machine learning techniques, like those that are used to power self-driving vehicles here on Earth, have now been deployed in the Mushroom Kingdom. Mario and his compatriots can now drive circles around Bowser and their other foes — all without doing any actual driving. Talk about a power-up!

The security researcher and hardware hacker known as stacksmashing was impressed by Kevin Hughes’ popular TensorKart project that leverages TensorFlow to add self-driving capabilities to the racing video game Mario Kart 64. However, this project relies on an emulator to play the game. Not quite content with an emulator-based experience, stacksmashing set out to create a modified version of TensorKart that runs on actual Nintendo 64 hardware. This project, which has been named TensorKartRealHW, is open source and available on GitHub.

The first part of the hack involves using adapters to convert the Nintendo 64’s A/V signal to HDMI, such that image frames can be transferred to a laptop computer. Each frame is fed into a machine learning model, which outputs the predicted best values for controller output. This output is then used to control the game.

To collect training data, stacksmashing wrote a Python script that captured controller input along with the corresponding frames from gameplay. Game speed was effectively slowed down by pulsing the accelerator control to allow for smoother steering inputs to be captured. Also, game images were cropped to remove the on-screen map, and other irrelevant, confounding information. The machine learning model was then trained on this data set.

Next, an interface between the laptop, where the model runs, and the Nintendo 64 was needed. A Raspberry Pi Pico and the cable from an old, broken controller fit the bill nicely for this purpose. While this solution did introduce some lag, the speed of the Pico’s Programmable I/O kept it to an acceptable level.

After 150 epochs of training, the model was able to guide Mario around a race course with near-perfect control. The training data was collected from two courses, and that excellent performance is only seen on those courses. On courses that were not seen in the training data, the performance appears to be quite poor, indicating that the model is not well generalized. Substantially more training data may be required to produce a well-generalized model.

While playing Mario Kart 64 is not really a chore that we need to automate, it is still a very cool hack that one can learn many lessons from. Be sure to check out the source code for TensorKartRealHW, as well as for the original TensorCart, if you are looking for a fun idea for your next project.