Track Generation via Generative AI

I decided to enter the SparkFun/NVIDIA challenge for AI Innovation. My current passion is autonomous vehicles and in particular F1Tenth racing. F1Tenth cars are 1/10th scale RC cars racing autonomously.

I was thinking about how to mix my love for tech with competition and thought about making a Generative AI. But what kind? First, I thought about an AI for driving cars, but that's not an Generative AI. Then it hit me: race tracks in ROS are basically images so why not use Generative AI to make new tracks? This idea felt like the perfect way to combine my interests in a fresh and exciting direction.

I finally had a clear idea and was ready to bring it to life. This project was multi-faceted and the first critical step was to gather training data, specifically images or position data about existing racetracks. However, this task was far from simple. Finding a collection of racetrack images that were both consistent and licensed for use turned out to be a significant challenge. Despite extensive searching, I struggled to find enough suitable data for my needs.

I considered generating the images myself. Although I found some databases with coordinates or images of track layouts, they were insufficient for comprehensive training data. During my quest, I stumbled upon the Racelogic website, which had exactly what I needed: detailed track data. Unfortunately, this data was tightly controlled under a license agreement, restricting its use solely for the intended software to safeguard their collection efforts.

Respecting their reasons but undeterred, I proceeded to install the database. With some exploration, I discovered the data I needed in ".cir" files. These files appeared to be straightforward to read, offering a glimpse of hope for my project. It's important to note that this is a fictional example for illustrative purposes, not representing actual data from a real ".cir" file.

File created on 4/1/2024
[header]
satellites
time
latitdue
longitude
height
CAN1
CAN2
CAN3
CAN4
CAN5

[channel units]

[comments]
Log Rate (Hz): 1
Software Version : 1.9.8 (Build 004)
Created from KML File
Track Offset (metres):0.00;0.00

[column names]
sats time lat long height CAN1 CAN2 CAN3 CAN4 CAN5 
[data]
020 000000.00 +02245.32786 +07325.08806 +00105.42 +0.00000E+00 +0.00000E+00 ...
020 000000.00 +02245.32786 +07325.09342 +00105.42 +0.00000E+00 +0.00000E+00 ...
020 000000.00 +02245.32786 +07325.09543 +00105.42 +0.00000E+00 +0.00000E+00 ...

The... just means there was more data but all the values were zero.

Adhering to the principle that it never hurts to ask, I reached out to Racelogic's support team to inquire if I could obtain permission to use their files. I detailed my project goals, what the data would be used for, and the nature of the content I intended to release. To my relief, Racelogic was supportive of my project. Their primary concern revolved around the potential release of the GPS coordinates of the tracks. I assured them that my project would only involve releasing images generated from those coordinates, not the coordinates themselves. This compromise addressed their concerns, as the images derived from the ".cir" files lack the detailed data necessary for the files' original use.

Not every track in the dataset of 600+ tracks was utilized for the AI's training; I am only releasing images of the tracks that were used. Tracks that overlapped or had other qualities I didn't want for generated tracks were removed.

I am including the code I developed for reading the ".cir" files, making it available for anyone interested. However, it's important for users to respect and adhere to the licensing agreements associated with the files. Here's a sample of what the processed images look like after being generated from a ".cir" file. The lines are the inner and outer part of the race line and the start/finish line.

These images are processed to generate the final image used for training by filling in between the inner and outer line.

In order to augment training images, I took each track and rotated and flipped it to create a total of 4 tracks for each image.

Now that I have the training data I needed to actually train an generative AI. My first attempt was using a scaled down generative AI called Lightweight GAN. What I liked was the training time was low, but the results were also not very promising. The following is an image with 16 tracks generated by Lightweight GAN.

The initial approach seemed promising, yet I felt compelled to explore other options before dedicating more time to it. In my search, I discovered two more frameworks on a website based on the same paper but struggled to get them up and running swiftly.

After investing weeks into this exploration, I decided to shift my focus towards other Generative AI frameworks and ultimately settled on StyleGAN3. Once the setup was complete, I began the training process but quickly encountered a significant obstacle. The performance of my 4070ti graphics card was too slow. My estimates indicated it would take over 30 days to complete the training, and even then, the outcome was uncertain. Additionally, I suspected that the card's 12GB of RAM was another limiting factor.

In search of a solution, I decided to give StyleGAN2 a try, which was also included with the StyleGAN3 code. Fortunately, this switch proved to be a game-changer, significantly accelerating the training process. After adjusting the learning parameters a few times and processing 315,000 images, I achieved results that I was pleased with. There's room for further optimization. Adjusting the amount of training done, or the training parameters could provide a more diverse set of racetracks. Here is an example output from the Generative AI.

After the raw image is generated it is processed to make a final track image.

As part of my final results I generated 100 new race tracks and include those in the repository of this project.

In order to use these images as race tracks a new yaml file needs to be created with certain details about the track. This file can be generated using the TrackPreProcess utility. Using this you can generate full size tracks or 1/10th scale tracks for use for F1Tenth by simply changing the scale of the image. Here is documentation on the TrackPreProcess utility.

The TrackPreProcess utility can be used for reading the ".cir" files, processing raw images into final images, or generate a map yaml file from a final image.

To run use the TrackPreProcess executable with one of the following commands.

TrackPreProcess pre inputFolder

This will process all ".cir" files in the inputFolder and place them in inputFolder/wall subfolder. Please create this wall subfolder before running.

TrackPreProcess post inputDirectory OutputDirectory

This will process all images in the inputDirectory and place final track images in the ouputDirectory.

TrackPreProcess map imageFile1.png

This will bring up an UI where you can select the starting point, and heading of the initial starting position for a full size track (1:1) scale. This information will be written to a file named imageFile1.yaml.

TrackPreProcess f1tenth imageFile1.png

This will bring up a UI where you can select the starting point, and heading of the initial starting position for an RC car sized track (1:10) scale. This information will be written to a file named imageFile1.yaml.

Once you have a track, you can then use it with other programs to test whatever code you are working on. This way you can make sure your code can handle a large variety of tracks without having to generate them manually.

Here is an example using a lap optimizer from a generated track.

You can load the tracks into the F1Tenth simulator or ROS and work on your code to drive the car around the track in an optimal matter. Please note there seems to be an issue with the heading data in the F1Tenth simulator due to a limitation of the Nav 2 Navigation Stack. A bug I found using a generated track!

Steps to replicate TrackGen generation process and training.

Windows

I did not have an Nvidia AGXOrin during the initial development of this project, so I spent my time developing on Windows before I had an AGX Orin available.

Install Cmake, Anaconda, Python 3.8 (64bit), Visual Studio and vcpkg.

From an Anaconda command prompt setup an new environment called stylegan3 and install all the required packages. For details on setup for stylegan3 see this link. I recommend python 3.8 and Pytorch 2.1.2+cu118 (latest no longer supports cuda acceleration). Read more details about the Pytorch installation at this link. If you get everything setup correctly the rest is easy!

git clone https://gitlab.com/jesse_brockmann/trackgen.git
cd trackgen
git submodule update --init --recursive
git lfs install
git lfs pull
cd TrackPreProcess
vcpkg install --triplet x64-windows
cmake -DCMAKE_INSTALL_PREFIX=. --preset=default
cmake --build build --config Release --target install
cd ..
python patch.py
python generate.py

Linux/Jetson for Nvidia Orin with Jetpack 6 developer preview

sudo apt-get install git-lfs cmake libopencv-dev pip ninja-build
wget https://developer.download.nvidia.com/compute/redist/jp/v60dp/pytorch/torch-2.2.0a0+81ea7a4.nv24.01-cp310-cp310-linux_aarch64.whl
pip install numpy=='1.24' torch-2.2.0a0+81ea7a4.nv24.01-cp310-cp310-linux_aarch64.whl
git clone https://gitlab.com/jesse_brockmann/trackgen.git
cd trackgen
git submodule update --init --recursive
git lfs install
git lfs pull
cd TrackPreProcess
mkdir build
cd build
cmake ..
make
cd ../..
python patch.py
python generate.py

The packages mentioned are the ones I required to run that were not included by default with a Jetpack 6 dp setup. I use a link to a separate repository for the stylegan3 files, and also use lfs to store the pkl files hence the numerous git commands instead of the normal git clone. Then there are commands to build the TrackGenPreProcess executable. Then the patch.py is used to modify a stylegan3/gen_images.py to work with grayscale images (it was expecting rgb images). Then finally generate.py is used to create new tracks.

Setup to train network

This is a typical process for training like any stylegan2 setup. Feel free to experiment with different options to get the best results.

You will need to create the track.zip used here using dataset_tool.py in the stylegan3 folder. And then start training.

Start from the trackgen folder and then run the following commands

Windows

cd stylegan3
python dataset_tool.py --source=..\Data\ --dest=datasets\track.zip
train.py --cfg=stylegan2 --kimg=315 --gpus=1 --batch=4 --gamma=52.4288 --map-depth=2 --data=datasets\track.zip --outdir=training-data --snap=5 --tick=1 --metrics=none

Linux (Jetson AGX Orin recommended)

pip install tqdm psutil
cd stylegan3
python dataset_tool.py --source=../Data/ --dest=datasets/track.zip
train.py --cfg=stylegan2 --kimg=315 --gpus=1 --batch=8 --gamma=52.4288 --map-depth=2 --data=datasets/track.zip --outdir=training-data --snap=5 --tick=1 --metrics=none

With my AGX Orin set to maximum performance I had a rate of around 850 seconds per 1k images, for a total training time of around 3 days.

Once you have generated a new network file you will need to modify the generate.py file to point at the new generated network file. This is the line you need to change.

os.system(f'python stylegan3/gen_images.py --outdir={outputFolder1} --network=PreTrainedNetwork/network-snapshot-000315.pkl --noise-mode=random --seeds={seeds}')

You then need to point at your new pkl file and follow the standard generation steps.

This project could not have happened without the support of RaceLogic, Sparkfun and NVIDIA. Thank you for the opportunity to participate in this competition.

(1) The F1Tenth car image was from https://f1tenth.org/build.html and is licensed as CC BY-NC-SA 4.0 DEED.

(2) Track image downloaded from https://github.com/f1tenth/f1tenth_racetracks/ under the GPL 3 License.

(3) StyleGAN3 is licensed for by NVIDIA