TestPilot

What TestPilot is

TestPilot is a growing set of tools for helping developers grow the automated test coverage of Java server-side codebases using software that purely runs locally.

What TestPilot is not

TestPilot is not an AI-based interactive application-code generation tool. There are plenty of other solutions for that. TestPilot focusses on automated testing.

Why?

Although there are an ever-increasing number of tools and plug-ins that can help with this they are mostly running on the internet, which means that their use involves Intellectual Property (i.e. their source code) leaving the company to be shared with the AI and the company running it.

A lot of enterprises do not want to expose their Intellectual Property, specifically their internal codebases, to the outside world on the internet. This is especially the case in the Financial Industry.

These enterprises have large Java-based server-side applications that have been in production for many years, with poor code coverage by automated tests, making developers reluctant to make simple software changes, let alone enhancements or replacements.

What we need is fewer LLMs that create reams of application code and more that primarily help creating and maintaining automated tests, especially for pre-existing codebases.

As we are all in the end dependent in our daily lives on the reliability of these applications it makes sense to provide these developers with a locally-running AI-driven toolset that focusses on increasing the test coverage by automated tests as well as the keeping the maintenance overhead of the test suites as low as possible.
The latter will likely require refactoring of the code structure, and the aim is that TestPilot will also be able to give guidance with that.

How?

TestPilot consists of mainly 2 components:

• An IDE plugin that interacts with the developer while using their favourite IDE.
  The initial focus is on JetBrain's IntelliJ IDEA, as this is one of the most-used IDEs used by the developer's for whom TestPilot is meant.
• The Ollama server, running the Llama3.1:70b model locally on the developer's workstation using the excellent AMD Radeon Pro W7900 GPU.
  The Radeon Pro W7900 is currently the only GPU compute+video card that can locally run the full Llama3.1:70b model without needing to take out a small mortgage for a rival GPU compute-only card + an additional separate video card.

As you can see no data is leaving the company, it isn't even leaving the workstation that the user uses, which is a very important property of the system.

The graph database is a future improvement that is in the works to mine the structure of the project's codebase to enable TestPilot to, together with the local LLM, make refactoring suggestions that will make tests more robust to change, but this is currently not functional yet so this will be added at a future date.

The journey so far

"What a time to be alive":

The good
Every week new AI models are being released that make last week's prime-candidate-for-self-awareness look like a gibbering zombie, at least when it comes to writing code.

There is no doubt that Llama3.1:70b with its 128K context size is really impressive and the one to use as you can really build on it.
When it was released just over a week ago it pretty much wiped the slate clean.

The Ollama team somehow very quickly adds support for these to their system, which is great as you can nearly immediately start using them.

Additionally the Ollama team itself have greatly and rapidly improved Ollama, for example by adding the ability to load multiple models at the same time and running multiple queries at the same time which were only added a month ago.

The bad
Every week new AIs are being released that make last week's prime-candidate-for-self-awareness look like a gibbering zombie.

This means that it is hard to ignore the new models, especially as their code generation abilities have improved by leaps and bounds, superseding prior model customisation work.

The ugly
Despite all the arguments about the effects of AI one the thing that AI models definitely do is make other AI models obsolete.

Which in the end can cause a lot of wasted effort, the price of progress.

What's Next?

The TestPilot project is an ongoing project that aims to improve the quality of Java server-side codebases, and there is still plenty to do:

• Incremental improvements to the plugin such as support for more operating systems
• Fine-tuning of the code interpretation and generation
• Getting the graph database system that mines the developer's project codebase to generate refactoring suggestions that make the system more test-friendly fully working.
• Extend support to other IDEs
• Add support for more computer languages than Java

Etc.