Keep the Noise Down
MIT's new PAC Privacy approach keeps AI models secure and accurate — we no longer need to choose between privacy and performance.
With nearly every technological innovation, it seems like a new attack surface emerges for bad actors to exploit. We are seeing this in a big way with the latest wave of generative artificial intelligence (AI) tools to be released. These algorithms are trained on large volumes of data, and in many cases that training sets contain a whole lot of sensitive information. Since that information went into the model, it can be retrieved in one way or another.
Early models would sometimes give up the goods when someone simply directly asked the model for the information. Protections have improved since that time, and the retrieval of sensitive information is not quite so easy anymore, but in many cases, it is still possible. And given the value of this type of data to a crook, a lot of effort goes into finding new exploits. Better protections need to be developed if users are going to trust these AI applications in the future.
A small team led by engineers at MIT is working toward designing safeguards that prevent AI models from spilling their sensitive secrets. This is, of course, not the first attempt to solve this problem. But previous attempts have generally compromised the performance of the model to make it more secure. The researchers recognized that even if a model is perfectly secure, it is of no value if it does not perform well. So, they developed a novel technique that maintains performance.
The new approach, called PAC Privacy, addresses the trade-off between security and accuracy by introducing a more precise method for adding protective noise to an AI model. Traditionally, to protect privacy, engineers would inject noise — random variations — into the model’s outputs. This makes it harder for attackers to reverse-engineer the sensitive training data. But too much noise harms the model’s accuracy. The trick, then, is figuring out the minimum amount of noise needed to provide privacy without degrading the model’s usefulness.
The PAC Privacy framework automatically estimates the smallest amount of noise necessary to meet a specific privacy goal. An earlier version of the framework already showed promise, but the new, upgraded variant significantly improves its computational efficiency. Instead of calculating an entire complex web of correlations across outputs, it now only needs to measure variances, drastically speeding up the process and allowing it to scale to larger datasets.
The framework can be used on a wide variety of algorithms without needing to look inside them. This black-box compatibility means that developers can apply PAC Privacy to protect everything from medical image classifiers to financial risk models — without having to change how those models work internally.
The team also discovered an interesting link between stability and privacy. Algorithms that are more stable — meaning their outputs do not change drastically when the input data is slightly altered — require less noise to privatize. In testing, the team found that classic algorithms like Support Vector Machines, PCA, and Random Forests were easier to protect when regularization techniques were applied to reduce instability.
Through simulated threat scenarios, the team showed that models protected with PAC Privacy could withstand state-of-the-art methods aimed at exposing sensitive training data. Now the only question that remains is: Will the method stand up to highly-motivated attackers in the real world?
