There's been a lot of public discussion lately about the risks of increasingly capable AI and how to keep those systems under control. One proposed approach is monitoring and independent evaluation: using other systems to evaluate what an AI is doing and determine whether it's behaving as intended.
AI systems have already demonstrated that they can deceive their monitors, cheat evaluations, and find ways around controls meant to constrain them. Then there are systems deliberately built by people, companies, or governments whose intentions we have no control over in the first place.
A lot of the monitoring is going to have to happen at machine speed. That means using AI to monitor AI.
If several AIs are monitoring another AI, how do we know whether the monitors themselves are reliable? Putting one more supposedly trustworthy AI above them and asking it which monitor is right doesn't solve the problem. It just moves the problem up a level.
Who judges the judges?
It's a version of a problem my friend Andrés Corrada-Emmanuel and I have been working on for about sixteen years, although neither of us was thinking about large language models when we started.
Andrés approached me with an idea he'd been developing out of some of his earlier work in computer vision and error estimation. We thought there might be something there worth pursuing commercially, and incorporated Data Engines in 2010.
The basic question was: What can you learn about the accuracy of algorithms labeling a stream of data when you don't have ground truth to compare their answers against? Could you get the next best thing to ground truth from the information you do have?
One algorithm gives you a label, but no independent information about whether it's right. A second gives you more information, but not enough. If they agree, they can still both be wrong. If they disagree, you know at least one is wrong, but their answers alone don't tell you which one is wrong. Three turned out to be the minimum that gives you enough information to start drawing useful conclusions about both the labels and the labelers.
Three isn't automatically enough. The method depends on getting useful information from the relationships among the algorithms' answers. If their errors are sufficiently correlated, that information can disappear and the method can become effectively inoperable.
It also isn't just a complicated way of taking a majority vote. Sometimes the majority is your best answer and sometimes it isn't. There are cases where the less common label is more likely to be right, depending on the particular label and the voting pattern.
Andrés developed the mathematics that turned the basic idea into a workable method, including what could be inferred from the relationships among the labelers' answers and under what conditions. The approach uses observable statistics and algebraic relationships among their answers without requiring a theoretical model of how the algorithms made their errors. Because it evaluates the algorithms from their behavior on the data they're actually seeing, differences between that data and what they were trained or tested on don't by themselves invalidate the method. That can be particularly valuable when changing conditions make previous measures of their accuracy less reliable.
We saw a lot of possible uses for it. One was generating useful training data without having people curate a ground truth dataset. Another was evaluating algorithms on a live stream and using their recent performance to change how much influence each one had. If one had been performing better, its labels could count more heavily; if another had been performing poorly, they could count less. Some of what we wanted to do, particularly making those adjustments while a system was running, required calculations that couldn't be done fast enough at the time.
There was also the question of whether the additional evaluation was worth doing in the first place. That depends in part on the cost of using it compared with the cost of the errors it might prevent. If a recommendation algorithm suggests a rom-com when you'd rather watch a buddy movie, probably not a big deal (Netflix might disagree). The risk of failing to recognize a pedestrian in front of a moving car is a very different calculation. Autonomous vehicles were one of the applications we looked at for exactly that reason. If several algorithms are trying to determine whether something in the road is an object the vehicle needs to avoid, you would rather know how much confidence to place in their answers before the vehicle has to act.
The same basic problem exists in medical imaging, where the cost of being wrong can run in either direction. Missing a tumor can obviously have serious consequences, but so can finding one that isn't there. If several algorithms are evaluating the same image, having most of them agree doesn't mean the majority is right.
Large language models created another application neither of us had anticipated. Andrés realized that the same approach could be applied when AIs themselves are being used as evaluators or judges. That's become particularly relevant as more attention has turned to independent evaluation and automated monitoring as part of AI safety.
The same problem remains: we need some way to evaluate the monitors without simply assuming that one of them, or a majority of them, is right.
Andrés never stopped developing the mathematics, and NTQR is where he's taken it. NTQR isn't going to work for every problem. Sometimes the available data and the relationships among the judges simply don't provide what the mathematics needs. Some systems already do, while others can be modified or expanded to make NTQR possible, or engineered with it in mind from the beginning. And even when NTQR works, the information it provides has to be useful enough to justify the cost of getting it.
When those conditions are met, NTQR can use patterns in the monitors' responses to determine which evaluations are logically consistent with what it observes and which aren't. It doesn't have to know what the responses mean to do that. In some cases it can establish that something has gone wrong without being able to tell you exactly which monitor is responsible.
Andrés has also made the calculations fast enough to do things that weren't practical when we started, including evaluating recent performance quickly enough to adjust a system while it's operating.
In some systems, NTQR can provide a logical alarm when the evidence can no longer show that defined monitoring requirements are being met, without first having to identify exactly what went wrong. Even an AI refusing or failing to answer can, where the system allows it, become another observable response rather than simply disappearing from the analysis.
The math doesn't decide what "safe enough" means. People still have to decide what performance they require, what risks they'll accept, and what a system should do when those requirements aren't being met. But NTQR may contribute to solving a particular part of the AI control problem: how to evaluate the systems doing the evaluating when we can't simply assume they're reliable.
Sixteen years later, the problem Andrés and I started working on has turned up in a place neither of us anticipated. As AI systems increasingly become responsible for evaluating other AI systems, "Who judges the judges?" has become a practical question about how we keep increasingly powerful AI under control.