Lately I've been thinking about how to test a sophisticated piece of equipment that might attempt to deceive the tester. Every company that makes equipment wants their product to perform well when evaluated. There are ways they can cheat.

Recently I was testing the audio performance of an external USB soundcard, the Creative Audigy NX2, which claims to do 24-bit recording and playback. At first glance, the dynamic range appeared to be amazing. When I connected just a terminator to the analog input, the recorded signal was all 0's! That would mean that the analog front-end and the analog to digital converters have a noise floor that is more than 140dB below the maximum level. That is just too good to be true. After a few measurements and analysis of the recorded 24-bit WAV file, it became clear the manufacturer had programmed the sound card to send blocks of all 0's to the computer when the mic input gets below some level. This is a cheap way of making the card appear really good on a simple audio benchmark that compares the maximum output signal with the minimum output signal and attempts to deduce the dynamic range. They implemented their cheating trick in a very straightforward way, so it was very easy to detect when I looked at the probability density function of the recorded signal. It should have been something like gaussian, and it was, except for a HUGE spike at p(vin=0). But if they anticiplated that the pdf would be examined, they could have shaped the pdf by adding the right amount of noise at the right time.

So now I wonder whether we need to bring a security mindset to the laboratory to test modern equipment. The embedded code running in the USB soundcard has logic that is intended to circumvent the audio measurement process. So maybe now the audio measurement process needs to circumvent the circumvention logic. For example, a secret signal could be applied by the measurer to the analog input of the soundcard. The digital output data of the soundard could then be checked to make sure it contains the secret signal. Of course it is ridiculous that this is necessary, but this is our world, like it or not.