I've got some more information which will help explain why I've asked this question:

I have 3 HikVision cameras, and they have confused me for a long time, but I finally figured it out:

These cameras send RTSP packets without start codes (i.e. there is no 00.00.01 or 00.00.00.01 prefix to the packets). BUT, the packets are "Annex-B" encoded, because they have "emulation prevention bytes" added. In other words, in the byte stream I find a lot of 00.00.03.01 sequences. I have verified this by trying all permutations of decoding/encoding, etc, and testing it out on ffmpeg.

So what I am saying, is that it seems clear from this example that the presence of start codes is not 1:1 correlated with Annex-B escaping. And because of this, I'm not sure that it is a good idea to automatically remove the Annex-B encoding if you detect start codes.