Virtual particles are never observed directly, so (subject to the limits of experimental error) we don’t actually ever see any violation of conservation of energy.
What virtual particles are is just a part of one particular method for calculating probabilities of events that we do see; but this is not even the same as saying we observe them indirectly, as the various possibilities with different numbers of such particles all contribute to the overall calculation – with no specific numbers ever being required to actually exist.
The use of virtual particles is analogous to Feynman’s path integral approach to quantum mechanics where, as an alternative to solving the Schrodinger equation by traditional methods, Feynman noted that the probability amplitudes predicted from it for going from one event to another could also be calculated by adding up contributions from all conceivable paths between the two events (including unphysical ones). But neither the unphysical paths nor the unphysical particle number histories need to be considered as anything that actually happens.
Another point that is often made in answers to this question is that the contributions from paths or particle histories that violate conservation of energy are inversely proportional to the time durations of those violations in a way that is consistent with Heisenberg’s uncertainty principle \Delta E \Delta t < \frac{h}{4\pi}. But I am not sure how much this helps – other than to explain how (as pointed out in yet a third set of answers) “laws” of physics are not absolute but just expressions of the limits of what, according to current theories, we expect to see – and indeed conservation of energy can appear to be violated if we try to measure things too quickly (though the “violation” can be interpreted as just due to our inability to measure both energy and time with sufficient simultaneous accuracy).