In 1927, Werner Heisenberg noted that, in the standard interpretation of quantum mechanics, any knowledge of the position of a particle limits the accuracy with which it can be said to have a definite momentum (and vice versa, the more precisely we know the momentum the more uncertain becomes its position). This “uncertainty principle” has since been shown to be a fundamental aspect of all quantum theories, and similar relationships occur between many other pairs of “complementary” variables. This inherent uncertainty bothered many (including Albert Einstein) who felt that it should really just arise due to our lack of knowledge of the full state of the system and that it should disappear if we could identify and observe the missing, as yet “hidden”, variables.
In 1935, Einstein, along with Boris Podolsky, and Nathan Rosen noted an apparent problem with the uncertainty principle. Namely that in a physical process which emits two particles from a source (in opposite directions so as to conserve momentum), any measurement of the momentum of one would, because the momenta are opposite, determine that of the other. But then a separate measurement of the position of the second particle would allow us to have precise information about both position and momentum of that particle – which would violate the uncertainty principle for that particle. The only way out would be for the momentum measurement on the first particle to somehow affect the second particle by causing measurements of its position to become less certain (ie if the experiment is repeated many times the positions observed for the second particle would be more spread out than they would have been if the first particle’s momentum had not been measured). This effect would have to occur immediately regardless of the distance between the particles and so the choice of what to do to the first particle would seem to have an immediate effect on the state of the second – which should not be possible in a relativistic theory where influence cannot propagate faster than the speed of light. For this reason, Einstein (and others) felt that there must be more to the state of a particle than is captured in its quantum state or “wave function”, and suggested that a “hidden variables” theory would give a more complete picture and explain the connection without recourse to “spooky” action at a distance (SAAAD).
Other people, who were more willing to accept the inherently probabilistic aspect of the theory, argued that the quantum state for a system of two particles is not just the product of two one particle states but rather an “entangled” statistical combination of many such products which could reproduce the expected correlations between observations of the two particles without actually involving any super-fast signals travelling between them. The question remained though of whether or not all of the probabilistic results of quantum mechanics could also be explained as the effects of some “hidden variables”, whose values we couldn’t observe but which, if known precisely, would predict the behaviour of the system with absolute certainty.
One of the great ironies of the subject however, is the fact that a more detailed analysis of the “spooky action at a distance”, which was thought to be a major motivation for distrust in quantum mechanics in the first place, has led to perhaps the most serious challenge to the idea of hidden variables – while, in the end, not conflicting at all with quantum mechanics.
In 1951, David Bohm (who was very interested in the search for a hidden variables theory) proposed studying an aspect of the EPR situation which was easier to analyse because of not involving measurements of continuous variables like position and momentum, and not having to take account of the additional uncertainties in the position and momentum of the source. Just as in the EPR thought experiment, he considered the case of an electron-positron pair which can be created from a gamma ray photon when it interacts with an atomic nucleus. But instead of looking at positions and momenta, he just considered the spin (which is a relatively simple quantum property possessed by many elementary particles, and which we know must be opposite for the two new electrons as we started with zero angular momentum and total angular momentum is conserved).
Despite being a quantum property, the spin of an electron or positron is fairly simple to describe. No matter what direction we choose, when measured in our chosen direction, the spin can give only one of two values (which we might call up and down or + and -). Once it has been measured, another measurement in the same direction will give the same value, but any measurement in a perpendicular direction will give either + or – with equal probability; and after that, any repeat of the new measurement will give the same value, but if we go back to the old direction we’ll now get either +or- with equal probability (regardless of which way we saw it in the first measurement).
So just as in the uncertainty principle, measurement of one variable creates uncertainty in the other. But now the uncertainty is of simpler form (involving just discrete values rather than continuous probability distributions) and so might be easier to analyse.
In addition to giving probabilities for the parallel and perpendicular cases, quantum mechanics also predicts the probabilities for having the second measurement show spin up and down in any other direction – neither parallel nor perpendicular to the first measurement, with the probability of also being up when the first was up going from 100% when the directions are the same to 50% when perpendicular and 0% when they are the opposite; and in particular, when the angle is 45 degrees it gives about 85% up and only 15% down . (But I will put the actual details of that calculation on a separate page.)
The goal of a hidden variable theory might be to identify some property of the electron which predicts which way it will spin in (say) the x direction after having been measured in (say) the z direction. A first guess might be to say that each electron has a definite value for each of its x and z direction spins, but that the measurement process for either randomly flips the other. But that would involve immediate effect of one measurement on the other so we can’t allow it if we want to both keep conservation of angular momentum and avoid the SAAAD business. What we could do though, is require each electron to have a built-in program which tells it how to respond to any spin measurement regardless of what is done to its partner. This could just be a function which assigns either a plus or a minus value to each possible direction. The program can be changed whenever the particle interacts with its surroundings in a way that depends on the hidden variables of the surroundings, so once the spin is measured in any direction all of the other directions may have new values values of the spin function. Can we do this in such a way that the two electrons always give opposite results when first measured in the same direction? If there was just one possible spin direction we could just assign the two electrons opposite spins at the outset, and then there would be no more mystery about being able to know one spin from observing the other than there is in using your observation that you have the right glove from a pair that you previously split with a partner to predict that the partner will be certain to have the left. But when there are many possible spin directions it’s not so obvious.
In fact, in 1964, John Bell proved that we cannot assign spins in all directions separately to both electrons in a way that matches the predictions of quantum mechanics.
Here’s how he did it.
Imagine that each electron comes with a program or function which assigns either the up or down value to each direction. Each electron’s program is generated by some process we don’t know but which appears random to us. (So yes, we still do have probabilities, but only as a result of our own lack of knowledge. We are imagining that despite our lack of knowledge, each electron does have a specific response function.)
Now pick any two directions A &B, and for a pair of entangled particles with opposite spins, imagine measuring the spin of one particle in direction A and the other in direction B. Since the two particles are opposite, the probability of getting both to have spin up is the same as having the first particle up in the A direction and down (ie not up) in the B direction.
If A and B are the same direction then this probability is zero, and if they are opposite then the probability is one (since the spins must be opposite); but if A and B are perpendicular then the probability is 0.5 (since if a random particle has spin up in some direction then it is equally likely to give up or down in the perpendicular direction).
Now consider an angle of 45 degrees. In this case quantum theory predicts a probability of about 15% for both spins to be up (ie for two successive measurements of one of them to be opposite). But can we match that with the hidden random program idea?
Well, the 45 degree direction C is halfway between the two perpendicular directions A&B; and so, if the laws of physics are symmetric with respect to overall rotations, the fraction of all possible programs that give 1A+ and 2C+ (or equivalently 1A+ and 1C-) should be equal to the fraction that give 1C+ and 2B+(or equivalently 1C+ and 1B-) and also to the fractions that give other cases of a 45 degree separation such as 1A-2C-(=1A-C+) and 1B-2C-(=1B-C+).
But Bell noticed that for any three directions A,B,C, the cases which give either 1A+ and 1C- or 1C+ and 1B- include all of those that give 1A+ and 1B-. To convince yourself of this it might help to either list cases, or use a Venn diagram, or just work through the steps of the following inequality:
P(1A+&2B+)=P(1A+B-)=P(AB)=P(ABC)+P(ABC)<P(AC)+P(BC)=P(1A+C+OR1B-C-)=P(1A+2C-OR1B-2C+)
But when the angles from B-toC+ and A+toC- are equal, the corresponding probabilities must also be equal if the laws of physics are symmetric with respect to overall rotations. So in any hidden variable theory, if there is no further communication between the two spins after they have been created and separated, then twice the 45 degree match rate should be greater than or equal to the 90 degree match rate. And this conflicts with the quantum theory prediction (since 15%+15% is less than 50%). So, without actually needing to do the experiment, we can see that there is no local hidden variables theory which matches the predictions of quantum mechanics. In fact it is actually possible to do a real experimental test which shows a less than 50% correlation between the spins and so that (if the experiment has been designed and done perfectly) no local hidden variables theory can fit the data. Note that any additional randomness introduced by interaction with the outside world will have the effect of reducing the strength of the opposite spin relationship – ie bringing the match rate closer to 50% and so cannot be used to provide an excuse for the hidden variables’ failure. (But it could have been used to explain a failure of the experiment to observe the full strength of the quantum theory prediction, so this particular experiment would not have been a good choice for potential falsification of quantum mechanics. Indeed, designing experiments to maintain entanglement for extended periods is both very difficult and potentially useful – eg for quantum computing and cryptography.) However in many actual experiments there are experimental details which violate the assumptions of the idealized thought experiment and so provide “loopholes” which might explain the observed non-randomness and prevent the experiment from absolutely ruling out every local hidden variables theory. Some of these loopholes are related to the possibility of having not recorded all of the spins due to detector inefficiencies, others may be due to the lack of rotational symmetry in experimental setup, and others to the difficulty of getting the particles far enough apart and observations quick enough to rule out any speed-of-light signal getting from one to the other before the observation process is concluded. Nonetheless, over the years there have been successively more stringent efforts to close all of the loopholes and most physicists seem to believe that the experiments are now convincing (although it may never be possible in practice to close all of the loopholes completely).
Note: Bell’s general inequality P(1A+&2B+)<P(1A+2C-OR1B-2C+) can also be tested with other triples of directions. A popular choice is with ABC equally spaced around a circle (ie with 120 degrees from A to B, B to C, and C to A). And experimental tests are often easier to construct using oppositely polarized photons than pairs of electrons and positrons. So there are various different-looking discussions of the same idea. Unfortunately even some of the most popular ones, despite having admirable graphics and entertaining speakers, leave out key aspects of the situation and so create either a nagging uncertainty or a false sense of understanding. I hope that this discussion has closed all of the conceptual loopholes for you, but in case not please let me know and I will try to improve it.