There continues to be speculation on how to interpret this sort of experiment so it is worth reiterating the standard quantum mechanics view of the Kim et al case since it is the seminal experiment. Firstly, 'choice' is exercised randomly by BSa and BSb in blocking (reflecting) one or both alternate idler paths. When this happens it is generally accepted that it is impossible for any interference to show, so we will consider below the situation when the paths are not blocked. The signal and idler photons are entangled and this prevents any interference being exhibited at D0 because the total wavefunction is a 'mixed state'. However the wavefunction 'collapses' or is 'reduced' on detection of the signal photon at D0. After collapse the idler photon wavefunction has the two slit wave superposition restored (no longer a mixed state), allowing potential recombination at BSc, when it gets there, with a phase difference determined by the x-position of the detected signal photon (very much a non-local effect). The phase difference depends on the different distances from each of the two slits to x. Assuming they get passed BSa and BSb, the superposed idler waves combine at BSc going on to either D1 or D2 depending on phase difference and precise path lengths.[1] There is nothing retroactive in this picture.
In general with entangled pairs of particles correlations are preserved, whether of position or phase (or it might be spin) when this is measured (detected) for one of the pair. In this case the phase difference (due to the two slits) of the incoming waves at BSc is correlated with that of the signal photon at D0 and determines the probabilities of the recombined wave leaving the beam splitter to D1 or D2 to build the R01 and R02 charts - probabilities depending on degree of cancellation or coincidence. There is a normal distribution of photons at D0 (not shown) and at R03 and R04. The R charts added together must be normal so it follows that the interference pattern at R01 must complement that at R02. It is misleading to claim that in this experiment we know which path the photons in these two charts have taken at the slits for it is quite clearly 'both ways'.
Standard quantum mechanics does not talk about particles in flight (pace the earlier discussion where they are hypothetical) - if there is anything in transit it is waves[2] and mathematically 'which way' is meaningless. However the de Broglie-Bohm interpretation does have particles and they do not go both ways (but there is an accompanying wave that does). De Broglie-Bohm mechanics equally well accounts for the results of this experiment[1] but cannot say which way the photons have gone.