The directory frequency should not be incremented if the user leaves the directory right away and probably did not intend to go there. This sequence should not give the first directory a higher priority, because it's obvious that it was not the intended destination:

$ z ff
# goes to ~/repo/MyProject/.git/objects/ff
$z ffm
# goes to ~/repo/ffmpeg which is what was intended all along

The algorithm I suggest for removing the increment from accidentally incremented directories is:

• Was $OLDPWD incremented within the previous minute?
• Was this shell's previous search also a match for the new target?
• If the above are true, the previous search was probably meant for this target rather than $OLDPWD and $OLDPWD should be decremented.