Well, I think the difficulty is fine, especially if that is what you are going for, but the difficulty ramp from before and when you get the wall jump to that next section is HUGE. Like there isn't enough time to get that good yet so you just have to keep dying over and over to get good enough to do that section.
I might be going a little far here, but if you put some more time into developing this into a full game (6 to 10 hours), or even full mini-game (like 3 or 4 hours in length), I think you could probably publish and sell it, with some marketing and such of course.
If you want any help with some music, let me know. My friends and I play around and write all kinds of music all the time.
As far as the down input goes, it works but you seem to have to hold it down while you are on a ledge to make yourself fall. I think it would be nice if it was more of an instant or near instant effect.
P.S. Add controller support!