this is something i never really considered or gave any thought to.
i normally just run 170's or 175's (but preferably 170's recently, now that ive put some thought into it)
this is because they are readily available
recently i got an amazing deal on a brand new set of howitzer ruktion cranks, external howitzer bb and wellgo pedals (all new) for £30.
as my second set of hussefelts were looking sorry for themselves (again! hate them!) i bought the howitzers as it was a deal i couldnt turn down (who would)
when i got them it turned out they were 165's and i think my hussefelts were 170's
my bike setup is not far off downhill (36t ring, 11-25 cassette, 16'' frame and im 5'9''ish) but its my commuter/urban assault vehicle, but i havent worried about clipping pedals for years (since being a novice, which now im not) so not fussed there.
got them fitted (with different pedals as the insert stayed on the pedal axle, again!!!
still yet to remove it!)
tested them out on my street, as you do, and noticed the difference for about 50 yards, then back to normal
thought they would be a problem on my half hour commute the next day but they were fine
and have been fine since
doubt you will notice the difference, if you did due to leverage im sure your muscles will adjust and then you wont notice!
sorry for the long post
go for it!!