The great thing about this topic is everyone has an opinion, and they vary widely. Here is a shot from the 35th anniversary of Repack this last October 21 with every variety of clothes from jeans (JB , CK, Alan Bonds and Billy Savage wore them as an old school costume tribute) to baggies, to lycra under baggies, to knickers (myself and others) to long tights, to lycra shorts. Just goes to show that no one fashion is predominant, at least around these parts.
To me, THAT is mountain biking at it's best. Loads of different folk getting together having fun on bikes.
Anyway, the lycra thing was brought in by the emerging "professional" racing scene in the late 80's/early 90's... I don't think lycra was worn before that, was it?
On a road bike anything baggy or loose is going to rub and chafe pretty quickly, so lycra is really the practical option. Off-road you need that bit more protection, sometimes a bit more warmth, pockets to store the odd essential item (i.e. cake). On and off road have their distinct styles and it quite a practical difference. Of course you can wear lycra off-road, no problem but it's best used in race situations or in very hot weather.
I'm still trying to find some nice riding trousers that don't cost the earth. They seem to be coming back in for riding, which is good to see, as unless it's really warm some well cut trousers are way more practical than shorts.