I generally buy my parts second hand. Ebay, classifieds, jumbles, car boots, you name it.
One of the best sources is to buy complete bikes. Sometimes when you need a particular part it can be worth buying a complete bike for that part. For some reason people sell parts at high prices and complete, but tatty, bikes for pennies. The great thing about this is that you can often sell the parts you don't want for more than you paid for the bike. The problem a lot of people have with this method is that it ends up very expensive because they find they end up restoring the bike they just bought as a donor as well as the bike they already have. The biggest issue being that they never finish restoring anything because they end up with fifty "donor" bikes all of which need restoring.