Have you sprayed the bushes/spacers with something like WD40?
Unless you are lucky, just trying to knock out the shaft with a drift and hammer, you are going to damage things. Try using my method of using the nuts with washers to ease the shaft from side to side, applying penetrating fluid to loosen corrosion between shaft and bushes/spacers.
As I said there's not a lot of threads on the ends of shaft so you'll have to alternate from side to side. When you've got it moving keep spraying it with WD40 so that it penetrates right in to loosen the corrosion. Only then use a drift and and hammer.
A suitable sized bolt (smaller in diameter so it can pass through spacers) can be used as a drift but you will more than likely damage the end of shaft if you have to hit it hard, that's why I used a centre punch so I could hit the centre of shaft.
If you damage, flatten, the end of shaft it will jam in the spacer as you try to knock it through. Support the swing arm during thumping the shaft with tool number 1.