Has it always felt like that? The fact that it's done the same thing with two different setups may mean it's something to do with the headtube, perhaps it needs reaming/facing? I did have to do this on a bike of mine with similar issues.
Yes, fine when riding but when lifting the bike up from the stem there's a very slight 'rock' to it, almost as if the steerer has nudged from the 'rear' of the headtube when on the ground to the 'front' of the rear of the headtube. Same feeling with two very different set ups. No issues when riding that I've noticed though.
Difficult to describe accurately!
1990 Ridgeback 704XT 1990 Raleigh Dyna-Tech Encounter (SWMBOs) 1992 Orange Aluminium 'O' 1992 Orange Prestige 1994 Orange Clockwork 1996 Orange P7