In my experience the quickest way to get faster is to ride with guys, girls that are faster.
Fitness and machinery etc aside I have found you will get faster quicker doing this rather than riding on your own.You tend to push yourself a bit harder and for longer than by yourself and all of a sudden its not an effort as much, you've just picked up your average.
My experience over the years has proved this time and again, not just with cycling but from racing motocross, enduro, dirt track and even speedway back home.
I have also found that as I get older, even when my fitness is up, that I dont have the 'punch' or anywhere near the acceleration I did when younger. Not that I had much to start with but I just have less now
Thats the beauty of cycling though, its many things to different people and abilities. It doesnt matter how slow or fast you ride, MTB or road, touring even unicycling as long it brings a smile, then it cant be bad.
Carlsberg dont build Singlespeeds but if they did.......they would probably look just like mine.
I've spent half my life on bikes.........the other half I wasted