Trouble is that even with little charge teh voltage level could be not far from 3.7V.
A proper battery tester puts teh battery under some significant load whist measuring the voltage. But for home use this is a bit complex.
Get the 18650 batteries that stop charging when they get to a specific voltage. Then simply stick in charger. You cant overcharge these.