The jockey wheels are in good nick, and I bought a new cassette and chain when I got the bike, so I don't think the issue rests there.
Since it shifts up just fine, and the top half of the cassette (easier/bigger gears) shifts down just fine, I figure it has to be shifter related?
From what I...