Oh yeah, you gotta charge them... that's definitely your problem.
Most cheap chargers out there aren't very good because they'll keep high current on a charged battery, making it warm and reducing its lifespan considerably. Make sure you get one with autodetection circuitry that will trickle charge once the batteries are full. I got this one ages ago and it is still one of the best:
http://www.thomas-distributing.com/maha-mh-c204f.htm