My hair keeps falling off?

I have natural hair, I RARELY straighten or blow dry it. I have wavy hair so I don't use a curler or w/e it's called. I don't use any hair products besides shampoo. I wash my hair every other day. I also don't brush my hair often...sometimes non at all(but it's not a problem because it looks brushed). What am I doing wrong here? Sometimes I would brush my hair after I wash it and I would get a handful. It's freaking me out because I don't have thick hair to begin with...I have normal hair and if this keeps happening it will be THIN.