These calories are 'empty' without much nutrition, even though you are still consuming calories. The loss of these minerals is one reason flour is 'enriched'. They remove the nutrition and then try to add it back in again. Moving to whole grains and unrefined sweeteners allows you to feel more full in part because you are getting minerals from the food. The first thing you'll notice after making the switch is you can eat less volume and yet feel more full. I love that.
So AMP UP YOUR GRAINS...your body (and waistline) will thank you for it.