When shingles are manufactured, there are mineral granules pressed into the hot asphalt. These granules create the color of your shingles and protect them from UV rays, as well as make them fire resistant and more durable. As your shingles are exposed to the elements, it is not uncommon for small numbers of granules to come off and get washed down into your gutters. They will look like little rocks and you might see them here or there as your roof ages.
If you are seeing huge numbers of granules, to the point where it looks like there is gravel in your gutters or coming out of your downspouts, your roof might be nearing the end of its useful lifespan. However, there are other factors that can cause granule loss, such as heavy rain, hail, or damage from foot traffic. It is normal to see some granules loss on these occasions and does not necessarily mean your roof is failing.
If you are seeing a large number of granules on a regular basis, and you start to notice that your shingles look spotty or bare in areas, you should call a roofing professional to take a look before you have a leak.