If it’s flavor you’re referring to, potatoes can certainly impart a slight nuance of ‘earthy’ fragrance upon preparation, due to the natural cultivation within the nutrient-rich Idaho soil, and to a much lesser extent in the finished prepared flavor. This is normal. However, ‘gritty’ is more of a texture description. Just as some seafood (especially clams) can impart a slight grittiness from the sand in their growing environment, so too can some potatoes. Especially if the potatoes were harvested in areas of the state that have more of a sandy soil base. Despite washing at the packing facility and washing at point of preparation, traces of tiny particles may indeed be the stowaway grit you detect. The best remedy to avoid all such minute traces is to peel the potatoes and rinse thoroughly.