Do sprouts affect flavor?

No. And by the way, potatoes are safe to consume even if some sprouting is present, as long as the potato itself is firm to the touch. Just remove and discard any sprouts, then prepare the potato as desired.