For most citrus the pith has no flavor at all, definitely not bitter. Most people object to the texture and blandness/lack of flavor, which I can understand. I have a bunch of citrus trees in my yard, and learned to not mind the pith, mainly due to laziness (I eat a lot of oranges) - and it’s a good source of fiber.