Not hearing the difference is absolutely a thing. I took a university class on the nature of language and I still have clear memories of some of the example videos we watched when we studied the phenomenon. It’s a very “how is this possible” kind of feeling.

Iirc it just depends on the language(s) you spoke while developing. You could probably hear the difference when you were very little.