Apparently it happens when there are more than 10 new comments, which takes up space and makes this icon shrink.