That doesn't make someone selfish or arrogant.
Let me introduce you to emotional bandwidth.
Every social interaction (positive, negative, or indifferent) consumes a person's limited emotional bandwidth. Some people have more than others.
Extremely lonely people might never have consistently exhausted theirs before, so if this feels like a foreign concept, beware that it's a thing you'll run into when the circumstances in your life are less lonely. But it's a real thing that all social creatures have to deal with.