Why are earphones labeled Left and Right?

Does it make any difference if you wear them the other way round?

Yes, it does. Stereo recordings presuppose that the sound from the left channel will be coming louder from the left earphone. It isn't that crucial for music. This is more about videos. When there's dialogue, and somebody stands to the left, their voice will be louder in the left earphone. The same rule works for car sounds, gunshots, and other stereo effects.