Does it make any difference if you wear them the other way round?
Yes, it does. Stereo recordings presuppose that the sound from the left channel will be coming louder from the left earphone. It isn't that crucial for music. This is more about videos. When there's dialogue, and somebody stands to the left, their voice will be louder in the left earphone. The same rule works for car sounds, gunshots, and other stereo effects.
This article presented by Jason Zabowski Coppertino Inc.
"Hey! My name is Jason and Good Sound Quality means a lot for me. I’m researhcing the best Music Playback Solution and sharing my knowledge with you."