Its very simple. 3 rules should be applied while living in Japan, and you will be amazed by such rules.
First: Keep your mobile on silent mode and use headsets
I can say that I rarely hear a mobile phone rings. Either to inform a phone call, or an sms. Japanese people are very -very- concerned about having quite atmospheres, so their mobiles are always on silent mode. If the phone vibrates, they go outside the room to take the call. If they were in the streets, they take a corner to finish the call. If they had to walk while talking, they talk in a very -very- low voice that you would never hear. This way, they guarantee their privacy and their peace, since everybody is doing the same.
Same story with the headsets. If you are playing a portable game (mobile, DS, PSP.. etc) then you have to use a headset. I never saw a Japanese playing a game with a loud sound. That would be very annoying, especially in closed spaces. Its a very smart idea. This way, everybody is happy. Same case for ipods, mp3 players and music devices.
Second: Take your shoes off
You have to be ready for this. Whenever you enter a room, especially in a private place, you have to take your shoes off. I take my shoes off when I enter my lab. This way the lab is always perfectly clean. When we spent a night in a hotel in hakoneh, we took our shoes off before we entered the hotel. When I enter my own room, I have to take my shoes off, eventhough I have no carpet. But this is the Japanese style and way of living. Its a clever idea to keep the room clean.
Third: Be busy, all the time
You can just be doing nothing. You have to do something all the time. Its weird to sit in the train and just look at people around you. When I used to do that, I used to see people reading books, playing videogames, listening to music (with headsets ^-^ ) putting make up (yup, inside the train girls spend time putting make up), or asleep. But you cant just sit there doing nothing, staring out of the window or looking at other people. The train is only one example of many others, everywhere. When I learned that, I installed many games on my mobile to play while on the train.