Take off your hat (civilian, that is) whenever you are indoors, except in a synagogue and except in places which are akin to public streets: lobbies, corridors, street conveyances, crowded elevators of non-residential public buildings (department stores, office buildings). Apartment house elevators and halls are classed as indoors, and so are eating places!
Take it off whenever you pray or witness a religious ceremony, as at a burial, outdoor wedding, dedication. Take it off whenever the flag goes by. And fergodsakes take it off when you have your photograph taken for the place of honor on her dressing table – and take it off before you kiss her!