Viggo in LotR = hot. Viggo otherwise? Not so much. But I love the scruffy look. It does surprise me that I like his long hair since I'm more of a short hair on guys type of gal
Johnny in some things is hot and others, not so much. I never found him attractive in 21 Jump Street, but as he got older he became more attractive. Brad Pitt now is hot. Several years ago? Not so much. George Clooney gets hotter as he gets older too.
I can also admit when I find a woman attractive. It doesn't mean I want to do her, just means she's attractive. I think for men, finding a woman hot equates with wanting to sleep with them. So if they find a man hot they may be concerned that others will think they want to sleep with said man.