Now imagine him in a fez.

or a sprig of celery and overly-long scarf.

But Tom Baker didn't wear a bow tie and I'm pretty sure Peter Davison didn't, either. They were Matt Smith's thing.

...although Colin Baker and Paul McGann had those big, floofy things around their collars and Jon Pertwee wore all kinds of neckwear...
...and Patrick Troughton wore bow ties, too....

but then again the fez wasn't Matt's big thing. Through his time on the screen there were several things he called cool. The Fez and The Bow-tie are only a couple of the thing he called cool.
Other things including Stetsons, Jammy Dodgers, Bunk Beds,and glasses.
I'd even go on to say his main quirk during his incarnation was simply trying to be cool. Which seems like a mid-life crises followed by his becoming a crotchety, old man and actually died several times on his way to gallifrey.

Several times? I thought he died *billions* of times in that episode.

(You know, it never occurred to me until just now, but if he'd kicked the wall instead of punching it, he would have reached the TaRDiS sooner. Shoes are more resilient than knuckles. Not as dramatic, though. :)