I remember back in seminary a professor excoriated me for selecting unpopular, if not virtually unknown books as influential in my life. I had demonstrated a lack of spiritual maturity by not agreeing with his choices.
There's a lot of literary spinach out there -- stuff that's supposed to be good for you, even if it's boring. Well, I don't really care what people think of my tastes in art, literature, or anything else. Every now and then I get the bug to try out heavy literature. Recently, it was Kerouac's On the Road. Before that, it was Nabokov's Lolita. But if it doesn't grab me, I put it down and read something else.
Years ago, I met a high school girl who kept a copy of Atlas Shrugged around. She would pull it out and conspicuously read from it so that people would be impressed by her intelligence. And I think that many of the works that make it into and stay in the canon of great literature do so only because rejecting them would make a person like a dullard. Sort of like abstract art.
As for me, I don't apologize for what I like and what I don't like.
Which is not to say that there is not an advantage to a culture asserting an aesthetic canon. A common body of literature (or other arts) gives cohesion to members of that culture. Darmok and Jalad at Tanagra and all that. But at a certain point, it's time to strike out on one's own and set one's own aesthetic values.