Is there such a thing as “taking things too literally”?

Yes, and there’s a linguistic pragmatics set of principles at work there, over and above the inherent limitations of language pointed out by Daniel Bamberger : see Daniel Bamberger’s answer to Is there such a thing as “taking things too literally”?

The Cooperative principles defined by Grice are a way of making sense of how people don’t take things literally. The underlying understanding, when you’re talking with someone, is that your interlocutor is not being an arsehole, and is not talking to you just to troll you. You assume that what they are telling you makes sense and is relevant. So if their literal meaning comes across as trolling, you try to think up figurative and indirect meanings, which make what they’re saying make sense.

This kind of second guessing of literal meaning underpins humour, figurative language, metaphor, literature, wit, allusion—all the potent stuff in language. The fact that the meaning is indirect in such expressions, and has to be teased out by listeners assuming that you are not trolling them, is a big part of their potency.

And of course doing that teasing out of indirect meaning requires a large amount of emotional intelligence and social context—which notoriously puts autistic people at a disadvantage. But yes, there is a societal expectation that you will use Gricean principles to make sense of figurative language, and if you fail to do so, you are taking things too literally for that social norm.