Do I hyphenate “upper middle class family”, if yes, then how?

Yes, hyphenation is less fashionable than it used to be, and yes, people think that it is finicky to introduce a distinction between two levels of punctuation.

But may the fire of a thousand Harts and Fowlers rain down on all respondents, for not one of them suggesting as an alternative something involving an en-dash:

upper-middle–class family.

Regrettably, that would these days be regarded as an affectation. Particularly as most people don’t know how to type en-dashes (even if they do know what they are).

It is true, though, that upper middle class family without hyphens will usually be understood just fine, and hyphens are more avoided these days than not.

I disagree with Paulette Smythe and agree with Jeff Christensen, btw: the nesting is surely (upper middle) class, not upper (middle class).