Why are the generic male endings -er and -or accepted as gender neutral but -man isn’t?

The archaicness of -trix is indeed very very relevant to the topic.

I agree with Jason Whyte’s answer, I’ll just elaborate on it.

In the past of English, gendering was overt, and feminine actor suffixes were quite marked. –er was masculine and had a –ress counterpart; –or was masculine and had a –trix counterpart; –man was masculine and had a –woman counterpart (kinda).

When English ideologically moved away from gendering of roles, –trix was pretty marginal already, so people just forgot it was ever there: dominator/dominatrix is the only instance where -trix is alive and well. –ress is far from dead, but English is pretty ungendered much of the time, and it was easy for –er to generalise from masculine to generic. Not all the time: the US prefers she’s a server to she’s a waiter—but note, it’s still serv-er. The masculine connotation of -er was weak enough, and -er was generic enough already in meaning, that it could be ignored.

But –man? Well, it’s identical to man.

At this point, you could retort that –man in Old English meant “human being” (“man” was were), and people should have accepted that –man was generic. Well, they could have, but they didn’t. People interpret the meaning of morphemes in a synchronic paradigm (what other words and suffixes do I know right now, not historically). The synchronic identity of –man and man has been too compelling for people, and (this is also crucial) the use of –man as a suffix infrequent enough that it was never as generic as –er.