What is the plural form of the word “vertex”? Why is it irregular?

Vertices.

Why? Because the word is straight of out Latin, and Latin has a lot of declensions that look weird from the perspective of English.

In particular, the plural vertices suggests that the singular should be vertix, just like the singular of matrices is matrix. There are a lot of –ices plurals corresponding to –ex singulars in Latin thoough, and historical linguists mumble something about analogy and/or vowel reduction. Sihler’s New Comparative Grammar p. 67 is my source for the mumblings.

Why use the Latin vertices rather than a regular vertexes? Because (a) vertex is a learnèd, rare word; (b) vertex is a word straight out of Latin, that never really was assimilated into English; (c) even common Latin words following the –ex -ices pattern don’t readily get regularised into English, precisely because –ex is so redolent of Latin.

We say indexes now, but we still say indices too; it’s the more common meaning of index, as what’s in the back of a book, that gets the English indexes plural.