In ancient Greek, how is the root determined in τὸ τεῖχος?

Humphry Smith’s answer is right, but let me spell it out a bit more.

We come up with stem suffixes in proto-Greek, to explain the diversity of case endings of classes of nouns—a diversity between dialects of Greek, as well as trying to make intuitive sense of where they came from. The nouns in your details, neuter teîch-os, masculine triē´r-ēs, proper name Themistokl-ēˆs, are all explained by proto-Greek stems ending in –es-.

Why do we do that? Because if you look at the cases other than the nominative, they’re pretty similar. We account for that by saying they’re actually underlyingly the same.

A few things to keep in mind whenever we’re reconstructing proto-Greek:

• First: Attic contracts vowels (mooshes them together); Epic Greek tends not to contract them, and proto-Greek is more obvious if we move back from Homer rather than Attic. So (interleaving the masculine noun triērēs where it’s different):

Attic:

teîch-os
triē´r-ēs
teích-ous
teích-ei
teích-ē
triē´r-eis
teich-ōˆn
teích-esi

Epic:

teîch-os
triē´r-ēs
teích-e-os
teích-e-ï
teích-e-a
triē´r-e-es
teich-é-ōn
teích-e-ssi

See that -e- in the Epic? You can’t see it as clearly in the Attic. That’s part of the reason why it’s an -es- noun.

Oh, and Themistocles? The uncontracted form is Themistoklé-ēs. The only difference between Themistoklé-ēs and triē´r-ēs is that Attic contracts the last two vowels.

• Second: Proto-Greek -s- between vowels is deleted. (In Latin, its equivalent turns into -r-.)

So our Epic paradigm now becomes:

teîch-os, triē´r-ēs
teích-es-os
teích-es-ï
teích-es-a
triē´r-es-es
teich-és-ōn
teích-es-ssi

And that’s where the -es- stem comes from.

The catch is that horrid nominative singular, teîch-os, triē´r-ēs.

• Third: When you’re reconstructing nominals, always leave the nominative singular till last. They tend to be… different.

The masculine nom. sg can be understood by compensatory lengthening. As you’ll know from other third declension nouns, the nom. sg. masc ending here should be -s. The stem is -es. So Proto-Greek nom. sg ‘trireme’ would have been triē´r-es-s. When the first s drops out, the vowel before it lengthens to compensate: triē´r-es-s > triē´r-ē-s.

That leaves the neuter singular -os. In fact, cracking open my copy of Sihler: New Comparative Grammar of Greek and Latin §296, the nominative singular is as old as Proto-Indo-European: it’s –os all the way back.

Now, whenever you see o in Proto-Indo-European, you immediately think of the ablaut alternation of ø/e/o. Sihler notes (§297) that “Scholars have long maintained that the nom./acc. form -os, albeit of securely PIE date, is secondary; presumably the zero grade nom./acc. of the the type krewH̥2-s [Greek krea-s] is a relic of a more original state of affairs.” What he’s saying is that the -o-s looks suspicious, and it may be a ø/e/o alternation; the word for ‘meat’, which has nothing rather than e or o in front of the -s, was the original way of doing those neuters.