Ἡλληνιστεύκοντος

Last post had its share of egg on my face, showing systematic overcounts of word forms in the corpora. This post is another healthy serving of omelette, correcting the lemma counts given in Lerna VIa. The overall story is:

• There are less distinct word forms in the PHI #7 corpus than I thought
• There are less scribal alternate forms left in PHI #7: if an editor thought they knew better than the scribe, the scribe’s form is left out of consideration
• There is less dialectal and orthographic wiggle-room allowed to PHI #7
• So as a result of all this, the count of lemmata distinctive to PHI #7 has crashed: ignoring proper names, 3,800 lemmata that the lemmatiser thought it saw in PHI #7 are no longer there.
• The count has still crashed, even though I’ve added a fair few lemmata to deal with PHI #7—the most frequent names, the overlaps with Trapp’s dictionary, a few stragglers from DGE—as well as some dialectal grammar and some more respelling rules. I’ve picked up around 800 non-names and 1200 proper names; so I’m down by 1800 lemmata from before, rather than 3800.
• I could have kept going to add more names than that, but it’s been two weeks already, for gorsakes.
• OTOH, because I’ve added extra names in particular, recognition of the TLG has slightly improved. So there are a few more lemmata for just the TLG-based corpora. (*Very* few.)
• I also did some debugging of orthographic variation in lemmata, which resulted in some conflation of variants.
• So if you ignore proper names, the TLG lemma count… actually ended up losing a few lemmata. (Again, *very* few: a couple of hundred lemmata each way.)

So.

I also had a tally including also-rans analyses:

In all of this, I’ve not been paying the PHI #7 corpus that much attention, though I did make a point of slipping it into the LSJ corpus. (The LSJ coverage of inscriptions and papyri are in fact why I called up PHI #7 in the first place.) I knew there would be extra lemmata there, and this lemma count is the PHI #7 disc’s chance to shine. PHI #7 has added 6.5% more word instances to the TLG’s, but 16% more word forms, and 6% more lemmata! That’s phenomenal!

… What on Earth am I talking about? Remember Zipf’s Law: the cumulative number of word forms that turn up is inversely proportional to the instance count for each word form. It’s a Long Tail. If you add 6% more word instances, by the time you’re already at 95 million instances, you should be getting… well, I can’t do the maths, but you should be getting at most hundreds of new lemmata, not (as the table above shows) 12,000, of which only a couple of thousand are proper names. The 10,000 more lemmata of ordinary vocabulary shows you that the inscriptions and papyri—the Greek of daily life and of far flung dialects—has a very different vocabulary from the Greek of literature.

Of course, that you get 16% more word forms in PHI #7 means there’s a lot of different inflections in the corpus that lie outside the TLG’s ambit, because of all the non-literary dialects represented in the inscriptions. It also means a lot of misspellings that didn’t belong in the TLG, as well.

In VIa, I went into an extended riff extrapolating how many more lemmata of Greek could turn up. Let me attempt that again, this time with more detail on proper names—but *not* including proper names in the final estimate.

The reason proper names don’t belong in a final tally is worth restating, because not enough people are laughing at the notion. When we want to know how many words of English there are (which we shouldn’t, but I’ve already been through that), we don’t add the New York State White Pages to the Oxford English Dictionary, and we don’t start screen-scraping geonames.org. We recognise that proper names are a different kind of thing from normal words (although the boundaries are fuzzy); and we also recognise that it’s problematic to say a name belongs to one language and not another.

Does Κόρινθος count as a Greek name, even though it has the prehellenic telltale -νθ-? Well sure it does. Does Ομπάμα count as a Greek name? Or Σαίξπηρ for Shakespeare? Surely not. But what about the older declinable transliteration Σακεσπήριος? Doesn’t that at least look Greek? What about Αὐρήλιος? But then again, what about Ἰσαάκ? Is Αμπντουλάχ not a Greek name? But does it become a Greek name when it was hellenised, as the Byzantines did, as Ἀβδελλᾶς? And is counting these names as part of the vocabulary of Greek a meaningful thing to do?

Well, better not to count proper names in the final tally at all; but let me add the counts I do know of, just in case someone is curious.

• Right now, the TLG lemmatiser knows about almost 42,000 proper names. That includes most names of the Strictly Classical canon; a fair few names from later literature (including lots of Byzantine surnames), the names in Smith’s Dictionary of Greek and Roman geography , and the thousand-odd names I was shovelling in over the past fortnight, to deal with the inscriptions and papyri.
• Pape-Benseler went into its second edition in 1863, which increased it by a third. It covers geographical, personal, and mythological names in Ancient literature, and has some coverage of later stages. It has good coverage of such inscriptions as were known at the time, and is starting to notice papyri—though remember, this is thirty years before the discovery of Oxyrhynchus. And the dictionary is reasonably good about conflating variants.

  Benseler does not say how many names he has in total, but he does say that Alpha under his revision went from 3820 names to 6120. Extrapolating based on LSJ, that should mean 38,000 names overall. There are clearly lemmata in Pape-Benseler that aren’t in the TLG lemmatiser: I add 500 names because of dealing with PHI #7, and that was only dealing with names occurring 10 times or more in PHI #7. How much more am I missing? No idea. But I’d be surprised if it was more than 10,000.

• In the following, I need a sense of how many of these names are personal, and how many are geographical. The Heidelberg word lists for papyri are a bit more reluctant to conflate variants than I prefer, but at least they list personal and geographical names separately: 8838 personal, 2637 geographical. Good enough for me, I’ll say personal numbers :: place names are 4:1.
• 1863 is a long time ago in epigraphy, and the Lexicon of Greek Proper Names has been running for the past three decades to record the torrent of names found on inscriptions. It avoids mythological names (which are covered well enough in literature and Pape-Benseler), and it also does not do geographical names. It’s ongoing, but its online search knows of 35,000 distinct names of people (whereas Pape-Benseler has 38,000 names of people, places, and gods). Now, the TLG lemmatiser recognises 17,600 distinct names, personal and geographical, in the ancient inscriptions on the PHI #7 disc. Guessing that 14,000 of those are personal names (4:1 ratio), that means it’s missing at least 21,000 personal names.
• The Leuven projects recognise 16,000 personal names in the papyri (with 7,000 extra variants), using the Duke Documentary Papyri corpus. The TLG lemmatiser recognises 9,600 distinct names, personal and geographical, in the same corpus on PHI #7. Guessing that 7,700 of those names are personal, it’s missing at least another 8,000 names.
• Some of the Leuven names will overlap with LGPN; but the Egyptian names won’t. Let’s say that all up, we’re owed at least another 27,000 personal names. And using that 4:1 ratio again, another 5,000 place names. Heidelberg counts 9,000 personal names to Leuven’s 16,000, and Heidelberg counts 2,600 geographical names; extrapolating up, that’s consistent with 5,000.
• That’s not even scratching the surface of Byzantine and Modern names (let alone Σαίξπηρ or Ομπάμα, or the Thessalonica and Environs phone book). But so far, we can guess 42+27+5=74,000 names.
• Flipping things around, there are 72,000 unrecognised capitalised words in PHI #7. That does not mean 72,000 missing names: lots of these will be misspellings of known names that the lemmatiser isn’t dealing with, or different inflections of the same name. And those names are in the scope of LGPN and Leuven. I’d say the personal names are already accounted for in the 27,000 (say) personal names of the two initiatives.
• There are a further 42,000 unrecognised capitalised words in TLG. Most of these won’t be in LGPN and Leuven—though some will be in Pape-Benseler. Most of these by far are from post-Classical texts, and they include ancient gazeteers. (Ptolemy’s Geography alone accounts for close to 3,000 unrecognised names.) How many of these are legitimate novel proper names? Again, no idea, but by this stage we’re getting into one-offs, because all proper name word forms occurring more than 7 times in the TLG have been added to the database. I’ll guess 30,000. There’ll be some overlap with Leuven and LGPN, but not a lot, because many of these names are Byzantine.
• As mentioned, the TLG is maybe 70%, maybe 75% complete for Byzantine literature, and only starting to go into Early Modern literature. It does have a lot of Byzantine surnames through church deeds (which account for 5,000 unrecognised capitalised words); so it’ll have a reasonable cross-section. I haven’t gone through the Byzantine proposopographies though (285-641, 642-1265, Palaeologan), to work out how many surnames they’ve unearthed in sum.
• And I have not spent quality time with the Attica or Thessalonica or Nicosia phonebooks.
• So at least 70,000 proper names to go, adding up to something like 110,000 proper names, and that count only goes up to the Fall of Constantinople.

Anyone who wants to start boasting of the 110,000 proper names of Two And A Half Thousand Years of Greek needs to be smacked upside the head with all three volumes of the Dictionary of American Family Names, and have the printout of all 8,000,000 places on geonames.org dropped on their foot. Because all of those count as proper names of One Year of English, by the same criterion.

(The Blogger Writing These Lines enjoyed contributing to the Dictionary of American Proper Names, even before he realised its value as a tool of percussive persuasion.)

So. Banishing proper names, we’re left with 173,000 lemmata, as guesstimated. How much is left to go again? As it turns out, I’m doing the same guesstimates as before—but they make more sense without including proper names:

• I keep my guesstimate of 20,000 lemmata more from Trapp (including texts not yet added to the TLG and volumes not yet published), and 10,000 lemmata more from Kriaras (ditto). That’s 203,000.
• There are words in LSJ that are not represented in this corpus. The biggest gap is the mediaeval Latin-Greek glossaries, with 1,000 missing lemmata; but there are several other oddities. The latest I’ve encountered, under ἐλεφαντουργική “of or pertaining to ivory-working”: the 1161 AD commentary to the astrologer Paul of Alexandria, writing in 378—and last published in 1588. (The irony here is, the same adjective turns up in the rather more mainstream Heliodorus, a century beforehand.) But again, once the PHI #7 texts are in, and with the changes in text editions between the original LSJ and the TLG—not to mention the rejected scribal forms—I don’t think there’s more than 3,000 lemmata to add. That takes us to 206,000.
• I’m inclined to revise my extrapolation for DGE downwards. Volume I updated may have 3500 lemmata not in LSJ, but it’s competing not only with Bauer, Lampe, and Trapp, but also with the LSJ Supplement—which on its own adds 10,000 lemmata to LSJ, and which also has made a point of covering more inscriptions and papyri. I haven’t taken the time to do any counting with DGE. It’s a long plane trip tomorrow to Montreal—so maybe I will.

  But there’s no way Volume I has 3,500 lemmata not also in LSJ/Bauer/Lampe/Trapp/LSJSupp. DGE looks like taking 20 volumes if and when it finishes. (I wasn’t planning on living until 2100 AD to find out.) If there’s just 500 novel lemmata in Volume I, that means 10,000 novel lemmata all up; if 1000, then 20,000, as I proposed last time. I’m feeling jaundiced, but I’ll still give them 20,000. That takes us to 226,000 lemmata, up to the fall of Candia.

Ούφ. On those figures, English still wins, 🙂 though not by much. The level of precision I’ve given is of course illusory, and in a following post I will tackle what is a more sensible question: how much vocabulary do you need to recognise n% of a text. But these counts should at least be indicative.

You may have noticed an extended radio silence for the last couple of weeks in the series counting lemmata. The people at the Magnificent Nikos Sarantakos’ blog, where the good fight against Lerna is fought, know why: I found some problems in the way I was counting lemmata in the inscriptions and papyrus corpus (PHI #7), which I’ve been nowhere as familiar with as the TLG corpus. As a result, I’m down 2,000-odd lemmata from where I thought I was. Because I spent lots of posts on how contingent and provisional any count of lemmata is, that should not be that big a deal: a ±1% in the lemma count is within the bounds of what can happen when you fix first-cut errors.

Still, it’s embarrassed me enough, now that people are starting to quote the Lerna VIa count of 211,794 (including Nikos Sarantakos, fighting the good fight), that I tried to get to the bottom of it. In the process, I’ve worked to treat the PHI #7 corpus less cursorly than I had done. Cleaning up problems in the PHI #7 markup, and clueing the lemmatiser in on some of the peculiarities of the dialects in the corpus, mean that the counts would give a more accurate picture of what was going on with those texts. The problem is, the longer I spent fixing my handling of PHI #7, the more the lemma count fell—*even as I was busy adding lemmata from elsewhere* (DGE, Pape-Benseler, Foraboschi). Erk. The counts are more accurate (with a catch I’ll talk about), but they’re not what they were.

I’m going to air some of the dirty laundry here, to cement the point yet again that any count of lemmata is going to be unstable. After that, next post is going to revise the counts that need revising. Then, the promised posts that got derailed: how many of these lemmata count as Ancient; relating lemma counts to recognition percentages (which is the only way lemma counts are meaningful); and distinguishing word variants from lemmata.

The first issue was when I wanted to count how many lemmata should be considered Ancient. I realised I had not been counting a couple of thousand lemmata from Lampe’s and Trapp’s dictionary (I-VIII AD and “IX-XII” AD) as post-classical. That did not particularly affect the accuracy of recognition for the TLG (as I confirmed by rerunning the program), but it was distorting the numbers: there are less “word forms of good pedigree” than I said there are. So you’ll get new numbers for that.

The second catch was when I found a bug in how I was extracting word forms from the PHI #7 corpus, which meant that several hyphens were being ignored—so a hyphenated word would be extracted as two separate words. Once I fixed that bug, I also noticed that some of the markers that a word was fragmentary weren’t being picked up. For instance, I knew that notation like …]atisatio[ indicated bits of a word were missing from the papyrus or inscription; I didn’t know that PHI #7 was also using dashes, like – – ]atisatio[ – –. Fixing these problems results in less complete word instances extracted—but of course, more correct word instances extracted. Even if some lemmata that looked like being there were no longer recognised, there should be more correct long words turning up. So that should not cause any drastic drops in the size of the vocabulary.

The next three problem fixes seem to be what’s caused issues. Papyri are spelled phonetically, by the norms of Koine Greek, so the lemmatiser allows for some spelling variation: ι for ει, for instance, or ω for ο. Inscriptions and legal deeds from Late Byzantium need to allow for a lot more spelling variation, because of the many Ancient phonemes that had ended up pronounced identically: so ι could now be a misspelling of any of η ει οι υ υι.

Archaic inscriptions, on the other hand, may have a narrower range of respellings than papyri (depends on how early), but they also have different spellings of their own, because they use different versions of the Greek alphabet: ω and ου were Ionic innovations in the alphabet, for example, and what conventional Ancient orthography spells as ω and ου, most inscriptions before iv BC spell as just ο. So unlike papyri or church deeds, a system dealing with inscriptions has to allow ο to stand for ω or ου.

The lemmatisation run over PHI #7 that I’d reported was allowing all possible respellings from all periods indiscriminately. So an XIV AD document was being allowed the same latitude in spelling as a vii BC document.

Yeah, you can see how that might be a problem. I fixed this by allowing different respelling rules for the three parts of the corpus: the ancient inscriptions, the papyri, and the Christian inscriptions (which run all the way to Ottoman times). There’ll still be some wrong respellings, because each part corpus spans a long period. But it’ll be a lot better than allowing XIV AD iotacism in a vii BC text. Of course, restricting respellings means that lemmata that were being over-recognised in texts now aren’t. That’s fair enough.

I also tried to restrict the lemmata that were allowed for each part of the corpus, to prevent absurdities. Modern Greek words couldn’t be allowed for Ancient texts of course, but they do show up in the late Christian inscriptions. The ancient inscriptions do keep going well into Roman times, so I couldn’t ban Koine lemmata from there; but I did try to keep recognition plausible, by blocking from the papyri and ancient inscriptions any words unique to Trapp’s dictionary.

That’s underestimating both Trapp and the papyri. The papyri keep going until Greek yielded to Arabic in Egypt—a generation or so after the Islamic conquest, so VIII AD. Trapp, OTOH, badges itself as IX-XII AD—but it also sets out to fill in gaps left by other dictionaries, so it can be the only place where late papyri get covered. So some lemmata that should have been allowed for the papyri were being blocked. But having checked, only 150-odd legitimate lemmata were affected (and are now back in). So that wasn’t the major disruption.

The other problem, as far I can tell, was that PHI #7 allowed in its markup both the word or phrasing the editor thinks the text is saying, and (in special brackets) the odd wording the scribe actually wrote; e.g. lemmatisation {4lmmeatsiantion}4. If an editor has decided to correct lmmeatsiantion as lemmatisation, I decided, I shouldn’t be trying to analyse both. The editor’s fix should count as the word instance for recognition: the “misspelling” (as the editor has judged it) shouldn’t be considered an independent word. It looks like, in the process, some words LSJ says existed no longer turn up, because LSJ didn’t trust the editor as much as I do. But all texts from a papyrus or inscription get filtered by the editor publishing it, and making sense of it—just like all the literary texts in the TLG. So that’s the consistent thing to do.

All up, skipping proper names, 3,500 odd lemmata are no longer turning up as recognised. OTOH, 700 lemmata are now newly turning up that weren’t before. Those numbers are still subject to change; but most of the 3,500 lemmata that disappeared should have disappeared. The scribal originals like lmmeatsiantion‘s arguably shouldn’t have disappeared, and I may end up revisiting them down the road. But I’ve already spent two weeks trying to deal with the vanished 3,500, and I shouldn’t be holding postings up much longer.

To compensate for the missing 3,500, I went through the PHI #7 corpus, and looked more closely at what kinds of words weren’t being recognised—making sure that words occurring frequently in the corpus were accounted for. That involved some tweaking in the allowable spelling variations, and some filling in of the more obscure dialects’ grammar.

I had no idea what the Arcadian first declension genitive was like—or how it’s spread. Arcadian τρίταυ /trítau/ “of the third” corresponds to Homeric masculine τρίταο /trítao/ (Attic τρίτου /trítoː/), but it’s also spread to the feminine, displacing Proto-Greek and Doric τρίτας /trítaːs/ (Attic τρίτης /trítɛːs/). Arcadian τρίταυ reminds me of the Esperanto -aŭ ending; I wonder if I’m the first person to have had that mental short-circuit.

Beyond that, if the dictionaries that the TLG lemmatiser already knew about didn’t account for frequent word forms, I checked it in DGE. After all, part of DGE’s reason for existence was to broaden the coverage of LSJ into new finds in inscriptions and papyri. For lowercase words (i.e. excluding proper names), I went through all word forms occurring more than twice in the corpus; DGE is up to εκ-, and I did end up adding new lemmata from DGE, unique to this corpus.

The count of lemmata I added to the vocabulary from DGE… was 12. This surprised me, especially because even between α and αλ—for which DGE went back and redid Vol. I—there were a few word forms still unaccounted for on PHI #7. Going down to word forms occurring just twice or once will account for a lot more than 12 lemmata from DGE; but it won’t account for thousands. The remaining gaps even after DGE is something I’ll be looking at again: I’m curious to work out what’s going on. Of course, PHI #7 is nowhere near a complete corpus even for 1995 when it was published—let alone now, with the continuous stream of inscriptions and papyri being transcribed and published. Only the Athenian curse tablets from Audollent’s 1904 collection, for example, are in. (So when I looked at how καταχθόνιος and χθόνιος are used in the tablets for a paper, I had to do eyeballing as well as keyboard searching.)

I also wanted to improve the recognition of proper names particular to PHI #7, where the lemmatiser is really struggling: It now recognises 46% of all capitalised words, vs. 89% of all lowercase words. As I keep saying, proper names shouldn’t count at all, but a couple of thousand instances of Πεθέως drawing a blank from the lemmatiser was a bit much for me. Moreover, if the lemmatiser isn’t told about a proper name, it will end up making wrong guesses about what the lemma actually is. There are several inscriptions-only names that I was able to find in Pape–Benseler; but the big store of unrecognised names are in the papyri. And there’s a simple reason why so many names from papyri drew a blank from the Greek lemmatiser: they’re not Greek names, but Egyptian.

Of course, adding 500 or 1000 Egyptian names to improve Greek word recognition sounds suspect, right? But no more suspect than adding Hebrew names to improve recognition of words in the Septuagint, or Roman names to improve recognition of Cassius Dio. That, after all, is why proper names don’t count when you count lemmata.

I’m using Foraboschi as my Egyptian phone book; it’s the update to Preisigke’s Namenbuch, which Foraboschi updates—and which seems to be AWOL at the moment in transit from Monash University to the University of Melbourne. My bloody fault for not waiting to drive over to Monash on the weekend—it’s just 10 minutes up the road from my place.

People whose day job it is to look at names in papyri (several projects based at Leuven) have already been counting the proper names in the Duke Database of Documentary Papyri, which is what PHI #7 uses for the papyri. So they’re doing the electronic counterpart to the dead tree phone book I’m sampling. The Leuven projects have come up with 26,000 name variants in the corpus, in 16,500 lemmata—and the majority of them are Egyptian, and unknown to other corpora of Greek (although a few of them make it to Athanasius of Alexandria or the Desert Fathers, who after all were also Egyptians). I’m not proposing to sit down and add 16,500 lemmata to the lemmatiser database: this is not my day job. I’m aiming at adding around 1,000, as triage prioritising the most frequent names; that’ll account for uppercase word forms turning up 10 or more times in the corpus.

So, I’m going to tell you I know of 1,000 Egyptian names in PHI #7, when the Leuven papyrologists know there are 16,500? Why yes. Just like I’m telling you I know of 35,000 proper names in the TLG, when there are 42,000 uppercase words in the TLG unaccounted for. I don’t know how many names there are, and that’s not what the current lemma count is about. But I do know how many names account for n% of the corpus, for a suitably large number of n%. The name count is not open-ended, but it is pretty large—larger than for common nouns.

In fact, the good folk at Heidelberg Uni Centre for Research on Antiquity have produced lists of lemmata in papyri. They’ve got around 22,000 lemmata, half of them names. So Leuven knows more names than Heidelberg knows—presumably because they’re using a smaller corpus. I know less than either. And the more papyri turn up, the more names and nouns and verbs will turn up. Lemmata are open-ended.

But while Heidelberg’s 11,000 names can turn into 16,500, it won’t turn into a million. And while 175,000 lemmata without names can turn into 173,000 when I fix PHI #7—and maybe 220,000 once both the dictionaries and the corpus is complete up to 1453 or 1669—it’s not going to turn into five million. Even if you count all the variants of dialect and spelling and phonology in lemmata, as I’ll attempt in the final installment—which is how Leuven get from 16,500 names to 26,000, and the OED gets from 230,000 lemmata to 610,00 variants: even then, you’re not getting to a million. (My current back-of-the-envelope calculation without names is around 350,000.)

OK, I’ve got some Egyptian names to go before I revise the published counts.

[Counts in this post have been corrected in Lerna VId]

At long last, after nine posts of teasing, will I finally give the punters a count of lemmata of Greek?

Why yes. Yes I will. And then for a change, I will also set to work inflating it, to extrapolate from the current corpus and lexicon I have access to, to how much larger it could conceivably get.

Ready for it? The count of lemmata, known to date to the TLG lemmatiser, and recognised in the four corpora we’ve set up as they stand to date, is…

Did I have to quibble even here? Why, of course I did. The lemmatiser makes sense of Milesian numerals like αϠοα = 1971 and χξϛ = 666, but including them in the vocabulary of Greek as lemmata is a bit much. And dictionaries do not include proper names. So if you’re going to compare the headword count in LSJ with the headword count in OED, you won’t be including proper names in your count. Proper names are not exactly open-ended in count, but they do work differently from core vocabulary, they come from a lot more sources and cultures, and knowing lots of people doesn’t really prove your vocabulary is more expressive.

So if we’re comparing the TLG corpus to the OED’s, we’ll say 162,000. OTOH, adding proper names is most of my fun with the TLG lemmatiser: Athenian courtesans one minute, Albanian chieftains the next. In terms of the lemmatised search engine, they are search targets like any other. So if we’re not comparing the TLG corpus to any dictionaries, we’ll say 202,000 lemmata.

Not So Fast!

Only if you’d asked me two years ago, I’d have told you 231,000. And that was when I was recognising 90% of all word forms—as opposed to now, when I’m recognising close to 94%. Does that mean the Greek language has lost 25,000 lemmata in the intervening two years, even as the lemmatiser now recognises 60,000 more word forms? No. The lemmatiser has just gotten more discerning about when it claims a new lemma has shown up.

There is a lot of ambiguity in dealing with three thousand years and six dialects of Greek, and incomplete dictionaries. The lemmatiser has been allowed to make up its own lemmata (more on this below); it does this to cover gaps in dictionaries, whether they’ve gotten to pi or omega. But in the past two years, the lemmatiser has been constrained to make up a new lemma only if it doesn’t have a “legitimate” alternative, already recorded in a dictionary.

The lemmatiser has also gotten better at conflating variant stems under the one lemma. That’s a huge issue, which will have to wait for a couple of posts: the number of stems I count as distinct lemmata is in several ways different to the number LSJ counts as being distinct lemmata. There are 216,000 lemmata in the overall corpus, following the lexical database’s definition of when two stems belong to different lemmata. But its definition will be different to someone else’s definition. As we’ll see, it’s not always clear from the dictionaries when they consider two lemmata distinct, or whether they should when they do.

All this should make you distrustful of any lemma counts I give you. As well it should. Counting lemmata is an artefact of how you set about counting lemmata; different criteria, and different methods of analysis, will give different results. As will different sizes of corpora. So in this and the next couple of posts I’m going to stretch the lemma count, then shrink it, then stretch it again. (And then I’ll bring this thread to an end; there’s Mariupolitan dialect waiting to be blogged about.)

Fuzzy Boundaries

To begin stretching, I’m going to allow for the uncertainty of lemmatisation. The TLG lemmatiser, confronted with much too much ambiguity, ranks potential analyses of word forms as belonging to different lemmata. The counts I’ve just given are for the “winning” lemmata for each word form. If I include the “also-rans” in the analyses, then I’ll also be counting lemmata which never give the preferred analysis for any word form—but which the lemmatiser keeps in reserve, in case one of them turns out to be correct after all. You will get search results if you look up those lemmata in the TLG. You will also get lots of warning, saying “but this form is probably X instead, and that form is probably Y instead.”

If I include these also-rans in the counts, the counts go up to:

This gives a curious result. There are lemmata which the Strictly Ancient corpus rejects as implausible for all its forms; but somewhere in the subsequent mediaeval morass, a word form turns up for which the rejected lemma makes the most sense after all. Of course, a lot of those lemmata rejected for Strictly Ancient Greek will turn out to be Byzantine after all. So it makes sense they’d become more acceptable, once bona fide Byzantine texts are included.

More dictionaries

There are two constraints on what a lemmatiser recognises: how many words it knows about, and how many words you’re asking it to recognise. Increase the corpus—as we did by excluding and including Byzantine texts—and it will find more lemmata. Give it more dictionaries—as the TLG did by adding Lampe, Trapp, and Kriaras—and it will also recognise more lemmata. So these counts could be bigger with more texts (and more texts are being added), and more dictionaries.

You can increase your dictionary size by allowing the lemmatiser to do what human beings do: make words up. You can make words up from whole cloth, as a random but plausible sequence of sounds. But that’s fairly rare in human language. What is much more usual is making up words based on existing words, using rules present in the language (derivational morphology). The lemmatiser does know something about Greek derivational morphology: as a result, the TLG counts include some 15,000 lemmata that are not in its dictionaries, but are derived from lemmata that are. Two thirds of these are from prefixing prepositions to verbs, which is quite productive in Greek. One third is from derivational morphology forming new stems through suffixes, and those proposed analyses are more tentative.

But derivational morphology will only catch some words. Otherwise, if a lemma is unrecorded in the dictionaries that the lemmatiser has access to, then that lemma won’t show up in the list of lemmata recognised: you have to tell the lemmatiser that πιθανοθηρία exists, for it to make sense of πιθανοθηρίας. If you give the lemmatiser access to more dictionaries, it will recognise more words.

Much of the TLG is Byzantine, and more of the TLG is going to be Early Modern; so the fact that the dictionaries of both stages of the language are currently stuck at pi means there are future volumes of those dictionaries that the lemmatiser hasn’t been given access to yet (because they don’t exist). That means a lot of lemmata in the corpus going unrecognised, and being missed in these counts.

My back of an envelope can beat your back of an envelope

How many? Here start the back of the envelope calculations: take them with several satchels of salt. Trapp has finished six volumes out of a projected eight, and adding Trapp’s lemmata to the lemmatiser has accounted for 25,000 lemmata newly recognised in the TLG. The remaining two volumes (to be completed by 2013) should add another 8,000 lemmata to the TLG. So 202,000 will go to 210,000—give or take a couple of thousand. Remember those 15,000 lemmata the lemmatiser is recognising, even though they’re in no dictionaries? Some of them will turn up in the forthcoming volumes of Trapp.

The same holds for Kriaras. With the 2.5 million words of Early Modern Greek in the TLG, adding Kriaras’ dictionary to the lemmatiser accounts for 2650 lemmata. Since Kriaras is up to 15 volumes of a projected 19, that would mean we’re owed another 550 lemmata. Except, the TLG is not going to stay at 2.5 million words of Early Modern Greek: it’s expanding deliberately into the Early Modern corpus, and there’ll be a lot more lemmata added to the TLG as it does so. The 15 volumes of Kriaras have added 9900 lemmata to the lexical database, so with with the 2650 already seen in the corpus, we can expect another 9900 lemmata once the entire Early Modern corpus is in and Kriaras is completed. That takes us to 220,000 lemmata.

In fact, because we haven’t run out of Byzantine learnèd texts to data enter into the TLG corpus either, 8,000 more lemmata from Trapp is an underestimate. There are 35,000 headwords in Trapp that the TLG does not already recognise from its other dictionaries; but the current corpus only accounts for 25,000 of them. There’s some spelling variation and derivational morphology clouding the results, but all up, and assuming headwords and lemmata are the same (which they’re not), we should expect not 8,000 more lemmata once all the texts are in (and all the spelling variation accomodated for), but 20,000. That takes us to 232,000 lemmata.

“A headword is not a lemma”?! But the definition of “lemma” *is* “headword”! I’m being a little idiosyncratic in my usage: the source dictionaries each have their own headwords, but I’m calling “lemma” the canonical lexeme that the lemmatiser uses, to bind those headwords and variants together—in case it merges two different headwords from its sources into the one lexeme. In other words, I distinguish dictionaries’ print “headwords” from the lemmatiser database’s “lemmata”.

Even with Strictly Ancient Literary Greek, there are word forms that the dictionaries are missing, because editions have changed, or new fragments have turned up, or lexicographers (who often copied each other) had a blind spot. The DGE do use the TLG to compile their list of words, so there’s a good chance they’ll catch the gaps in the TLG corpus at least. But the corpora are fluid, for reasons we’ve already discussed. So πιθανοθηρία “hunting for possibilities” turns up as a variant reading in Plato’s Sophist; because it’s a variant reading, lexicographers have not been panicked to include it. The dictionaries don’t (yet) register anything that deals with ὀκταχοίνικον “eight choinixes heavy” in Aristophanes, or πεντηκοντακισμυρίους “five hundred thousand” in Polybius. (That’s the other blind spot of lexicographers: numbers are boring.)

The count of missing lemmata won’t be massive for antiquity: there are currently just 1,600 word forms unrecognised in the Strictly Ancient corpus, and from inspection, skipping proper names and geometric lines, there’ll be a lot less than 500 lemmata’s worth there. Still, 500 is 500; and it’s thousands, not hundreds, of skipped lemmata moving forwards from Strictly Ancient to Mostly Pagan. Moreover, there are a couple of late ancient texts that, once added, will give us a lot of lemmata: we’re owed 1000 lemmata from the old Latin-Greek dictionaries, a couple of hundred from the Hexapla.

PHI #7 has even more treasures to unlock—although it’ll have to be someone else’s job to do the unlocking. Remember that Vol I of the DGE in the second edition has 3500 lemmata not in LSJ, from α to ἀλλά; extrapolating, that would mean around 80,000 lemmata all up not in LSJ, if and when DGE finishes. How many of those lemmata are already going to be in Bauer, Lampe, and Trapp? My guess is, a fair few; DGE is nowhere near as Pagan-centric as LSJ.

But there are also lemmata specific to the inscriptions and papyri, and not recorded elsewhere. The lemmatiser failed to deal with 40% of the 300,000 word forms unique to PHI #7; and that corpus is not the latest and greatest in papyrology and epigraphy. With 6 word forms per lemma in the Strictly Ancient corpus (which has about the same number of forms), that could mean 20,000 more lemmata unaccounted for. I’m sure that maths is completely wrong, but let’s pretend it’s not: that would mean that, if Kriaras, Trapp, and DGE were completed and added to the TLG lemmatiser, the overall lemma count for PHI #7 and TLG would go from 216,000 to 268,000. Give or take several thousand and more and more satchels of salt.

Keep going? The Triantaphyllidis dictionary of Modern Greek (the contemporary language) has around 47,000 headwords. There’ll be substantial overlap with LSJ, let alone with Kriaras. Maybe 10,000 more? Maybe 15,000? And there’s the dialects of Modern Greek to count too. It’s starting to look like all the lemma of any form of Greek ever spoken anywhere will be around 300,000.

And is that count meaningful? Of course not. Modern Greek is borrowing words from English all the time. (Some of them, it even spells with Greek characters.) And if I did a lemma count for every Italic language spoken over the same time span in the general area of the Apennine Peninsula—from Iron Age South Picene through to modern Bulgnais, via Latin, Standard Italian, and a lot more than six literary dialects in between—then I’d probably get more than 300,000. And of course, any lemma count spanning more than a century is not linguistically kosher anyway. But hey, people ask.

(Bulgnais? The dialect of Bologna. Sounds like Provençal with a head cold. See description in Bulgnais.)

Back to LSJ

You may have noticed that I was only half-heartedly using counts of dictionary headwords in all of this, even though dictionary writers have a bigger corpus than I do, and a more authoritative sense of what should count as a headword. Mapping headwords to lemmata is more complicated than you might think, especially if you’re spanning millenia.

For instance, are ἀνήρ and άντρας the same lemma? We’ve decided they are. But the headwords look completely different, because they are two thousand years apart. And this doesn’t just happen going from Ancient to Modern Greek; it happens from Ancient to Byzantine Greek, and even between Ancient dialects. So there will be less lemmata than headwords. On the flip side, some variants in the printed dictionaries are being treated as separate lemmata, because there’s no consistent indication when they’re variants and when they’re more loosely related forms.

For the record, I’ll note that the 1940 LSJ has 122,000 headwords by my reckoning, and 6,000 of them are cross-references; that leaves 116,000 headwords. The LSJ supplement of 1996 has an additional 10,000 headwords, which makes it 126,000. These headwords span the Mostly Pagan corpus (75,000 lemmata that aren’t names or numbers), plus the inscriptions and papyri (at least 10,000 more lemmata), plus a bunch of technical vocabulary that I’d skipped in excluding Galen and his fellows. If I put Galen and his fellow technical writers back in, but I still leave out the Church Fathers (like LSJ does), and I also add in the papyri and inscriptions from PHI #7 (but not the “Christian Empire” inscriptions), then I get a corpus pretty much like LSJ’s corpus. And my lemma count for that corpus, leaving out numbers and names, is 98,333.

98,333 isn’t 126,000; but then, lemmata aren’t the same as headwords, some LSJ lemmata have disappeared with new editions, we’re missing a few late ancient texts, the lemmatiser is really struggling with understanding inscriptions. In addition, the lexica of Hesychius and Photius alone, which are outside the Mostly Pagan corpus (but document older stages of Greek) account for close to 9000 headwords in LSJ, and well over 1000 for the LSJ Supplement. With them taken out, 98,333 is certainly in the same neighbourhood as 116,000.

I’ll have more to say about headword-to-lemma mapping in a couple of posts, anyway.

For Zeus’ Sake, How many lemmata of Greek?

So. With the evidence available to me right now, with the current status of the TLG corpus and lemmatiser, and my satchels of salt, and OK, OK, enough already…

• How Many Ever? Depending on how long a time window you put, and whether names count or not, anything from 55,000 to 300,000.
• How many lemmata of Ancient Greek? Depends again, but if names don’t count, and we allow Synesius and not St Athanasius—i.e. LSJ’s Pagan-centric definition of Ancient Greek—then 98,000 is a reasonable first guess. Though the way DGE is going, and with new material showing up in the sands of Egypt, add maybe 20,000 lemmata to that.
• How many lemmata of Literary Ancient Greek? Depends for a third time, but if time didn’t stop with Aristotle but Synesius (and names still don’t count), 75,000. If time did stop with Aristotle, 55,000.
• If names count, Ancient Greek goes up to 124,000. Literary Ancient Greek goes up to 99,000. Homer-to-Aristotle Greek goes up to 66,000.

No, that’s not a straight answer. You want a straight answer, don’t do lexicography. And in the next couple of posts, I’ll complicate the answer more again.