Is there an upper bound to the amount of words a language will realistically contain?

If a language is agglutinative, or has a halfway decent derivational morphology, you can keep making up words based on other words for as long as you like, and those words will be perfectly acceptable. So there is not much of a limit.

There is a limit in how many building blocks of words (morphemes) someone can retain, and those morphemes will correspond to the vocabulary of someone speaking a purely isolating language. (Spoken Chinese isn’t as pure about this as it likes to think it is. Classical Chinese is, but classical Chinese is clearly heavily stylised.)

So, to turn this question into a question somewhat more clearly related to the limitations of human linguistic processing: how many characters can a Chinese speaker retain? Or, how big is the vocabulary of the average English speaker? (which is somewhat close to this, though English derivational morphology is still productive).

The answer for an individual is in the order of magnitude of 10,000. For a contemporary language with a wide range of specialist vocabularies, you are ranging across the vocabulary of all members of the speech community. That means you add one order of magnitude to the size of the available stock of morphemes; you don’t add two.