Is it mathematically possible to create a language where terms describing complex ideas can be made up starting from simpler ideas, with simple logical reasoning in real time, so that knowing vocabulary is not necessary?

I’m sceptical to what extent mathematics enters into any reasoning about human language (and Lojbanists actually highlight that language is not reducible to truth-conditional logic). But much of what you’re saying is the bet behind Natural semantic metalanguage, which tries to define every concept ever in a language that looks like English, but that has only an extremely small number of primitive words.

NSM was a thing of cruel, adamantine beauty back in the 70s and 80s, when it had just 14 primitives. It was also of course utterly unusable as a practical tool for eliciting meaning. It’s now up to 63.

A favourite party trick of Anna Wierzbicka’s undergrads, at least in my day, was to try to hold conversations in NSM. It can be done. It can’t be done efficiently enough to count as a real conversation; but it does meet a generous definition of “in real time”.