What language first introduced punctuation such as the period, comma, exclamation point, and question mark?

See Punctuation on Wikipedia. David Crystal has a lovely book out on the history of punctuation: Making a Point.

As Adam Mathias Bittlingmayer indicated, there were anticipations of punctuation for a while; the notion of systematically indicating pauses (period, comma) was a Hellenistic Greek invention, which became systematic in the late Empire.

Punctuation as we know it is a mediaeval Latin thing, and it kept evolving up until after the invention of printing; parentheses for example are a 16th century thing. The question mark is 8th century; quote marks as we know them (as opposed to quote marks the way email does them) is 12th century.

But the main written language of Western Europe was Latin up until at least the 16th century, and that’s the language for which most punctuation we are familiar with was introduced