Did the ancient Greeks use a different language for a special purpose like it was the case with Latin in Europe and Sanskrit in India?

Did the Ancient Greeks have a different *language* for sacred purposes? No, Ancient Greek was their language.

But the Ancient Greeks did use different dialects for different genres of literature, to an extent that has not been paralleled since. Epic dialect (a mix based on archaic Ionic) for epic poetry, and allusions to it, is the closest they had to a Latin or Sanskrit, given the immense prestige of Homer. Doric for choral poetry, Aeolic for lyric poetry, Ionic for history and medicine (following Herodotus and Hippocrates), Attic for default prose.

The Mediaeval and Modern Greeks are a better comparison. There was a little Epic, Ionic and Doric written now and then, but the main distinction was between learnèd registers and the vernacular, with the vernacular avoided thoroughly until the 12th century, and by most writers right until the 20th century. The learnèd registers were varying mixes of Attic, Koine, and calqued French. (Those who’ve read Psichari know what I mean by the last bit.)

And to this day, the Greek Orthodox (and those of that heritage) are very uncomfortable with the vernacular being used in a Christian religious context. I did a spit-take walking past a Greek Catholic church, and hearing the mass in Modern Greek. Catholics in Greece are post-Vatican II, after all. Unthinkable in Orthodox  services. The most vernacular language used in the services is probably the Gospels…