Would the Byzantines have spoken Ancient Greek or something closer to modern Greek?

Modern Greek.

Being literate in Greek has always meant being literate in Ancient Greek; so all our evidence of the vernacular is tainted, right up until the Cretan Renaissance (and there it’s tainted in a different direction, of conventionalised dialect). In the period between the Arab conquest of Egypt (when the papyri run out) and the first experiments with vernacular poetry in the 12th century, we have almost no direct evidence at all, outside of Bulgar inscriptions presumably written on their behalf by Greek prisoners of war.

But what we do know and can reconstruct tells us that the spoken language looked close to Modern Greek by the 7th century, and the texts we have in the 12th century, though macaronic, are identifiably macaronic with Modern Greek.

There would have been registers of spoken language as with every language. We have a hint from Filelfo, writing in the 15th century, that the language of the court in Mistras and Constantinople was “purer” than everywhere else. That suggests a proto-Puristic Greek, with more influence from written, atticist Greek than in the countryside.