What is the origin of the scientific name of the apple tree “malus”?

This has been answered already, I’ll just answer it more anecdotally.

Indo-European has two words for apple, that show up in different daughter branches:

• *h₂ébōl shows up in Germanic (… apple), Celtic, Balto-Slavic, and probably Hittite šam(a)lu- ‘apple tree’
• *méh₂lom shows up in Greek (Doric mālon, Attic mēlon), Latin (mālum), Albanian (mollë), and Hittite maḫla ‘apple’

This has been a puzzle for Indo-Europeanists.

• Some Indo-Europeanists have assumed the genuine Indo-European word was *h₂ébōl, and *méh₂lom was a pre-Greek loanword.
• Some Indo-Europeanists have assumed that Indo-European had split up into northern and southern dialects, and dialects are allowed to have different words for the same thing—without one word being necessarily more Indo-European than the other.
• This was news to me: Proto-Indo-European phonology – Wikipedia says that some Indo-Europeanists have tried to unify the two forms as *h₂eml-:
• *h₂eml- > *h₂ebl- > *h₂ébōl
• *h₂eml- > *meh₂l- > *méh₂lom
• EDIT: And add the speculation by Guus Kroonen in On the origin of Greek μῆλον, Latin mālum, Albanian mollë and Hittite šam(a)lu- ‘apple’ that *méh₂lom, which he reconstructs as *smh₂l, is related to proto-Kartvelian (as in Georgian) *msxal- ‘pear’.

So much for apple. What’s the story with mălus ‘evil’?

As others have pointed out, the vowel in malus is short; so whatever it’s derived from, it’s not going to be derived from *méh₂lom (where the laryngeal h₂ serves to lengthen the preceding vowel).

malus – Wiktionary

From Proto-Italic, related to Oscan mallom and mallud (“bad”). Originally associated with Ancient Greek μέλας (mélas, “black, dark”), but support for this is waning. Perhaps from the same Proto-Indo-European root as Avestan [math]unicode{x10B28}unicode{x10B00}unicode{x10B0C}unicode{x10B2D}unicode{x10B0C}unicode{x10B0C}unicode{x10B00}[/math] (mairiia, “treacherous”).

Which means… we don’t know. All we do know is, it is indeed a coincidence. Although yes, it’s a coincidence mediaeval theologians have had a field day with. In fact, it’s likely the reason why Westerners assume the forbidden fruit was an apple: Forbidden fruit. De ligno autem scientiae boni et mali “of the tree of the knowledge of good and evil/apples”. Jewish tradition instead pointed to the fig, the grape, or wheat.