I was recently in a World of Warcraft instance with a party and one of the members would use something that would post a window with the question ARE YOU READY? and a yes and no button on it.
Does anyone know what the name of this mod is?

Hello,

That was a command part of CT_Raid addon back in the days but i think Blizzard took the idea and it's a built-in Blizzard UI command now, should be something like /raready when you are in a raid group, not sure tho.

Note that only Raid Leader can do that.

It is a build-in feature of the raid management UI now, yes :)
The raid leader can click a button to initiate a "ready check" (or use commands, I don't know) to make sure everyone are ready to go before they pull the boss