If I were the Raptors, I'd rather have Casey and the roster minus Derozan and Lowry -- like literally, trade them away for nothing -- than fire Casey and keep the core stars.
The core guys are the problem with that team. There's no reason to think Casey had lost the team or anything like that. A new coach won't give them a higher ceiling.
If Derozan and Lowry are to blame, then maybe the GM should be fired first. After all, he was the one who gave them those awful contracts, and built that weak minded team.
They are good but not great players, but how exactly is Toronto going to get a great player? They basically have to tank, find some way to trade for one, or get lucky later in the draft. I mean Toronto, as is, has a very difficult time of even keeping their stars. In fact every player they've had that has been better than DeRozan has left in free agency (McGrady, Carter, Bosh). It is a tough job.
That said, I think they should trade DeRozan, JV, and/or Lowry and see if they can find a better mix of talent or just start over completely. The Sixers and Cavs both have lottery picks this year. Both would make some sense for DeRozan or Lowry (that assumes James stays of course). Clarkson, Smith, Osman/Nance, BKN for DeRozan as an example. Or after the new season flips (though it can be worked out before draft so it is Toronto's pick) - Lowry for Bayless, McConnell/Holmes/Anderson, LAL (or something like that). The Raptors could do both trades and end up with 8 and 10 (Bamba and Sexton on nbadraft.net), plus some other younger players to really start their rebuild and could likely get other 1st round picks thrown in (maybe the Cleveland and Philly 1st's from this year or future such picks). They then try to find trading partners for JV, Smith, etc. to really rebuild.