Probably between Frieza and Buu. Frieza was the one that pretty much destroyed the Saiyan race and anything else before Goku fought him. And Buu, well, He was just a monster. Everyone fought him and couldn't really destroy him.
If Broly had a larger part in the series. He would really be my top pick. But since he didn't he was lowered. Imagine a Broly saga. That would be quite awesome.