Simon Cowell was worried Gary Barlow would set up an "entertainment empire" to rival his own, according to a new biography of the X Factor supremo.
He also fell out with some of the show's stars, including Matt Cardle and Will Young, after a series of rows.
Extracts from Tom Bower's book Sweet Revenge: The Intimate Life Of Simon Cowell said he rowed with Pop Idol winner Will and branded him "a fake, snivelling, little drama queen" after the pair fell out when Simon mocked his lisp in a television interview.
It also quotes Simon as saying Gary "chased that madly to get three initials after his name" after the Take That frontman was made an OBE.
Bower said: "Ever since Barlow has clawed his way back from a disastrous attempt to become a solo star, Cowell was niggled by his conviction that Barlow was bidding to establish a rival entertainment empire and, like Paul McCartney, secure a knighthood."
The extracts published in The Sun said Simon offered Dannii Minogue £2 million to return to the hit talent search show but the deal fell apart when she demanded £5 million.
It claims he told ITV boss Peter Fincham to "Bring her back to X Factor... everyone will be looking at her t**s".