Simon Cowell is known as the man with the golden touch in the pop industry but despite securing a third season of the X Factor US, his show is rapidly losing viewers.

The latest series saw a radical reshuffle that included hiring pop singers Britney Spears and Demi Lovato for the judging panel and tapping celebrity personalities Mario Lopez and Khloe Kardashian to co-host the show’s live episodes.

However the Fox reality show saw another substantial ratings drop for its season finale last week according to industry reports.

The man with the golden touch: Simon Cowell's X Factor is struggling in both the US and the UKThe man with the golden touch: Simon Cowell’s X Factor is struggling in both the US and the UK

While rival show The Voice’s finale hit a season high and ranked as the show’s biggest closer yet, the X-Factor finale was below its best numbers this season and fell notably from last year.

The two-hour finale in which Tate Stevens was crowned X Factor champion was loaded with guest stars but only pulled in 9.6 million viewers, down 18 percent from its first season finale in which Melanie Amaro won the $5 million record deal.

Time for a rethink: The X Factor is losing out to rival show The VoiceTime for a rethink: The X Factor is losing out to rival show The Voice


And things don’t look good for 37-year-old country singer Stevens either, as one year on from Melanie’s victory she is struggling to make an impact in the Billboard charts.

The father of two beat teenager Carly Rose Sonenclar into second place to scoop the top prize of a $5m record contract last Thursday night.

But reviews of the second season were not favourable with reporting: ‘The song-and-dance productions were as clunky and forgettable as ever, Britney’s faces got real old real fast, and Mario and Khlomeister still can’t even figure out which camera to gape at.’

‘Above all, the show is just devoid of any real character … Simon Cowell is the worst of them all – barely awake, looking insulted to even be spending his time like this.’

‘It’s his show, and he’s embarrassed, and he doesn’t care enough to inject any life into it … it’s like an unintentional parody of a reality show with zero sense of humor about itself.’

It was also announced that judge LA Reid, chairman of Epic Records, the Sony-owned label which will sign the winners of X Factor USA, is leaving the show.

And things aren’t going any better on the X Factor UK.

Simon is ‘frustrated’ and ‘worried’ about the future of the show, according to his ex-girlfriend Sinitta.

The 80s pop star, who is close friends with the music mogul, said he is even considering a possible return to the show, which has suffered a fall in viewing figures this series.

Sinitta told the People: ‘This has been the worst series ever and Simon wants to come back.’

Since quitting the ITV show following the 2010 series to start up X Factor USA, Simon had found it difficult to manage the British version from Los Angeles.

Sinitta said: ‘Simon says X Factor is his first baby. He worries about the show and there is a lot for him to worry about at the moment.

‘The wrong people are going. He finds it frustrating. He doesn’t want the whole thing to fall apart.’

‘It is also no secret that he has missed working on the UK show.’