Now, sources close to the show say it's "largely because producers were concerned her thick Newcastle English accent would be too difficult for an American audience to understand," according to TMZ.
Cole's Northern English accent -- commonly referred to as a Geordie accent -- can be extremely difficult to understand to the untrained ear. However, her accent endeared her to UK audiences when she hosted the show there, so it's likely American audiences would have grown accustomed to it over time.
But the British tabloid The Sun reports that Cole's exit is due to a much different reason.
"It hasn't worked out. She's homesick and wants to return to the UK. She's had enough," a source told the gossip mag.
"Whether she ends up back on UK X-Factor is open to speculation."
Others speculate that Cole lacked chemistry with Paula Abdul -- the kiss of death when it comes to reality singing competitions. One source went as far as to say that Cole was the "expendable" one out of the group. Ouch!
America's Got Talent judge Sharon Osbourne called this one back in December when she said that rumored Chezza replacement Nicole Scherzinger was the better choice for The X-Factor.
"Cheryl's perfect for England -- she's the nation's sweetheart and they absolutely love her," Osbourne said. "But over here it's Nicole's town. You have to think of the audience and most of them would never have heard of Girls Aloud."
Osbourne was blasted at the time, but she's right -- fellow X-Factor judge L.A. Reid said he didn't even know who she was, and he's a big-time music exec.
At least one person's upset about her departure: Japan troublemaker Russell Brand.
"Our Cheryl, kicked off X Factor? What, the American one? And I've come to this country in good faith and they've kicked Cheryl off, our Cheryl? Oh no! She's Britain's sweetheart," Brand wrote on Twitter.
And you'll see personalized content just for you whenever you click the My Feed .
SheKnows is making some changes!