The controversial comedian, real name Royston Vasey, is due to perform at the venue on Saturday October 2 but protesters say his jokes are "incredibly racist, misogynistic, and homophobic".
They are calling for the council, which owns and runs the Whitby Pavilion, to cancel the performance and are planning to stage a protest outside the venue on the night of the gig.
Caitlin Webb-Ellis, 33, who organised the protest, said since organising the event she has received threats, abuse and misogynistic and homophobic comments.
"I don’t feel safe in my hometown at the moment, it’s a really sad state of affairs," she said. "I am also, sadly, not really surprised and it just reveals the true extent of the problem.
"We’re very worried about the threat of violence and I think what we have already achieved with this movement is massive. We’ve managed to get people talking about this and to get people to see that in a small town like Whitby, we are having these conversations."
Mr Vasey's performance at Whitby Pavilion was booked by the venue's previous owner, Sheffield International Venues, but was postponed last year due to the coronavirus pandemic.
In a statement, Scarborough Borough Council said: "As the new operator we don't make a judgement about an act which isn't law-breaking but we do recognise that Mr 'Chubby' Brown has a form of humour that some people will find objectionable and highly offensive.
"Other venues may take a different view, but there are no plans to cancel the performance in Whitby. It is a personal choice whether someone wishes to pay to see him on stage."
Ms Webb-Ellis said it was "cowardly" for Scarborough Borough Council to "hide" behind the defence that the booking for Mr Vasey was made by the venue's previous operator, which has since rebranded to Sheffield City Trust.
In September, the Trust decided to cancel a Roy "Chubby" Brown show at Sheffield City Hall as it "didn't reflect its values" and concerns about racism, homophobia and sexism.
A petition calling to reverse that decision has since reached 42,000 signatures.
Addressing the protesters in a statement on Facebook, Mr Vasey said: "The protesters have the right to freedom of speech and choice at their protest, as we all should have, whether we agree or not, so let’s respect that and not become like them.
"I find it more than ironic though that that is all we are asking them to do, to allow people to attend the Chubby Show behind closed doors at all venues across the country.
"Why can’t they respect that instead of trying to dictate and force the minority opinion on the majority!"
In his statement, Mr Vasey asked the show's audience to ignore the protesters.
A Scarborough Stand Up To Racism spokesperson said: "How could we not protest at the kind of 'humour' that normalises racism and contributes to a culture of white supremacy in a society already suffering from institutionalised racism?
"His are not just words – they have the consequence of emboldening people who share his views, sometimes to the point of violence."
Supporters of the "Chubby" Brown performance on Facebook said: "There's Chubby showing more tolerance than those who dislike him. It tells you something, doesn't it."
Another said: "He is actually a really nice guy, it's purely an act he does. People need to get over themselves. He isn't forcing anyone to watch him."