Property prices have jumped 17% in the town over the past year, says Rightmove, giving an average asking price now of £254,218.
Only Padstow was higher, with a 20% increase.
But Gazette readers are becoming increasingly fed up with being priced out of buying property locally.
Peter Croft of Upgang Lane wrote to us to urge local authorities to “grasp the nettle” and stop conversion of family homes into holiday lets, before Whitby ends up as a “holiday theme park with no community”.
In another letter to the Gazette, Asa Jones of The Garth said young people are leaving Whitby in their droves, unable to afford property prices on low wages, urging the town to build a community in which it is sustainable for local people to work and thrive.
Whitby borough councillor Phil Trumper recently called for a primary residence scheme for the town, making it easier for local people to buy local homes.
We asked our readers what they thought of the current housing dilemma.
Sarah Garrod said: "We need to concentrate on our community and its needs, ie people who live here need support, not holiday homes taking over.
"We, the community, have a voice and we should be heard and listened to."
Jessica Pound outlined one of the biggest problems for people looking to get on the first rung of the ladder.
"The people buying properties for holiday homes are also outbidding us first-time buyers massively when they won’t even be living in it - extremely frustrating when it’s so far above the asking price you don’t have a chance!" she said.
Steve Akeroyd, who was born and bred in Whitby, says he doesn't recognise the town any more.
"Locals cannot afford to live in their own town and are slowly getting pushed out.
"It's such a shame, it's gone from a small fishing town into an unrecognisable holiday circus."
Cllr Trumper also recently said that he'd spoken to the leader of Cornwall Council about St Ives, another tourism hotspot, where a vote in favour of a Neighbourhood Plan means that new-build housing projects will only be given planning permission if they are reserved for people who live in St Ives and surrounding areas full-time.
Anthony Noble agreed that would be a good idea for Whitby.
"It should have been stopped years ago like in Cornwall, they put a stop to second homes and holiday lets.
"It's criminal what has gone on in this town."
Graeme Shepherd said that the problem seemed to have got worse in the last few years.
"Us locals have watched our town and community slowly die," he said.
"We have been saying the same thing for years. It's like we are being ignored by those who are in power."