Monday's poll, which was open to residents of Whitby parish only, saw 2,111 say yes to the question: Should all new build and additional housing in Whitby parish be restricted to full time local occupation as a primary residence only and forever (in perpetuity)?
Only 157 people said no.
The outcome is not legally binding but moves are afoot to address the problem.
Town Mayor, Cllr Linda Wild, said: “This demonstrates the importance of this issue for local people and the depth of feeling of being crowded out of affordable housing by an overabundance of second homes and holiday lets which can be in the hands of unscrupulous landlords.
“The power to tackle this lies with the borough council – which is making attempts to confront the problem – and with Government, which needs to amend the planning regulations to make it possible to protect local housing for primary residence.
“We need a ‘use class’ which applies to holiday lets.
“Then the planners can manage that change of use.
"We also need to tax second homes and holiday homes more effectively through council tax and business rates to reflect the impact they have on local people.
"I hope the town council will take up this fight and invite our neighbouring parishes to join in.
"For the holidaymakers, Whitby is bigger than the parish boundary.
“I hope that people in Sandsend, Sleights, Sneaton and Stainsacre feel this is a campaign for them too.
“Whitby is not unique in this predicament and local people want their voice heard by government alongside people from Cornwall, North Norfolk, Northumberland and the Lakes.
"We need Government to give local people the power to keep holiday resort communities sustainable.”
Cllr Phil Trumper, who represents Whitby on both the borough and county council, said Whitby Town Council must now start the process of a Neighbourhood Plan, which gives local communities a greater say in what goes on.
He added: “Unfortunately the poll holds no legal obligation. To be legally binding it would have to be part of a Neighbourhood Plan.
“However, in January this year we introduced a motion at the Scarborough Borough Council full council meeting and it was unanimously passed to make it part of the new local plan process.
"This should go out to public consultation this September.
“The opinion has highlighted the issue further but we need to get on and do it through the proper process.”
A second question in the poll asked: Do you want to stay in the same area committee as Scarborough in the new North Yorkshire unitary council?
There were 253 yes votes while 1,982 said no.
The turnout of Whitby parish voters was 22.72%.