Members of the borough council’s planning committee yesterday approved the demolition works which they said would remove an "eyesore" from the town’s seafront.
However, concerns were raised that it could worsen crime in the area with one councillor saying the grassed open space would create a "racetrack" if left unfenced.
Councillor Phil Kershaw said: "This area has been causing lots of antisocial behaviour problems already – we have got youths racing cars around the car park and on the grassed area in front of the swimming pool.
"By recommending that the area be grassed over, it will just make an even bigger racetrack."
Councillor John Casey also said: "This is going to be a massive, wide open piece of land which will attract travellers. It is also going to attract racers and other antisocial behaviour problems."
Council officers told yesterday’s meeting that they would explore the possibility of fencing-off the area once the swimming pool is demolished.
The 48-year-old venue on Ryndle Crescent has stood empty for four years following the opening of Scarborough Sports Village, which is now home to a new pool, gym and Scarborough Athletic Football Club.
The approved works will involve the removal of asbestos from the building, as well as the demolition of foundations and the pool structure.
The plans are part of a wider regeneration scheme for Scarborough’s North Bay, known as the Sands development.
This involves hoardings to be placed around the former Atlantis water park which had been earmarked to become a multiplex cinema before the plans collapsed earlier this year.
Toilets at the Northstead Upper car park will also be refurbished as part of the scheme.