The town's former indoor swimming pool opened in January 1973 but it has remained closed and boarded up since June 2017 following the opening of Scarborough Sports Village.
Plans to remove the venue were approved in November last year and finally got underway today.
Preparation works involved the removal of asbestos and disconnecting the building from the mains electricity grid. Demolition works will include removing the building's foundations and pool structure.
In January, council leader Cllr Steve Siddons said that the area will be grassed over and that the authority is in early discussions with interested developers for the site.