Manager Jono Greening rested Ryan Watson, Bailey Gooda, Will Thornton and Simon Heslop, with James Cadman, Dylan Cogill and Nick Hutton all handed starts, along with Josh Barrett, following his lengthy absence through injury, writes Steve Adamson.
Whitby made their customary bright start, with Brad Fewster and Jacob Hazel having shots blocked, and Cogill doing well to clear a right-wing cross from Soni Fergus.
Ash Jackson fired over in Boro's first attack, then Cam Wilson drove a free-kick narrowly wide.
RETRO SPOTLIGHT: 15 nostalgic Scarborough photos from the Scarborough News sports archives
Ebberston battle back to sink hosts Filey in SBL Premier Division thriller
Scarborough Athletic slip to 2-0 defeat at Brackley Town in National League North opener
Commonwealth Games 2022: Scarborough's Chris Turnbull and Team England win bowls mixed para pairs bronze medal
Jack Carradice-Clarkson takes 6-11 as Flamborough sink Wold Newton
Former Scarborough striker Hazel blazed a shot over the Boro bar, a 20-yard strike from Fewster flashed inches over, and Harry Green shot narrowly wide after a lay-off from Hazel.
Winger Josh MacDonald burst into the area, but was foiled by a crunching tackle from Jackson, then Hutton shot wide for Boro
Shortly before the interval Michael Coulson was replaced by Kieran Glynn, who came on for his 50th competitive appearance.
Whitby had been the better side in the opening period but the score remained blank, and Boro upped a gear after the break, a Luca Colville shot was deflected towards Wilson, who fired wide and Cadman had a shot saved by keeper Shane Bland.
Jackson was playing a blinder, and he tackled Hazel as he raced onto a through- ball, then Ryan Whitley saved at the feet of Green in a crowded goalmouth.
A Wilson corner was headed just over by Barrett, then livewire Jackson had a shot blocked, and from the rebound he struck another shot that brought a diving save from keeper Bland. Whitby had a couple of chances, as Marcus Giles shot straight at Whitley and a Jake Hackett corner was headed over by Dan Rowe.
Scarborough Scholarship striker Joe Wood came on for his debut, as Boro, with Glynn and Colville turning on the style, began to get on top.
A Jackson cross to the far post was headed at the keeper by Day, and Colville fired just wide from a free-kick, before Glynn went on a surging run through the middle of the park, skipping past two defenders before passing to Jake Day, who slotted past the keeper from close range to snatch an 87th minute winner.
Hazel fired straight at Whitley late on, as Boro booked their place in the last four and a trip to Redcar Athletic.
This was not a great game, but very competitive, and there were a number of good performances.
Jackson, Colville and Glynn were superb for Boro and Kieran Weledji and Cadman both did well despite lack of game time in recent weeks.
Whitby, who chased and closed down all game, will be disappointed to lose, with Jon Burn, Dan Rowe, Harry Green and Brad Fewster catching the eye.
BORO: Whitley, Barrett (Wood 72) Jackson, Cadman, Weledji, Cogill, Hutton (Plant 76), Colville, Day, Coulson (Glynn 43), Wilson.
MAN OF MATCH: Ash Jackson.