Young striker Josh Venner opened the scoring for the hosts, but Mikey Hartnett levelled for Dan Argyle’s side on the half-hour mark.
Sub Jack Pinder put Cayton back in front, but once again the away side drew level, this team Hartnett the creator for Matty Smith to head home at the far post to send the game to penalties.
Both side scored their first three spot-kicks, with Hartnett, Ben Webster and Dan Peel netting for the visitors, then Scalby’s man of the match, keeper Matty Scott saved Jake Sands’ effort but young Cayton keeper Corey Towell saved Mark Oldroyd’s penalty to keep it at 3-3.
Luke Newbold’s spot-kick hit the bar, leaving Smith to smash in the winning penalty to send Scalby into the quarter-finals.
Venner was named as the Cayton man of the match.
In the battle of the big guns, West Pier won 1-0 at home to title rivals Newlands Park in a scrappy encounter.
Billy Logan scored the only goal of the game with an acrobatic finish, hooking a shot over his shoulder with his back to goal to secure the win for Pier.
Manager Andy Spivey admitted the cup-tie was a dour encounter, he said: “Billy’s goal was probably the only clear chance of the first half, we hit the woodwork three times in the second period, and our keeper Steve Sheader tipped a fierce free-kick onto the bar to keep us in front.
“Newlands had the better of the possession but we just about got our way through into the last eight.”
Edgehill came back from a goal down to earn a 5-1 home success against a depleted Sleights side.
Tom Kilpatrick put the away side in front, but a Dean Craig hat-trick, along with goals from Danny Collins and Liam Cooper secured Edgehill’s spot in the final eight.
Right-back Jamie Stewart was the man of the match for the victors, while central midfielder Dean Craig also took his goals very well on his first start of the season.
Seamer Sports also moved into the quarter-finals with a 2-0 win at fellow first division side Westover Wasps.
A Luke Delve free-kick put Paddy Parke’s side in front and they doubled that lead before half-time when a Danny Bradbury cross was forced into his own net by a Wasps defender.
The home side improved after the break but could not break through the determined Seamer defence.
Delve and Bradbury were named as the joint men of the match for Seamer, with Chris Hannam the man of the match for Wasps, while 16-year-olds Tom Sutherland and Connor Bell also impressed for Jamie Heritage’s side.
Whitby RUFC suffered a 3-2 defeat at home to Eskvale & Cleveland League side Lealholm.
The away side opened the scoring but star man Benn Sullivan levelled for Ste Mothersdale’s side with a left-footed curling free-kick, and Mike Kilpatrick’s drilled shot into the bottom corner made it 2-1 at the break.
Two goals after the break Lealholm took them to victory.
Bradley Lewis scored twice and an own goal guided Lealholm to the win.
Kirkbymoorside moved through to the quarters with a 3-2 win at Whitby Fishermen’s Society.
Third division outfit Scalby Reserves gave Scarborough & District FA Junior Cup holders Filey Town Reserves a few nervy moments before finally falling to a 4-2 penalty defeat after the scores were locked at 4-4 after 90 minutes.
Striker Tom Bayes put the hosts in front, and with teenager Callum Randerson pulling the strings in central midfield the villagers were on top for the first 20 minutes.
Town re-jigged their midfield and this soon paid off as they went ahead through a Scott Sheldon double by half-time, with further goals from Lee Cappleman and Tom Haxby 4-1 ahead and seemingly cruising into the next round.
Scalby had other ideas though, with Bayes pulling it back to 4-2, then George Wilson added a third and Bayes completed his hat-trick to take the tie to penalties.
Karl Hansell converted Scalby’s opener, but tame efforts from Nick David and Craig Furnell - the first-ever penalty miss in a long career - were saved by Town keeper Jonny Hunter, and despite Wilson scoring Scalby’s fourth, Haxby kept his cool to send Filey through as they scored all four of their penalties.
Bayes was named as the Scalby man of the match with Randerson also shining, while the whole Town team shared their man-of-the-match award.
A Tyson Stubbings hat-trick fired Goal Sports to a 4-0 home win against Pickering Knights.
Paul Hill also netted for the victors, with Stubbings their man of the match.
Kirkbymoorside Reserves were given a few worrying moments before sneaking to a 2-1 home win against Sherburn.
The Moorsiders opened the scoring after five minutes and doubled their lead just before half-time, but the visitors paid the price for the errant finishing of striker Paul Mills, who missed a hatful of chances.
Mills finally notched in the 85th minute but it was a case of too little too late.
Jordan Wharton was the Sherburn man of the match.
A Jon Cairns double failed to save Edgehill Reserves from a 3-2 home defeat against Malton & Norton Reserves.
The visitors went in front twice, Cairns levelling on both occasions before the York League side clinched victory against a below-par Edgehill side.