Jamie Whitelaw gave the visitors a deserved lead, and although Ian Laing levelled, a John Hodgson strike and a Jake Sands spot-kick put the villagers 3-1 ahead at the interval.
Duchess recovered after the break but they only had a Gary Maw goal to show for their efforts, an 85th-minute penalty miss proving crucial for Paul McBean’s team as Cayton hung on for the win.
Left-winger Sands was man of the match for Cayton, with centre-back Adam Robertshaw also impressing, while Maw was the only player to shine for the hosts.
Whitby RUFC also bowed out of the competition, losing 5-4 on penalties at Beckett League division one champions Kirkbymoorside Reserves after drawing 2-2 in 90 minutes.
RETRO SPOTLIGHT: 15 nostalgic Scarborough photos from the Scarborough News sports archives
Ebberston battle back to sink hosts Filey in SBL Premier Division thriller
Scarborough Athletic slip to 2-0 defeat at Brackley Town in National League North opener
Racers ready to return to action at Steve Henshaw Gold Cup in September
Former Harrogate Town and Bradford City goalkeeper Joe Cracknell returns to Scarborough Athletic
The visitors looked to be coasting to victory when they roared into a two-goal lead after only 10 minutes, winger Mikey Kilpatrick’s cross setting up Sam Richardson for the fifth-minute opener and Kilpatrick then latched onto defender Dave Welham’s precise through-ball before slotting home his side’s second.
The Kirkby keeper made two fine saves to deny the visitors, who also hit the bar and the post before half-time, and the hosts nicked a goal back just before half-time.
Ten minutes from time the Moorsiders levelled the scores, but after this Whitby should have clinched the win as Benn Sullivan’s free-kick hit one post and then trickled along the line before hitting the other, while Paul Jobling and Damon Craig both had late efforts saved by the in-form home keeper.
Both sides converted their opening four penalties in the shoot-out, with Welham, Jobling, Chad Swales and Phil Spencer all netting for Whitby, Kirkby netted their fifth and then Harry Smith had his effort saved by the Kirkby keeper to send them through to the next round.
Edgehill eased to a 14-1 home win against Sleights in the Coastal Shellfish League Cup first round, Danny Collins netting five for the hosts.
Dean Craig also banged in a hat-trick, with the other scorers being Jordan Shaw (2), Dan Jones, Lloyd Henderson, Aaron Jenking and Danny White with the opener on 20 minutes.
Westover Wasps hammered second division visitors FC Rosette 15-4 in their second round clash, man of the match Luke Jones smashing six for the division one side.
Liam Buglass blasted in four goals, Josh Young adding a brace, and Aiden McCallion, Ryan Matson and sub Chris Hannam also notched.
The Rosette scorers were Lee Wardle, Alex Wray, Danny Thompson and Nick Gowan.
Newlands Park also swept into the next round, easing to an 11-0 home win against a Scalby side forced to play 80 minutes with only 10 men.
The second division visitors only started with a bare 11 players, including keeper Tom Manson in an outfield role, and after only 10 minutes they were reduced to 10 men when Danny Argyle went off injured.
Striker Adam Stride smashed in five goals against his old club, while Drew McCoubrey banged in a hat-trick and the impressive Danny Jenkinson scored a brace, with Matthew Griffiths also on target for Shaun Mancrief’s division one side.
As well as midfielder Jenkinson and striker Stride, left-winger Aidan Thomas also shone for the victors, while boss Daniel Gregory was quick to praise all 10 of his players for turning up when others had let him down by failing to show.
Seamer Sports Reserves were forced to work hard for their 7-2 home win against Commercial in the Frank White Trophy first round.
Commercial man of the match Luke Smith put the visitors in front on 15 minutes and a sluggish home side took until the half-hour mark to level, Dave Pallister heading clear out of the Sports defence to Kris Tate, who then teed up Danny Glendinning to fire home.
A fine run from Danny Birley on the right wing on the stroke of half-time saw him pull the ball back for Gareth Sygrove to slot in Seamer’s second.
On the hour mark Robin Coulson levelled for the visitors, but within a minute a Tate penalty had restored Seamer’s lead, and a strike from sub Steve Smith in the 77th minute gave the villagers a two-goal lead for the first time.
Tate’s 30-yard shot in the 83rd minute made it 5-2, and two minutes later Matty Dawson was quickest to react when a free-kick was parried by the impressive Commercial keeper Ryan Somers.
Super-sub Gaz Walters rounded off the scoring two minutes from time, with Tate named as man of the match for the hosts.
Tennyson’s youthful team romped to a 4-0 win at Sherburn, sub Kieran Elwell scoring twice for Nigel Trotter’s team.
Sam Mahoney and man of the match Neil Forsyth were also on target for the visitors, the latter, who missed a penalty, being named as man of the match, with Ryan Wood also impressive in defence yet again.
Assistant manager Billy Jo Johnson said: “The lads all played very well and they are starting to gel. Nigel and I are very proud of how well the young lads are playing.”
Westover Wasps Reserves earned a 2-1 home win against Scalby Reserves.
After a goalless first half, man of the match Dane Ramos opened the scoring for Wasps with a fierce shot into the top corner, and the lead was doubled when Will Smith’s clever finish slid past the Scalby keeper after a goalmouth scramble.
George Wilson pulled a goal back for Scalby but the Wasps had done enough to earn the victory.
Man of the match for Jeff Brader’s team was Tom Bayes.