Alex Lees’s men slipped to second-bottom in the North Group after a result that ended a run of three successive wins.
Yorkshire host Durham on Wednesday and Northants on Friday before the return fixture at Northants on Friday week.
They have little or no margin for error.
After Lees won the toss and chose to bat, Travis Head top-scored with 40 on debut and Kane Williamson and Adam Lyth both hit 39.
RETRO SPOTLIGHT: 15 nostalgic Scarborough photos from the Scarborough News sports archives
Scarborough Athletic slip to 2-0 defeat at Brackley Town in National League North opener
Ebberston battle back to sink hosts Filey in SBL Premier Division thriller
Commonwealth Games 2022: Scarborough's Chris Turnbull and Team England win bowls mixed para pairs bronze medal
Jack Carradice-Clarkson takes 6-11 as Flamborough sink Wold Newton
Yorkshire’s 160-7 was competitive but not wholly conclusive, and it left the door open for Dan Christian’s side.
Azeem Rafiq and Adil Rashid conceded just 44 from a combined eight overs of spin, and Yorkshire seemed on top when Notts fell to 102-6 in the 14th over.
But 58 from Samit Patel and an unbeaten 35 from Chris Read guided the hosts, Read square-driving the winning boundary off Liam Plunkett with two balls to spare after three runs were wanted from the final over.
Before a crowd of 12,595, Yorkshire – missing the ill David Willey – started strongly when Adam Lyth pulled the first ball of the match from Patel to the mid-wicket boundary and slog-swept the fifth delivery for six.
Yorkshire lost Lees to the penultimate ball of the second over, bowled by a full one from Harry Gurney, which brought Williamson to the crease for his final innings before international duty.
Williamson, who sat out last week’s County Championship game against Surrey at the Oval due to fatigue, got going with a four through mid-wicket off Luke Fletcher.
Lyth, fresh from a double hundred at The Oval, simply carried on from where he left off in London.
The left-hander hammered four boundaries in Christian’s opening over, which disappeared for 23 and included five wides.
Lyth had scored 39 from 20 when he sliced Patel to deep cover, Imran Tahir taking the catch to leave the visitors 55-2 in the seventh.
Tahir’s first over was almost as expensive as Christian’s; 19 came off it, including an enormous six by Head over mid-wicket into the Fox Road Stand.
Yorkshire were 90-2 at halfway and Head kicked off the second half of the innings with a straight six off Patel.
The left-hander then pulled Steven Mullaney into the Fox Road Stand and clubbed him straight down the ground for a fourth maximum, moments after he was dropped by Patel at long-on off the same bowler.
Head rode his luck at times, with one or two shots landing just out of fielders’ reach, but played well too.
His luck finally ran out when he lofted Gurney to long-on, where Riki Wessels judged a skier to perfection.
Yorkshire fell to 138-4 in the 16th when Williamson paddled Mullaney to short fine-leg, and to 146-5 in the 17th when Plunkett was stumped.
Tim Bresnan chipped in with a run-a-ball 17 before he was yorked by a slower ball from Gurney, who returned 3-16.
Will Rhodes was run-out off the last ball attempting a second to deep point as Yorkshire managed only 21 off the last four overs and just two boundaries in the final eight overs, which ultimately cost them.
Michael Lumb, the former Yorkshire batsman, hit the first ball of the reply for six over cover off Bresnan.
But he was then bowled around his legs by Rafiq, and Notts fell to 25-2 in the fourth when Wessels helped Steve Patterson’s first delivery to backward square-leg.
Patterson struck again in his next over when Christian picked out deep square-leg, but Greg Smith and Patel added 51 from 31.
Smith made 32 before slicing an attempted leg-side shot to short third-man, where Rashid took a fine tumbling catch, Rashid then trapping Brendan Taylor to leave Notts 100-5 in the 13th.
Patel turned Lyth’s first ball to short fine-leg, where a direct hit from Patterson did for Mullaney at the non-striker’s end.
Patel picked out mid-off in the final over, but it was too little too late for the visiting team, with Lees left blaming a no-ball that was given earlier in the innings when he was deemed to have been an extra fielder outside the circle, resulting in a free-hit and an extra ball.
Lees described the decision as “terrible” and said the umpire was not “doing his job properly”.