The senior civil servant’s findings investigated 16 separate gatherings and said there was a "failure of leadership and judgement" from Number 10 and the Cabinet Office.
Speaking to The Scarborough News, Sir Robert said: "Whether fixed penalty notices need to be handed out to some of the people or bigger penalties for those organising them [is not yet known]. I think we need to ensure that we get this quickly investigated and draw a line under it."
Ms Gray’s report confirms that the Metropolitan Police is investigating 12 of the 16 events for alleged Covid rule-breaking. It includes a gathering at Mr Johnson’s Downing Street flat on November 13 2020.
Post-war asbestos bungalows to be demolished in Scarborough as plans for new terraced homes approved
IN PICTURES: 25 Images from a hot and sunny day at Danby Show
Filey man wanted by police after 'serious' assault and criminal damage incident
Vandalised Scarborough allotments to be restored as council approves plans to help tackle hundreds stuck on waiting lists
COURT: Scarborough brothers threw man around 'like a rag doll' during vicious street attacks
Sir Robert, who has been a staunch supporter of every prime minister under whom he has served, said: "I know there were people in Scarborough breaking the rules and getting visits from the police.
"I also get letters from people who have been scrupulously careful in following the rules; in some cases not able to visit elderly or dying relatives in hospital and care homes, haven’t been able to go to funerals and have had to cancel weddings.
"I do get it that there is real anger out there from people who either did comply with the rules and feel that Number 10 should have also complied, and people who were fined who feel that 'well why are they getting off [the hook] when I was fined?'".
In her report, Sue Gray said: "There were failures of leadership and judgement by different parts of Number 10 and the Cabinet Office at different times.
"Some of the events should not have been allowed to take place. Other events should not have been allowed to develop as they did."
During lockdown, Scarborough borough consistently ranked the highest for Covid fixed penalty notices and illegal indoor gatherings in North Yorkshire with 92 handed out by the end of January 2021 alone. Officers were called to a number of house parties during lockdown, with one man even attempting to hide in a loft.
Sir Robert said there is support for the Prime Minister among MPs, and those involved in the alleged parties should have had a "little bit more self-awareness about what was within the rules and what wasn't".
He added: "It's not in the interests of the country for this to drag on. We need to find out exactly what has happened; if laws were broken then penalties should be applied and there should be consequences.
"Given what's happening in Ukraine and with energy prices and the cost of living, I think we can do without this distraction rumbling on and on."