The Blues' chairman took over the role in 1994 from Bob Scaife Snr, and steps down as the most successful chairman in the club's history having won nine trophies in his tenure, most notably the FA Vase at Wembley, in 1997.
The decision comes on the back of "unforeseen personal issue", but Mr Manser will be on hand to advise his successor, should he be needed.
“It became apparent to me that the club needed a new vision, new ideas and new vitality. I decided therefore that it was time for me to hang up my boots and stand down as chairman," he said.
“I totally support all the new ideas that are being suggested and I must stress that I am not walking away from the club, I am just taking a back seat and I will be more than happy to assist the new Chairman when requested.
“Due to an unforeseen personal issue, I am standing down with immediate effect and Eddie McNamee will take over as acting chairman.
“I would like to thank everyone who has helped the club in my time as chairman. Without this assistance the club could not have functioned.”
The club has called a member of its full members to discuss how it goes forward. In the meantime, Eddie McNamee, a former player at the club and a key fundraiser, boosting the club's coffers by £60,000 in the last six years, will take over as club chairman.
Commenting on taking the role, he said: “I am honoured and happy to take on this role for Whitby Town FC and feel somewhat humbled to have been asked.
“Whitby Town FC is about the supporters, without support there is no club. In the near future we plan to liaise more closely with the supporters, keeping them abreast of developments.
“As a club we hope that this will lead to more involvement of the supporters in the actual running of the club. We will also be looking for volunteers to take an active role in the club’s future.
“For me, the main aim is to maintain and improve Whitby Town FC, involving supporters and the wider community, including schools and the local youth, and ultimately to support Chris Hardy in his quest for success on the field.
“The community is very important to me and the football club. We have already started a process of connecting with local schools and encouraging youngsters to attend matches, and we will work on developing this further the future.”