Lord Herman Ouseley is to stand down as chairman of anti-discrimination group Kick It Out at the end of the season.
The organisation was set up 25 years ago by the 73-year-old to tackle racism and discrimination in football.
Lord Ouseley, who was formerly chair of the Commission for Racial Equality, said on Kick It Out's 25th anniversary in August that he was preparing to retire, albeit while conceding that he had made such comments before and ended up staying in place.
In a statement on Tuesday, he said: "When I set up the Let's Kick Racism Out of Football in 1993, I hoped that the football authorities would take it on board and tackle the problem.
"It proved to be a long haul with a slow process of change in pursuit of equality, inclusion and cohesion.
"Recent progress across the game in pursuing equalities for all is offset by the wider society afflictions of prejudice and hatred, from which football cannot escape.
"Over the past decade, I have indicated my wish each year to step down from the Chair and move on and hand over the reins to younger equality advocates and campaigners, but have always been told that 'not now, this is the wrong time', and no one else has stuck their head above the parapet to lead the organisation.
"Well, now is the right time for new leadership of Kick It Out, as I re-focus my life for new challenges after 56 years of public service."
The news that Lord Ouseley is planning to quit comes just a few days after Manchester City forward Raheem Sterling was allegedly racially abused during a loss at Chelsea - the fan involved denies using racist language.
Ouseley wants football's leaders to do more to combat racism and inequality.
His statement added: "I have thrown challenges at the leaders in football and they need to show their hands - they need to show they are capable of bringing about lasting change, if they all come together out of their silos and take a dynamic and coherent approach to meeting the equality challenges.
"Kick It Out is well positioned with excellent staff to keep the momentum of activity afloat in helping football to be a powerhouse for equality, inclusion and diversity.
"It is also well positioned to continue its excellent work in helping to prepare the next generation of players for the diversity of cultures in the game and to handle all inequalities, discrimination and prejudice.
"There is so much to be done before I sign off next year, so onwards with the action while I prepare to hang up my boots."