The Premier League has vowed to continue cracking down on discriminatory language used at games.
It follows an announcement of a major new partnership with Stonewall, in a bid to tackle homophobic abuse at matches.
"There are many million football fans in the country," Bill Bush, Premier League executive director, told Sky Sports News HQ. "There will be a minority of people who are bigoted. Our view is that if they can't keep themselves quiet, then we will remove them from the ground. We don't want them polluting our grounds with bigotry.
"We'd much rather say it's about education, it's about embracing difference whatever difference is."
"We know most fans want LGBT people to feel included," added Ruth Hunt, Stonewall's chief executive. "But the barriers to this, such as homophobic chants, too often go unquestioned."
Around 150 people attended the charity's inaugural Rainbow Laces Summit, hosted by Manchester United, at Old Trafford on Monday, including FA chairman Greg Clarke, Football League chief executive Shaun Harvey and PFA chief executive Gordon Taylor.
The TeamPride coalition is a group of major global organisations, including Sky Sports, who support Stonewall's campaign to promote equality for lesbian, gay, bi and trans (LGBT) fans and players in sport.
The FA chairman believes the men's game is "probably a couple of decades" away from feeling as inclusive as the women's, and he has spoken to LGBT athletes about their experiences and the support they need.
Clarke says gay footballers have been "reticent" to talk to him.
"We have to redouble our efforts on inclusion," he said, during a discussion on an inclusive culture. "I know twice as much on LGBT issues as I knew six months ago but only 10% of what I need to know.
"Something about the men's (football) game is not right."
EFL boss Harvey, who presented Hunt with a commemorative logo at the event, said: "The EFL strives to offer a football experience for everyone and it is vital that we work collaboratively with the football family and relevant partners to make our game inclusive for all.
"We are delighted to reaffirm our commitment to Stonewall's Rainbow Laces campaign which complements the ongoing work our clubs are undertaking as part of the EFL Equality Code of Practice."
"We're here because we know football can make a huge difference across the world," said Chris Paouros, co-chair of Tottenham's Proud Lilywhites. "Saying fans are a problem lets everyone else off the hook."
"It's the most incredible thing to see a youngster come in and find their difference accepted and see them bloom," said Kate Richardson-Walsh, the most capped female hockey player in her country's history.
"Being yourself is the best thing you can do, whatever yourself is," added Helen Richardson-Walsh, the youngest ever women to represent GB hockey at an Olympics and the other half of the first same-sex married couple to win an Olympic medal playing in the same team.
"To hear more sponsors say it's fine to be who you are, that would go a long way," said Danny Watts, the former professional racing driver. "It makes a huge difference to see the Pride flag in football games on the telly."
"I'm competing against people from all over the world and some of those countries have horrific records on LGBT rights," said British track cyclist Callum Skinner, whose father is gay. "It's great for that to be out in the open and he's been far more involved in my sporting career ever since (he spoke openly about his father's sexuality)."
The Rainbow Laces Summit also featured rugby league's Keegan Hirst, Wakefield Trinity, Lisa Wainwright, British Basketball chief executive, and Sophie Cook, AFC Bournemouth's club photographer.
This week, Sky Sports will broadcast a short film depicting the story of a young professional footballer and his struggle with homophobia issues in sport.
WONDERKID airs after Southampton v Manchester United on Wednesday night on Sky Sports 1 HD.
You can also watch it online at wonderkidfilm.co.uk/watch