An own goal from Mike Penders followed by a fine strike from Mathys Tel secured Spurs' first win at Stamford Bridge in over eight years.
Spurs dominated from the first minute, firmly deserving the win, but Cole Palmer's red card after the opening goal made their job easier.
Cole Palmer narrowly escaped a straight red card minutes earlier when shoving Matheus Fernandes, who was prone on the floor, in the face with his hip.
Minutes later a tough tackle on Conor Gallagher was enough for the referee to give Palmer his marching orders.
"Palmer has lost his head," Neville said while on co-commentary duty for Sky Sports.
Tel added a second goal from distance to wrap up the win, but there was still time for more wreckless tackling from Chelsea as Morgan Rogers tripped Gallagher as the game entered injury time.
Rogers was shown a yellow card for the tackle, but Neville believes the England man should have also been sent off.
"I think that's a red," Neville said. “I do. He's absolutely wellied him with no intent.
"That's ludicrous from Rogers. They have lost it. Xabi Alonso needs to get them off."
Neville added: "Tottenham have battered Chelsea in midfield. They have absolutely bossed the game."