Despite the 35th-minute dismissal of Kevin Mirallas, the hosts – remarkably – were 2-0 up with three-quarters of the match gone thanks to goals from Romelu Lukaku and Aaron Lennon.
But in a huge turnaround in fortunes, Michail Antonio and substitute Diafra Sakho struck within four minutes of each other, allowing Payet to nick three points in the 90th minute.
West Ham manager Slaven Bilic admits even he did not know what to do when Payet’s winner went in.
He said: “I went down the tunnel to calm down and came back straight away,
“It was a great comeback, a great game. To do it like this is unbelievable.
“I said at half-time, ‘We are one down but we are going to do it. We have to be less sexy and more lethal in front of the box’.
“We knew it was not going to be one-sided match when you have a player like Lukaku.
“When they had the ball, he makes that one-player loss less visible. He creates mayhem.”