Palace raced into a 2-0 lead in the Manchester sunshine - and for a while their lengthy unbeaten away record looked to be safe.
But City fought back to level the game by half-time, before adding three further goals without reply in a difficult second 45 minutes for the Eagles.
Wharton was once again in the thick of action. And the Palace midfielder felt the first City goal, when Kevin De Bruyne curled home a free-kick, was a critical turning point.
"I thought we had a strong start," he told Premier League Productions. "The lads were a bit down. But obviously, we know how good a team City are.
"We knew if they got on top, it was going to be a tough game. And once they got that first goal, the momentum switched.
"They had most of the ball, but I thought when we won it back we didn't hold onto it well enough.
"At the start of the game, we were countering well and creating chances. After their first goal, when we were winning it back, we were a bit sloppy on the ball or giving it away too fast. That sort of killed us."
Despite a difficult defeat, Wharton says Palace will learn from the setback.
"It's definitely one for us to learn from. But I'm sure we'll be back and we're ready for the next game.
"I think there's plenty of positives, especially from the first 15, 20 minutes of today. But there's plenty of things for us to learn from and improve.
"We'll obviously look back at it and try and take the positives from it. But we've got to be ready in four days for the next game [against Newcastle]."