Mboko dropped her racquet and covered her face in disbelief as the packed crowd leapt to their feet after the 18-year-old tournament wildcard sealed the  6-1 6-4 victory in just over an hour on Saturday (local time).
"Thank you, you helped me so much," she told the home crowd in French during an on-court interview.
Mboko called her first-ever appearance in the quarter-finals of a WTA 1000 event "incredible", especially after toppling reigning French Open champion Gauff.
The Canadian was helped along the way by her error-prone opponent, who never found her rhythm on serve and failed to convert any of her five break-point opportunities.
Gauff's sixth double fault set up match point, which Mboko converted when the American's backhand slice drifted into the net.
Despite her disappointment, Gauff hugged Mboko at the net and will look to regroup as she prepares to make a run at a second US Open crown next month.
In the quarter-finals, Mboko will face Spaniard Jessica Bouzas Maneiro, who scored a 7-5 1-6 6-2 win over China's Lin Zhu in a battle of unseeded players.
Earlier, ninth seed Elena Rybakina edged her way past Dayana Yastremska 5-7 6-2 7-5 in their fourth-round battle.
The Kazakhstani will next face another Ukrainian, 24th seed Marta Kostyuk, who came from behind to take out American McCartney Kessler 5-7 6-3 6-3.
