The hosts took the lead when Eric Lichaj headed in from close range, although the advantage lasted just three minutes as Per Mertesacker followed up after Rob Holding’s header had struck the post.
Lichaj reclaimed the lead for a Forest side led by caretaker boss Gary Brazil, the American defender scoring a spectacular dipping volley from the edge of the area a minute before half-time.
Forest pulled further clear in the second half when Ben Brereton sent David Ospina the wrong way from the penalty spot after Holding upended Matty Cash.
Danny Welbeck reduced the deficit for Wenger’s much-changed side after Forest goalkeeper Jordan Smith failed to hold a through-ball.
But Arsenal’s first FA Cup third-round defeat since January 1996 – in a replay against Sheffield United – was confirmed courtesy of Kieran Dowell’s 85th-minute penalty, before Forest’s Joe Worrall was shown a straight red card in the closing stages for a late tackle.
Wenger has not held back in his criticism of referees in recent weeks, so much so the Frenchman was forced to watch this tie from the stands as he served the first of a three-match touchline ban handed down by the Football Association.
The Gunners boss may feel he has another grievance to air at the end of this match, after referee Jon Moss elected to allow Forest’s second penalty kick despite Arsenal players arguing that Dowell touched the ball twice after slipping on his run-up.
It is unclear if Arsenal deserved to have their appeals upheld, with repeated replays proving inconclusive.
Wenger sent out a side showing nine changes from Wednesday’s thrilling 2-2 draw with Chelsea, the manager clearly with one eye on next week’s Carabao Cup semi-final first leg against the Blues.
The 68-year-old has fielded weakened sides in cup competitions countless times down the years, but on this occasion he appeared to underestimate his mid-table Championship opponents, who showed great desire en route to a famous upset.
For their part, Arsenal’s youthful side produced a disjointed display, with the experience provided by Mertesacker and Welbeck not enough as the visitors saw their hopes of winning this trophy for the fourth time in five years comprehensively ended at the first hurdle.
Last week’s 1-0 home defeat to struggling Sunderland was the trigger for Forest to sack manager Mark Warburton with the side 14th in the table and having won one of their past seven league games.
Academy manager Brazil oversaw Monday’s goalless draw with play-off hopefuls Leeds United and that turnaround in form continued from the first whistle against Arsenal.
The two-time FA Cup winners were hungrier in all areas, and despite Arsenal enjoying close to 70% possession, Forest fully deserved the win.
The stats also showed Arsenal had the ball in the Forest final third for 30% of the match, but crucially the home side had nine shots on target compared to four from the visitors.
Forest simply did more with the ball when they had it, and should they able to transfer that into the Championship for the remainder of the season, any fears of relegation should be far from their thoughts.
Arsenal boss Arsene Wenger: “It was very frustrating [being in the stands] and unfortunately it was a case on top of that of seeing the team lose. We played a very good Forest team who were sharp and focused. The challenges were decisive. Overall our performance wasn’t good enough.
“We have many players on the edge [of fitness] and we play again on Wednesday. When you don’t win the selection is always questioned. I believe we had an experienced team – eight or nine internationals. It would be easy to think that was the problem.
“When you’re against a team like that they’re very difficult to beat. Defensively you cannot afford the mistakes we made.”
Forest caretaker boss Gary Brazil: “That was a fantastic day for the football club. I may be a little bit biased but I think it was a well-deserved win. There were some big performances from the players this afternoon and I always felt comfortable we would go on and win the game and so it proved.
“It would have been foolish to play the game Arsenal wanted us to so we had to go on the front foot and get in amongst them. We didn’t want to give them space or time on the ball or allow them to dictate the game. Our first 10 minutes showed our intent and we really set the tempo of the contest.”