Montanier had been linked with the coach’s job at Lens, but has instead taken the opportunity to manage in England.
He took over at Real Sociedad in 2011 and guided them to a fourth-placed finish and a spot in the Champions League in 2013.
However, his contract was not renewed and he then spent three years at Rennes, who reached the Coupe de France final in 2014.
BBC Radio Nottingham’s Robin Chipperfield:
“It’s a time of great intrigue at the City Ground. Montanier becomes the first non-British manager appointed by the Reds – an appointment that has taken a long time to announce. Remember, Dougie Freedman was sacked on 13 March.
“One of the most interesting aspects of Montanier’s tenure could be how well he works with Pedro Perreira, who was recently appointed as the club’s new director of football.
“If they can forge a strong partnership, and their views on the style of football, and potential new signings dovetail, then a prosperous future is more likely.
“The success achieved by Watford and Sheffield Wednesday under Slavisa Jokanovic and Carlos Carvalhal respectively have to be the models for Forest to follow.
“There is still plenty of work to be done at Forest, but this feels like a very significant step.”