The Everton forward announced his international retirement on Wednesday, ending his England career as the country's record goalscorer with 53 goals in 119 games.
Neville, who worked with Rooney in the England set-up under Roy Hodgson, believes that Rooney will be held in high-regard by England fans for the commitment he showed to the shirt.
"Quite simply [we should remember him] as somebody who every time he was picked he gave his all," Neville said.
"The goals and all the rest of it come on top of that, but [he was] somebody who fought for his country every time he played, gave his absolute maximum, and did everything he possibly could to help England win a tournament. I think that ultimately will be the biggest disappointment of Wayne Rooney's career with England.
"The reality of it is he will be remembered as somebody who gave his all, with fantastic spirit, and with high quality."
Neville also believes the decision is best for all concerned, including England manager Gareth Southgate ahead of the 2018 World Cup.
When asked if Rooney had made the right call, Neville said: "There are very few personalities in English football, I think you can go back to maybe David Beckham before him, Paul Gascoigne before that, it can become a huge strain not just on the player, but also on the manager and the coaching staff, that every time the squad announcement comes up the question is whether Wayne Rooney is going to be in the team.
"I think it can drain everybody, and I think for that reason it's probably the right decision for everybody."