He had until Monday to respond to a letter from Mr Wright, the Hartlepool MP.
In his response, on 10 March, Mr Ashley asked the committee to come to the Sports Direct premises.
Mr Ashley said he will not “stand idle” while Sports Direct is “subjected to public vilification”.
“The current intention is not to go, because [MPs] ought to see it for themselves,” he told Sky News.
“In my opinion, they’re just showboating. In my opinion, they’re actually a joke.
“They don’t care about the people, they care about the business of politics.”
Mr Wright told the BBC the matter will be discussed at the next committee meeting on Tuesday.
“I’m very disappointed that Mr Ashley has refused to accept the committee’s request to come and give evidence in Parliament, like every other witness for every other select committee voluntarily does,” he added.