Olsson was part of the Swansea side that lost 1-0 on Saturday, with Diafra Sakho grabbing a late goal at the London Stadium.
Paul Clement’s men failed to any of their September fixtures and they have now picked up just five points from seven matches.
Swansea currently sit 18th in the Premier League table but Olsson insists the squad need to remain together if they are to improve their fortunes.
He told reporters: “It’s not the time to panic. We showed last season what we can do. We have almost the same kind of players.
“We know the strikers we have, their qualities, midfield, everyone. We need to stick together. The team, the fans, the staff.
“We’ve got a good group and everyone is behind the manager so we’ve just go to work hard and change that trend and get some more wins.”