‘It’s not just the rich, we’ve got regular people leaving and moving out, which is sad’