Pat Handley has stepped down as chairman of Pacific Brands, ceding his position to another board member, James MacKenzie.