If others want to complain about your job performance then offer to pass their responsibilities back to them. This ensures they’ll keep quiet in the future.