Were you upfront with everybody about your decision? If yes, no worries; if not, then a hurt party may take you to task.