You can overrule naysayers if you want, but their concerns are still valid. It would be in your best interest to work things out.