The on-again, off-again tariffs and the high-level discussions surrounding them have exasperated negotiating teams, sources say.