PM pulls back from Yosef’s words