"The simple fact is that the mandate (in June's referendum) was to leave the European Union - full stop. We need to keep that in mind when we are going through that process."