"As to any inconveniences that may be suggested from imagination, " the keeping strictly to the rule of not permitting a man to be judge in his own cause," is of more consequence than any such supposed inconveniences can weigh against."