Home World UK minister sees Brexit take care of Labour, opposition needs flexibility