This guide was provided by one of our beloved customer.
We first set up Linked Service for BigQuery in Data Factory. The video below (a bit old, but the approach still works) shows how that can be done with a service account.
Then we set up a pipeline that:
- reads all table names from the information schema and
- then loops over each table name and copies the data over table by table.
It’s a straightforward process that way and we haven’t had any hiccups.
Hope this helps!