Hi Vishal, Thanks for the video .Here, the connectivity is established between Cloud SQL instance and Bigquery. But in case - if it's to be done with On-Premise data (mysql, oracle or sql server) , do we need to follow the same steps or any change is to be done ? From ETL perspectives, can we consider this because if there is any transformation change then do we need to handle it in Bigquery (then it will become ELT ). If the table is huge then what would be the best option ?
If source is at on-prem then you have to use third connection method that is private connectivity, which can be done using VPN or Cloud interconnect . This is standard procedure if you have hybrid connectivity with your onprem datacenter. For large data , backfill may take time and need round of POC before actual migration. Transformation, I don't see any option directly with Datastream.
If I want to take data from SQL server or Oracle on premise (from my laptop) to Bigquery , will it be challenging to create it on laptop ? Any video which can be helpful to create this ? I tried it by using one artcle from medium but faced some challenges while setting up mysql instance as it was giving message i.e not able to allocate IP Connect Private Data Fusion instance with a Private Cloud SQL instance using CloudSQL proxy
Do you have anything that explains extracting data from BQ and writing it to cloud SQL ?
Hi Vishal,
Thanks for the video .Here, the connectivity is established between Cloud SQL instance and Bigquery. But in case - if it's to be done with On-Premise data (mysql, oracle or sql server) , do we need to follow the same steps or any change is to be done ?
From ETL perspectives, can we consider this because if there is any transformation change then do we need to handle it in Bigquery (then it will become ELT ). If the table is huge then what would be the best option ?
If source is at on-prem then you have to use third connection method that is private connectivity, which can be done using VPN or Cloud interconnect . This is standard procedure if you have hybrid connectivity with your onprem datacenter.
For large data , backfill may take time and need round of POC before actual migration.
Transformation, I don't see any option directly with Datastream.
If I want to take data from SQL server or Oracle on premise (from my laptop) to Bigquery , will it be challenging to create it on laptop ?
Any video which can be helpful to create this ?
I tried it by using one artcle from medium but faced some challenges while setting up mysql instance as it was giving message i.e not able to allocate IP
Connect Private Data Fusion instance with a Private Cloud SQL instance using CloudSQL proxy
Thanks