access_time 4 years ago languageEnglish
more_vert

Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

visibility 14,480 comment 4
This page documents the detailed steps to load CSV file from GCS into BigQuery using Dataflow to demo a simple data flow creation using Dataflow Tools for Eclipse. However it doesn’t necessarily mean this is the right use case for DataFlow. Alternatively bq command line or programming APIs ...
info Last modified by Raymond 2 years ago
thumb_up 3

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

comment Comments
2 years ago link more_vert
Raymond Raymond
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1466 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

I'm glad it is working. If you do want to run it with Java 11, you need to use dataflow 2.27.0 or above versions. 

SDK version support status  |  Cloud Dataflow  |  Google Cloud

format_quote

person Jesús access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

Yes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.

2 years ago link more_vert
J
Jesús
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 2
loyaltyKontext Points 2
#1465 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

Yes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.

format_quote

person Raymond access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

What is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:

Dataflow does not run with Java 11 · Issue #3376 · GoogleCloudPlatform/google-cloud-eclipse (github.com)

2 years ago link more_vert
Raymond Raymond
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1463 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

What is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:

Dataflow does not run with Java 11 · Issue #3376 · GoogleCloudPlatform/google-cloud-eclipse (github.com)

format_quote

person Jesús access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

I couldn´t run the dataflow configuration, this log appeared, any ideas?:

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address

WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe

2 years ago link more_vert
J
Jesús
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 2
loyaltyKontext Points 2
#1462 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

I couldn´t run the dataflow configuration, this log appeared, any ideas?:

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address

WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe

timeline Stats
Page index 10.65