Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Kontext Newsletter
Subscribe to Kontext Newsletter to get updates about data analytics, programming and cloud related articles. You can unsubscribe at anytime.
Log in with external accounts
comment Comments
#1465 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Yes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.
person Raymond access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
What is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:
#1463 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
What is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:
person Jesús access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
I couldn´t run the dataflow configuration, this log appeared, any ideas?:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe
#1462 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
I couldn´t run the dataflow configuration, this log appeared, any ideas?:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe
#1466 Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
I'm glad it is working. If you do want to run it with Java 11, you need to use dataflow 2.27.0 or above versions.
SDK version support status | Cloud Dataflow | Google Cloud
person Jesús access_time 2 years ago
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Yes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.