Forums

This forum is created for users to publish training related external information from other websites.

All advertisements about training must be published in this forum only. Otherwise the contents will be blocked. 

Discuss about cloud computing technologies, learning resources, etc. 

Any Kontext website related questions, please publish here incl. feature suggestions, bug reports, other feedbacks, etc. Visit Help Centre to learn how to use Kontext platform efficiently. 

New comments

Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Raymond access_time1 day ago

I'm glad it is working. If you do want to run it with Java 11, you need to use dataflow 2.27.0 or above versions. 

SDK version support status  |  Cloud Dataflow  |  Google Cloud

Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Jesús access_time1 day ago

Yes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.

Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time1 day ago
Sorry for the late reply. I've been busy recently and I have not got an environment with Spark and AD integrated. I will update you once I have that configured. 
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Raymond access_time1 day ago

What is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:

Dataflow does not run with Java 11 · Issue #3376 · GoogleCloudPlatform/google-cloud-eclipse (github.com)

Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Jesús access_time3 days ago

I couldn´t run the dataflow configuration, this log appeared, any ideas?:

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address

WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe

Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time20 days ago

We are using Spark 2.x. but the keytab was included in spark3.x could you please share the article to connect with SQL from the kerberos enabled spark cluster using kerberos authentication..

Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time21 days ago

For Kerberos enabled Spark cluster, it is usually used to authenticate with other Hadoop services like HDFS, Hive, HBase, etc. Access tokens will be retrieved from those services to be used in Spark application. There might be a way to use built-in functions to reuse but I am not familiar with those details.

To use Kerberos authentication to read data from SQL Server via keytab, you can pass in the keytab and principal parameters:

  • keytab Location of the kerberos keytab file (which must be pre-uploaded to all nodes either by --files option of spark-submit or manually) for the JDBC client. When path information found then Spark considers the keytab distributed manually, otherwise --files assumed. If both keytab and principal are defined then Spark tries to do kerberos authentication.
  • principal Specifies kerberos principal name for the JDBC client. If both keytab and principal are defined then Spark tries to do kerberos authentication.

Please follow this article find out more details: JDBC To Other Databases - Spark 3.1.1 Documentation (apache.org).

If you still could not work out the solution, I can find time to write a dedicated article for this.

Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time21 days ago

How this will work for kerberos enabled spark cluster.

Did you implement the ticket cache creation in python...? Please share it for reference.


Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time22 days ago

Hello,

The complete code is already provided here: Spark Read from SQL Server Source using Windows/Kerberos Authentication

The example code uses latest SQL Server JDBC driver which doesn't require keytab. Refer to the following article about how to generate Kerberos ticket using keytab (it also shows an example of generating that using Java programmatically):

Java Kerberos Authentication Configuration Sample & SQL Server Connection Practice



Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time22 days ago

Can you let me know how we set keytab location in the script.

Can you share the completed code..