Forums
This forum is created for users to publish training related external information from other websites.
All advertisements about training must be published in this forum only. Otherwise the contents will be blocked.
Any Kontext website related questions, please publish here incl. feature suggestions, bug reports, other feedbacks, etc. Visit Help Centre to learn how to use Kontext platform efficiently.
New threads
AWS Certified Cloud Practitioner Notes local_offer aws
Administrator access_time2 years agoIncorrect Captcha Code when registering or logging in local_offer kontext
Administrator access_time2 years ago[Fixed] Known bug - Language selection doesn't work local_offer bug local_offer kontext
Administrator access_time2 years agoNew site tags request for Kontext website local_offer kontext
Raymond access_time2 years agoNew comments
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Jesús access_time1 day agoYes, you're right, I changed the JDK for 1.8 and now works... more of less. In my dataflow console doesn't show any job crated but the big query table (csv archive) got uploaded succesfully.
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time1 day agoRe: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Raymond access_time1 day agoWhat is the version of your JDK? The example code can run successfully with JDK 1.8. It looks like you are hitting this error if you are using Java 11:
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Jesús access_time3 days agoI couldn´t run the dataflow configuration, this log appeared, any ideas?:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/jperezgarcia/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time20 days agoWe are using Spark 2.x. but the keytab was included in spark3.x could you please share the article to connect with SQL from the kerberos enabled spark cluster using kerberos authentication..
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time21 days agoFor Kerberos enabled Spark cluster, it is usually used to authenticate with other Hadoop services like HDFS, Hive, HBase, etc. Access tokens will be retrieved from those services to be used in Spark application. There might be a way to use built-in functions to reuse but I am not familiar with those details.
To use Kerberos authentication to read data from SQL Server via keytab, you can pass in the keytab and principal parameters:
- keytab Location of the kerberos keytab file (which must be pre-uploaded to all nodes either by --files option of spark-submit or manually) for the JDBC client. When path information found then Spark considers the keytab distributed manually, otherwise --files assumed. If both keytab and principal are defined then Spark tries to do kerberos authentication.
- principal Specifies kerberos principal name for the JDBC client. If both keytab and principal are defined then Spark tries to do kerberos authentication.
Please follow this article find out more details: JDBC To Other Databases - Spark 3.1.1 Documentation (apache.org).
If you still could not work out the solution, I can find time to write a dedicated article for this.
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time21 days agoHow this will work for kerberos enabled spark cluster.
Did you implement the ticket cache creation in python...? Please share it for reference.
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
Raymond access_time22 days agoHello,
The complete code is already provided here: Spark Read from SQL Server Source using Windows/Kerberos Authentication
The example code uses latest SQL Server JDBC driver which doesn't require keytab. Refer to the following article about how to generate Kerberos ticket using keytab (it also shows an example of generating that using Java programmatically):
Java Kerberos Authentication Configuration Sample & SQL Server Connection Practice
Re: Spark Read from SQL Server Source using Windows/Kerberos Authentication
venkatesan access_time22 days agoCan you let me know how we set keytab location in the script.
Can you share the completed code..
Re: Load CSV File from Google Cloud Storage to BigQuery Using Dataflow
Raymond access_time1 day agoI'm glad it is working. If you do want to run it with Java 11, you need to use dataflow 2.27.0 or above versions.
SDK version support status | Cloud Dataflow | Google Cloud