arrow_back Run Multiple Python Scripts PySpark Application with yarn-cluster Mode
Kontext Newsletter
Subscribe to Kontext Newsletter to get updates about data analytics, programming and cloud related articles. You can unsubscribe at anytime.
beta
© Kontext 2022 v1.0.0-beta - build-20220122.1
Powered by .NET 6.0.0 on Azure. Made with favorite_border in Melbourne.
#1545 Re: Run Multiple Python Scripts PySpark Application with yarn-cluster Mode
Hi Venu,
The code example you provided to me are local file write which has nothing to do with Spark:
with open("/user/user_name/myfile.ttl",mode='w+') as file:# It's a turtle file.file.write("This is truth")
The above lines will run in driver application container in the Spark cluster.
That is why I made the comments before.
For me to illustrate more, can you share complete script if it is okay?
person venu access_time 3mo
Re: Run Multiple Python Scripts PySpark Application with yarn-cluster Mode
Hi Raymond
Thanks for the reply!
I have some doubts. (.txt) is just an example but actually, I want to store .ttl type of file(turtle file)Want to store RDF(Resource descriptive framework) Triples. I don't want to read the file. I already have read the data using Spark. read and stored it in a data frame. After transforming the data I just want to write the output of the program to a file in spark-cluster mode.
Note:- I have tried your suggestion but still it gives the same error.
Can you please provide some more detailed explanation/solution?