arrow_back PySpark: Convert Python Array/List to Spark Data Frame

V
venu
Stats
articleArticles 0
codeCode 0
imageDiagrams 0
descriptionNotebooks 0
chat_bubble_outlineThreads 0
commentComments 5
loyaltyKontext Points 5
access_time 2 years ago link more_vert
#1565 Re: PySpark: Convert Python Array/List to Spark Data Frame

Hi Raymond,


But it takes lot of time because of df.collect()

Is there any way to fasten this process? I tried to use --num-executors 5 in spark-submit but no change in performance. Also if possible please provide a solution for this too on how can we leverage --num-executors in this case. Since it's a 'pyspark dataframe' i also used df1 = df.toPandas() but no change in performance.


reply Reply
recommendMore from Kontext