Thanks for the confirmation. 

Appproch 2 worded for me.

use the vs code. rename image.pyd to image.py.

Can you provide your table creation statement and also the insert statement?

It behaves in that way because you have converted the object to JSON string. 

Try the following code to remove to_json and let the final DataFrameWriter to convert it as JSON format properly:

df=df.select(struct("author", "title", "pages", "email").alias("json-data")).agg(collect_list("json-data").alias("list-item"))

The code will generate the following output:

{"list-item":[{"author":"author1","title":"title1","pages":1,"email":"author1@gmail.com"},{"author":"author2","title":"title2","pages":2,"email":"author2@gmail.com"},{"author":"author3","title":"title3","pages":3,"email":"author3@gmail.com"},{"author":"author4","title":"title4","pages":4,"email":"author4@gmail.com"}],"version":1}

You need to run some tests to find out.

One is using JDBC and another using Python native driver. I dont think there be much major performance differences as the Spark read from JDBC will not run in parallel anyway. So the main performance differences will be the difference between these two drivers.

If you do want to extract data using JDBC in parallel by utilizing partition columns, you can consider Sqoop.

question for you: Is is more or less performant to use the spark only technique of:

spark_df = spark.read.format("jdbc").option("url", "jdbc:mysql://<host_string>/<database>").option("driver", "com.mysql.cj.jdbc.Driver").option("dbtable", "(SELECT * FROM some_table) temp_spark_table").option("user", "theusername").option("password", "thepassword").load()

欢迎来到Kontext，Mengz！谢谢您的反馈，目前尚未支持Markdown编辑器，在今后发布的版本中会考虑此功能！

如果你在使用中有发现任何Bug或者可以改进的地方，欢迎告知！

谢谢，

Administrator

Thanks for the clarifications. It makes sense.