R RAVULA visibility 34 comment 6 access_time 11 months ago languageEnglish
more_vert

Json dataframe to CSV dataframe

Why do we need schema for a dataframe before converting json dataframe to CSV dataframe? In pyspark dataframe

thumb_up 0
comment Comments
R RAVULA access_time 10 months ago more_vert
#1749 Re: Json dataframe to CSV dataframe

Consider this array:




 {


Array 1:[


{


Info_:{


"a":1,


"b":4


},


Arry_inside:[


Item1,


Item2


]


},


{


Info_:{


"a":5,


"b":9


},


Arry_inside:[


Item1,


Item2,


Item3,


Item4


]},


]


}


Above json contains inside array is a growing array so how can we flat....i don't think that will work....can you please see these kind of json files




Some we can do it may be.....


I don't know that......




So basically the JSON in my case it is getting from RESTAPI response for a POST request to a server..




The response is a JSON...


Which look like above mentioned looks very simple now but it's not 😂,, but i can't share that one ..




In response Json file contains




Mainly one Array 1 inside that we are getting like 10k - 30k objects of struct type


Inside each object:


There 10 keys are struct type and 10 values arry type ...


All values are null is allowed


So here is the deal the one Array kind of key that is the Messege kind of stuff is there inside the complex types similar to the json i mentioned inside array if go inside items one key value pair message is there


We need to extract that each mesage in each row...



Raymond Raymond access_time 10 months ago more_vert
#1748 Re: Json dataframe to CSV dataframe

As CSV format is flat and doesn’t support nested types hence you will need to flatten your DataFrame before writing it as CSV format.

format_quote

person RAVULA access_time 11 months ago
Re: Json dataframe to CSV dataframe

For CSV :-

We can easily the data goes under easily ETL jobs.

But for json frames not

For json having complex levels like 

Example : struct {

Array 1:[

Info_:{

Some values

},

Arry_inside:[

Some values 

]

]

}

so we don't have choice to do like CSV so then we need to extract values the. We need to produce the output file as CSV from json.

R RAVULA access_time 11 months ago more_vert
#1745 Re: Json dataframe to CSV dataframe

For CSV :-

We can easily the data goes under easily ETL jobs.

But for json frames not

For json having complex levels like 

Example : struct {

Array 1:[

Info_:{

Some values

},

Arry_inside:[

Some values 

]

]

}

so we don't have choice to do like CSV so then we need to extract values the. We need to produce the output file as CSV from json.

format_quote

person Raymond access_time 11 months ago
Re: Json dataframe to CSV dataframe

Can you please define what is a JSON or CSV DataFrame?

If a JSON document is loaded into memory as a DataFrame, it can then be saved as other format like CSV. 

Before you load JSON document, you usually can define a schema or Spark can also infer schema.

R RAVULA access_time 11 months ago more_vert
#1744 Re: Json dataframe to CSV dataframe

For CSV :-

We can easily the data goes under easily ETL jobs.

But for json frames not

For json having complex levels like 

Example : struct {

Array 1:[

Info_:{

Some values

},

Arry_inside:[

Some values 

]

]

}

so we don't have choice to do like CSV so then we need to extract values the. We need to produce the output file as CSV from json.

format_quote

person Raymond access_time 11 months ago
Re: Json dataframe to CSV dataframe

Can you please define what is a JSON or CSV DataFrame?

If a JSON document is loaded into memory as a DataFrame, it can then be saved as other format like CSV. 

Before you load JSON document, you usually can define a schema or Spark can also infer schema.

R RAVULA access_time 11 months ago more_vert
#1743 Re: Json dataframe to CSV dataframe

For CSV :-

We can easily the data goes under easily ETL jobs.

But for json frames not

For json having complex levels like 

Example : struct {

Array 1:[

Info_:{

Some values

},

Arry_inside:[

Some values 

]

]

}

so we don't have choice to do like CSV so then we need to extract values the. We need to produce the output file as CSV from json.

format_quote

person Raymond access_time 11 months ago
Re: Json dataframe to CSV dataframe

Can you please define what is a JSON or CSV DataFrame?

If a JSON document is loaded into memory as a DataFrame, it can then be saved as other format like CSV. 

Before you load JSON document, you usually can define a schema or Spark can also infer schema.

Raymond Raymond access_time 11 months ago more_vert
#1742 Re: Json dataframe to CSV dataframe

Can you please define what is a JSON or CSV DataFrame?

If a JSON document is loaded into memory as a DataFrame, it can then be saved as other format like CSV. 

Before you load JSON document, you usually can define a schema or Spark can also infer schema.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

recommendMore from Kontext