Spark - How to create a DataFrame (different ways)
How to create a Spark DataFrame in different ways DataFrames are a table-like interface in Spark. The DataFrame structure is a distributed collection of data, build on Dataset. Since DataFrame have a defined schema and types it allows Spark to utilize different optimization on top of the schema and to execute calculations faster. Below we have initial code to instantiate SparkContext , SQLContext and SparkSession code. from pyspark.sql import Row from pyspark import SparkContext from pyspark import SQLContext sc = SparkContext() sqlContext = SQLContext(sc) from pyspark.sql import SparkSession spark = SparkSession . builder \ . master( 'local' ) \ . appName( 'Create DataFrame' ) \ . getOrCreate() toDF method Perhaps the most simple way to create a DataFrame is to use the toDF method. All you need is to supply a list of column names (strings) and a list of tuples as data. If you don't specify the column names Spark generate the...