Saturday, January 4, 2020

Rename column and operators(and & or I) using pyspark

#Renaming of column

>>> df2=(df.withColumnRenamed("last_name","lastnm"))
>>> df2.printSchema()
root
 |-- registration_dttm: timestamp (nullable = true)
 |-- id: integer (nullable = true)
 |-- first_name: string (nullable = true)
 |-- lastnm: string (nullable = true)
 |-- email: string (nullable = true)
 |-- gender: string (nullable = true)


#Using & (and ) operator

 df3=df2.withColumn("first_name",when((col("id")=='1') & (col("gender")=="Male"),"myvalue").otherwise(col("first_name")))

Because we are using operator so please enclose each conditions in round brace of  each side of operator.

No comments:

Post a Comment