Dev Stack Source

Answer : While you can use a UserDefinedFunction it is very inefficient . Instead it is better to use concat_ws function: from pyspark.sql.functions import concat_ws df.withColumn("test_123", concat_ws(",", "test_123")).show() +----+----------------+ |uuid| test_123| +----+----------------+ | 1|test,test2,test3| | 2|test4,test,test6| | 3|test6,test9,t55o| +----+----------------+ You can create a udf that joins array/list and then apply it to the test column: from pyspark.sql.functions import udf, col join_udf = udf(lambda x: ",".join(x)) df.withColumn("test_123", join_udf(col("test_123"))).show() +----+----------------+ |uuid| test_123| +----+----------------+ | 1|test,test2,test3| | 2|test4,test,test6| | 3|test6,test9,t55o| +----+----------------+ The initial data frame is created from: from pyspark.sql.types import StructType, StructField schema = StructType([StructField("uuid",Int...

Answer : Spark 1.6+ You can use stddev_pop to compute population standard deviation and stddev / stddev_samp to compute unbiased sample standard deviation: import org.apache.spark.sql.functions.{stddev_samp, stddev_pop} selectedData.groupBy($"user").agg(stdev_pop($"duration")) Spark 1.5 and below ( The original answer ): Not so pretty and biased (same as the value returned from describe ) but using formula: you can do something like this: import org.apache.spark.sql.functions.sqrt selectedData .groupBy($"user") .agg((sqrt( avg($"duration" * $"duration") - avg($"duration") * avg($"duration") )).alias("duration_sd")) You can of course create a function to reduce the clutter: import org.apache.spark.sql.Column def mySd(col: Column): Column = { sqrt(avg(col * col) - avg(col) * avg(col)) } df.groupBy($"user").agg(mySd($"duration").a...

Answer : I don't think your question makes sense -- your outermost Map , I only see you are trying to stuff values into it -- you need to have key / value pairs in your outermost Map . That being said: val peopleArray = df.collect.map(r => Map(df.columns.zip(r.toSeq):_*)) Will give you: Array( Map("age" -> null, "name" -> "Michael"), Map("age" -> 30, "name" -> "Andy"), Map("age" -> 19, "name" -> "Justin") ) At that point you could do: val people = Map(peopleArray.map(p => (p.getOrElse("name", null), p)):_*) Which would give you: Map( ("Michael" -> Map("age" -> null, "name" -> "Michael")), ("Andy" -> Map("age" -> 30, "name" -> "Andy")), ("Justin" -> Map("age" -> 19, "name" -> "Justin")) ) I'm guess...