Dev Stack Source

Answer : Two things. When you use :+ , the operation is left associative , meaning the element you're calling the method on should be on the left hand side. Now, Seq (as used in your example) refers to immutable.Seq . When you append or prepend an element, it returns a new sequence containing the extra element, it doesn't add it to the existing sequence. val newSeq = customerList :+ CustomerDetail("1", "Active", "Shougat") But appending an element means traversing the entire list in order to add an item, consider prepending: val newSeq = CustomerDetail("1", "Active", "Shougat") +: customerList A simplified example: scala> val original = Seq(1,2,3,4) original: Seq[Int] = List(1, 2, 3, 4) scala> val newSeq = 0 +: original newSeq: Seq[Int] = List(0, 1, 2, 3, 4) It might be worth pointing out that while the Seq append item operator, :+ , is left associative , the prepend operator, +: , is right as...

Answer : Spark 1.6+ You can use stddev_pop to compute population standard deviation and stddev / stddev_samp to compute unbiased sample standard deviation: import org.apache.spark.sql.functions.{stddev_samp, stddev_pop} selectedData.groupBy($"user").agg(stdev_pop($"duration")) Spark 1.5 and below ( The original answer ): Not so pretty and biased (same as the value returned from describe ) but using formula: you can do something like this: import org.apache.spark.sql.functions.sqrt selectedData .groupBy($"user") .agg((sqrt( avg($"duration" * $"duration") - avg($"duration") * avg($"duration") )).alias("duration_sd")) You can of course create a function to reduce the clutter: import org.apache.spark.sql.Column def mySd(col: Column): Column = { sqrt(avg(col * col) - avg(col) * avg(col)) } df.groupBy($"user").agg(mySd($"duration").a...

Answer : How about just wrapping it (I'm assuming there's an implicit ExecutionContext here): val scalaFuture = Future { javaFuture.get } EDIT: A simple polling strategy could look like this (java.util.Future => F): def pollForResult[T](f: F[T]): Future[T] = Future { Thread.sleep(500) f }.flatMap(f => if (f.isDone) Future { f.get } else pollForResult(f)) This will check if the Java future is done every 500ms. Obviously the total blocking time is the same as above (rounded up to the nearest 500ms) but this solution will allow other tasks to be interleaved in the same thread of the ExecutionContext . Starting Scala 2.13 , the standard library includes scala.jdk.FutureConverters which provides Java to Scala CompletableFuture/Future implicit conversions: import scala.jdk.FutureConverters._ // val javaFuture = java.util.concurrent.CompletableFuture.completedFuture(12) val scalaFuture = javaFuture.asScala // scalaFuture: scala.concurrent.Future[Int] = Future(...

Answer : Such statement can be used import org.apache.spark.sql.functions._ dataFrame.select(col("a"), substring_index(col("a"), ",", 1).as("b")) Suppose you have the following dataframe: import spark.implicits._ import org.apache.spark.sql.functions._ var df = sc.parallelize(Seq(("foobar", "foo"))).toDF("a", "b") +------+---+ | a| b| +------+---+ |foobar|foo| +------+---+ You could subset a new column from the first column as follows: df = df.select(col("*"), substring(col("a"), 4, 6).as("c")) +------+---+---+ | a| b| c| +------+---+---+ |foobar|foo|bar| +------+---+---+ You would use the withColumn function import org.apache.spark.sql.functions.{ udf, col } def substringFn(str: String) = your substring code val substring = udf(substringFn _) dataframe.withColumn("b", substring(col("a"))