Search between a Gas Station

How to flatten (or explode) the data along with row in a Dataframe, based on column data? Data frame should explode based on SPC column. Below is example My Input DataFrame. ID Name Level SPC Rating salry 23 sam 3 HBS 3.5 4000 43 Nair 4 KSTk 4 5000 56 Rom 5 MNC 3 3000 My output should be: ID Name level SPC Rating Salary 23 sam 3 H 3.5 4000 23 sam 3 B 3.5 4000 23 sam 3 S 3.5 4000 43 Nair 4 K 4 5000 43 Nair 4 S 4 5000 43 Nair 4 T 4 5000 43 Nair 4 k 4 5000 How can I resolve this problem in Scala or Java code? 2 Answers 2 If you have a dataframe/dataset as +---+----+-----+----+------+------+ |ID |Name|Level|SPC |Rating|salary| +---+----+---...

how do I fix the type mismatch issue? I have a spark streaming job and had a question about type converting. the below is my code: val component = data.get("viewed_objects").get.asInstanceOf[ListBuffer[Map[String, Any]]] but the exception is scala.collection.immutable.$colon$colon cannot be cast to scala.collection.ListBuffer what caused the issue and how do I fix it? yes,it is a json String,like this: u'data': {u'viewed_objects': [{u'location': 3, u'keyword': u'baby carriers'}]} – Zhang Xin Jul 1 at 3:47 I print the data value, it is for example: data:Map(viewed_objects -> List(Map(location -> 2, category_name -> Toys, Kids & Babies, category_id -> 27))), how do I get the viewed_objects and traversal the elements ...

Apache Spark: SparkFiles.get(fileName.txt) - Unable to retrieve the file contents from SparkContext I used SparkContext.addFile("hdfs://host:54310/spark/fileName.txt") and added a file to SparkContext . I verified its presence using org.apache.spark.SparkFiles.get(fileName.txt) . It showed an absolute path, something like /tmp/spark-xxxx/userFiles-xxxx/fileName.txt . SparkContext.addFile("hdfs://host:54310/spark/fileName.txt") SparkContext org.apache.spark.SparkFiles.get(fileName.txt) /tmp/spark-xxxx/userFiles-xxxx/fileName.txt Now I want to read that file from the above given absolute path location from SparkContext . I tried sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println) It considers the path returned by SparkFiles.get() as a HDFS path , which is incorrect. SparkContext sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println) SparkFiles.get() I searched extensiv...

PySpark - SparkContext: Error initializing SparkContext File does not exist I have small piece code in PySpark, but I keep getting errors. I'm new to this so im not sure where to start. from pyspark import SparkContext, SparkConf conf = SparkConf().setAppName("Open json").setMaster("local[3]") sc = SparkContext(conf = conf) print("Done") I ran this in cmd with the command : spark-submit .PySparkOpen.py I then get the following error statement: C:UsersAbdullahDocumentsMaster Thesis>spark-submit .PySparkOpen.py 18/06/30 15:21:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable 18/06/30 15:22:01 ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/C:/Users/Abdullah/Documents/Master%20Thesis/PySpark/Open.py does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) at...