Apache Spark: SparkFiles.get(fileName.txt) - Unable to retrieve the file contents from SparkContext


Apache Spark: SparkFiles.get(fileName.txt) - Unable to retrieve the file contents from SparkContext



I used SparkContext.addFile("hdfs://host:54310/spark/fileName.txt") and added a file to SparkContext. I verified its presence using org.apache.spark.SparkFiles.get(fileName.txt). It showed an absolute path, something like /tmp/spark-xxxx/userFiles-xxxx/fileName.txt.


SparkContext.addFile("hdfs://host:54310/spark/fileName.txt")


SparkContext


org.apache.spark.SparkFiles.get(fileName.txt)


/tmp/spark-xxxx/userFiles-xxxx/fileName.txt



Now I want to read that file from the above given absolute path
location from SparkContext. I tried
sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
It considers the path returned by SparkFiles.get() as a HDFS
path, which is incorrect.


SparkContext


sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)


SparkFiles.get()



I searched extensively to find any helpful reads on this, but ran out of luck.



Is there anything wrong in the approach? Any help is really appreciated.



Here is the code and the outcome:


scala> sc.addFile("hdfs://localhost:54310/spark/fileName.txt")

scala> org.apache.spark.SparkFiles.get("fileName.txt")
res23: String = /tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt

scala> sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:54310/tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
... 49 elided





Can you add code sample and error logs here
– vaquar khan
Jun 30 at 14:47





@vaquarkhan Added.
– Marco99
Jun 30 at 15:51




1 Answer
1



Refer to local file using the "file://" syntax.


sc.textFile("file://" + org.apache.spark.SparkFiles.get("fileName.txt"))
.collect()
.foreach(println)





Could you please explain how is that file related to local file system? I added the file to SparkContext.
– Marco99
Jun 30 at 15:50





When you add a file using sparkContext spark sends the file to driver and workers. SparkFiles.get call gets you the absolute path of the file which is local to the worker or driver it's not uploaded to hdfs.
– Sudev Ambadi
Jun 30 at 16:33





That was helpful, thank you!!
– Marco99
Jun 30 at 16:49






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

django NoReverseMatch Exception

django NoReverseMatch Exception I'm getting the following error while using hyperlink in Django. Error : django.urls.exceptions.NoReverseMatch: 'mywebapp' is not a registered namespace django.urls.exceptions.NoReverseMatch: 'mywebapp' is not a registered namespace mywebapp/template/about.html .. <body> <a href="{% url 'mywebapp:home' %}">Click here</a> Hello World!!!<p>Today is {{today}}</p> mywebapp/template/home.html .. <h4>Homepage.</h4> settings.py .. INSTALLED_APPS = [ ... 'mywebapp'#new ... ] Make sure you have app_name='home' in your mywebapp/urls.py . If that doesn’t solve the problem, then you need to show your urls.py . – Alasdair Jun 30 at 20:20 app_name='home' mywebapp/urls.py urls.py
Read more

List of Kim Possible characters

Image
From left to right: Wade (on monitor screen), Dr. James Timothy Possible, Mrs. Dr. Ann Possible, Ron, Rufus, Kim Possible, Shego, Dr. Drakken, Monkey Fist, Señor Senior Sr., Adrena Lynn, and Señor Senior Jr. This is a list of characters appearing in the animated series Kim Possible . Team Possible Kim Possible Main article: Kim Possible (character) Voiced by: Christy Carlson Romano and Dakota Fanning (Preschool Kim; Kim Possible: A Sitch In Time ) Kimberly Ann “Kim” Possible is a crime fighter and high school cheerleading captain who saves the world on a regular basis while dealing with the normal challenges of being a teenager, such as winning cheer competitions, turning in her homework on time, and maintaining a love life. Her name is a play on the word “impossible.” Kim has known Ron Stoppable, her sidekick for most missions, since preschool. She has also completed missions with Wade, Monique, her brothers, and even her mother. In season four, Kim and Ron end up developing romanti
Read more

Audio Livestreaming with Python & Flask

Audio Livestreaming with Python & Flask I'm currently strugling with my implementation of a simple live streaming web application using Python and Flask. It seems that I'm not able to stream my live recorded audio from the servers microphone input to the webpage. server.py from flask import Flask, render_template, Response import cv2 import framework import pyaudio import audio_processing as audioRec FORMAT = pyaudio.paInt16 CHANNELS = 1 RATE = 44100 CHUNK = 1024 audio = pyaudio.PyAudio() app = Flask(__name__) @app.route('/') def index(): """Video streaming home page.""" return render_template('index.html') # Stream routing @app.route('/video_feed') def video_feed(): """Video streaming route. Put this in the src attribute of an img tag.""" return Response(generateVideo(), mimetype='multipart/x-mixed-replace; boundary=frame') @app.route("/audio
Read more