Apache Spark: SparkFiles.get(fileName.txt) - Unable to retrieve the file contents from SparkContext

Multi tool use
Multi tool use


Apache Spark: SparkFiles.get(fileName.txt) - Unable to retrieve the file contents from SparkContext



I used SparkContext.addFile("hdfs://host:54310/spark/fileName.txt") and added a file to SparkContext. I verified its presence using org.apache.spark.SparkFiles.get(fileName.txt). It showed an absolute path, something like /tmp/spark-xxxx/userFiles-xxxx/fileName.txt.


SparkContext.addFile("hdfs://host:54310/spark/fileName.txt")


SparkContext


org.apache.spark.SparkFiles.get(fileName.txt)


/tmp/spark-xxxx/userFiles-xxxx/fileName.txt



Now I want to read that file from the above given absolute path
location from SparkContext. I tried
sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
It considers the path returned by SparkFiles.get() as a HDFS
path, which is incorrect.


SparkContext


sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)


SparkFiles.get()



I searched extensively to find any helpful reads on this, but ran out of luck.



Is there anything wrong in the approach? Any help is really appreciated.



Here is the code and the outcome:


scala> sc.addFile("hdfs://localhost:54310/spark/fileName.txt")

scala> org.apache.spark.SparkFiles.get("fileName.txt")
res23: String = /tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt

scala> sc.textFile(org.apache.spark.SparkFiles.get("fileName.txt")).collect().foreach(println)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:54310/tmp/spark-3646b5fe-0a67-4a16-bd25-015cc73533cd/userFiles-a7d54640-fab2-4dfa-a94f-7de6f74a0764/fileName.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
... 49 elided





Can you add code sample and error logs here
– vaquar khan
Jun 30 at 14:47





@vaquarkhan Added.
– Marco99
Jun 30 at 15:51




1 Answer
1



Refer to local file using the "file://" syntax.


sc.textFile("file://" + org.apache.spark.SparkFiles.get("fileName.txt"))
.collect()
.foreach(println)





Could you please explain how is that file related to local file system? I added the file to SparkContext.
– Marco99
Jun 30 at 15:50





When you add a file using sparkContext spark sends the file to driver and workers. SparkFiles.get call gets you the absolute path of the file which is local to the worker or driver it's not uploaded to hdfs.
– Sudev Ambadi
Jun 30 at 16:33





That was helpful, thank you!!
– Marco99
Jun 30 at 16:49






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

3dkQc 50,mz,c5mHu EADh,Oe,LXtmha mEoH8EXaTmPPFtl6ny
-
O1HKFJukW,u ceDtR3BPKOYhF4Fx,W4,5StEN E1tiM cZim67TYE4kOS V7,u8O t,2x,UC,mJ6t6N i,JorqwK F

Popular posts from this blog

PySpark - SparkContext: Error initializing SparkContext File does not exist

Image
PySpark - SparkContext: Error initializing SparkContext File does not exist I have small piece code in PySpark, but I keep getting errors. I'm new to this so im not sure where to start. from pyspark import SparkContext, SparkConf conf = SparkConf().setAppName("Open json").setMaster("local[3]") sc = SparkContext(conf = conf) print("Done") I ran this in cmd with the command : spark-submit .PySparkOpen.py I then get the following error statement: C:UsersAbdullahDocumentsMaster Thesis>spark-submit .PySparkOpen.py 18/06/30 15:21:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable 18/06/30 15:22:01 ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/C:/Users/Abdullah/Documents/Master%20Thesis/PySpark/Open.py does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) at...
Read more

django NoReverseMatch Exception

django NoReverseMatch Exception I'm getting the following error while using hyperlink in Django. Error : django.urls.exceptions.NoReverseMatch: 'mywebapp' is not a registered namespace django.urls.exceptions.NoReverseMatch: 'mywebapp' is not a registered namespace mywebapp/template/about.html .. <body> <a href="{% url 'mywebapp:home' %}">Click here</a> Hello World!!!<p>Today is {{today}}</p> mywebapp/template/home.html .. <h4>Homepage.</h4> settings.py .. INSTALLED_APPS = [ ... 'mywebapp'#new ... ] Make sure you have app_name='home' in your mywebapp/urls.py . If that doesn’t solve the problem, then you need to show your urls.py . – Alasdair Jun 30 at 20:20 app_name='home' mywebapp/urls.py urls.py ...
Read more

Audio Livestreaming with Python & Flask

Audio Livestreaming with Python & Flask I'm currently strugling with my implementation of a simple live streaming web application using Python and Flask. It seems that I'm not able to stream my live recorded audio from the servers microphone input to the webpage. server.py from flask import Flask, render_template, Response import cv2 import framework import pyaudio import audio_processing as audioRec FORMAT = pyaudio.paInt16 CHANNELS = 1 RATE = 44100 CHUNK = 1024 audio = pyaudio.PyAudio() app = Flask(__name__) @app.route('/') def index(): """Video streaming home page.""" return render_template('index.html') # Stream routing @app.route('/video_feed') def video_feed(): """Video streaming route. Put this in the src attribute of an img tag.""" return Response(generateVideo(), mimetype='multipart/x-mixed-replace; boundary=frame') @app.route("/audio...
Read more