Posts

Showing posts with the label hadoop

Nested Select not working in hive

Nested Select not working in hive FROM ( SELECT upper(name), salary, deductions["service tax"] as service_tax, round(salary * (1 - deductions["service tax"])) as salary_minus_service_taxes FROM employees ) e SELECT e.name, e.salary_minus_service_taxes WHERE e.salary_minus_service_taxes > 70000; When I run the above query I get below error FAILED: SemanticException [Error 10002]: Line 6:10 Invalid column reference 'name' improve forming for code – Osama AbuSitta Jul 1 at 6:13 1 Answer 1 The column should be aliased: SELECT upper(name) as name By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie ...

PySpark - SparkContext: Error initializing SparkContext File does not exist

Image
PySpark - SparkContext: Error initializing SparkContext File does not exist I have small piece code in PySpark, but I keep getting errors. I'm new to this so im not sure where to start. from pyspark import SparkContext, SparkConf conf = SparkConf().setAppName("Open json").setMaster("local[3]") sc = SparkContext(conf = conf) print("Done") I ran this in cmd with the command : spark-submit .PySparkOpen.py I then get the following error statement: C:UsersAbdullahDocumentsMaster Thesis>spark-submit .PySparkOpen.py 18/06/30 15:21:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable 18/06/30 15:22:01 ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/C:/Users/Abdullah/Documents/Master%20Thesis/PySpark/Open.py does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) at...