Integration of pyspark with hive

Problem:- If you want use hive tables with Spack Sql and vice versa. So that case you need to follow steps.

Solution:- Please find the steps.

Step 1- Copy hive-site.xml from  /hive/config to /spark/config location.

Step 2-  Add below property with hive-site.xml  in spark and hive
 
You can provide localhost or ip of machine.

            <property>
                       <name>hive.metastore.uris</name>
                       <value>thift://localhost:9083</value>
                      <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
             </property>


Step 2- Add hive path in spark-env.sh.
         e.g  export HIVE_HOME=$HOME/Software/hive 

Step3- Start Hive metastore using below command
           bin/hive --service metastore

Step4 - Start  pyspark
       >>> spark.sql("show databases").show()
+------------+
|databaseName|
+------------+
|     default|
|     sparkdb|
|      testdb|
+------------+

If you are getting  below error while starting hive or running spark sql commnd on spark.

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

Solution:- Provide any appropriate folder name in /tmp

<property>

    <name>system:java.io.tmpdir</name>

    <value>/tmp/hive/spark</value>

  </property>

  <property>

    <name>system:user.name</name>

    <value>${user.name}</value>

  </property>