arrow_back Apache Hive 3.1.2 Installation on Windows 10
person Orland access_time 3 years ago
Hi when I run the vi ~/.bashrc cygwin gets stuck and when I scroll down letters appear.
Hi when I run the vi ~/.bashrc cygwin gets stuck and when I scroll down letters appear.
Hi do you have a guide for cygwin installation for Hive?
I recommend using SQL Server as metastore. Each time when you init the metastore, you lose all the Hive metadata like databases, tables, etc.
person R4F43L access_time 4 years ago
Hi Raymond,
In your tutorial i use the derby embeded so i try to use and external derby following this tutorual "Installing Apache Hive 3.1.2 on Windows 10 - https://towardsdatascience.com/".
And it works fine but when i stop the hive server killing the process sometimes i have the same error so i remove metastore_db and with cygwin i do :
$HIVE_HOME/bin/schematool -dbType derby -initSchema
But all hive query are very slow... maybe it caused by the limitation you mention ". So i will try to change the db of the metastore maybe i will try to do with sql server running in Docker following this tutotorial how-to-run-sql-server-in-a-docker-container.
You think hive query can be faster if we change engin MR to Tez, you have a step-by-step to adapt it to tez ?
I share with you some issues i found during my test, in my case i want to write parquet files in hdfs then create hive external tables to query data and insert it on an internal hive table where i also need to delete data (this table must be transactional)
Error: User: MYUSERNAME is not allowed to impersonate MYUSERNAME
I fix it by following : stackoverflow - 43180305
java.lang.NoClassDefFoundError: org/apache/tez/dag/api/TezConfiguration
I fix it by following : stackoverflow.com - 29602670
Here the properties i added to fix my issues :
Hi Raymond,
In your tutorial i use the derby embeded so i try to use and external derby following this tutorual "Installing Apache Hive 3.1.2 on Windows 10 - https://towardsdatascience.com/".
And it works fine but when i stop the hive server killing the process sometimes i have the same error so i remove metastore_db and with cygwin i do :
$HIVE_HOME/bin/schematool -dbType derby -initSchema
But all hive query are very slow... maybe it caused by the limitation you mention ". So i will try to change the db of the metastore maybe i will try to do with sql server running in Docker following this tutotorial how-to-run-sql-server-in-a-docker-container.
You think hive query can be faster if we change engin MR to Tez, you have a step-by-step to adapt it to tez ?
I share with you some issues i found during my test, in my case i want to write parquet files in hdfs then create hive external tables to query data and insert it on an internal hive table where i also need to delete data (this table must be transactional)
Error: User: MYUSERNAME is not allowed to impersonate MYUSERNAME
I fix it by following : stackoverflow - 43180305
java.lang.NoClassDefFoundError: org/apache/tez/dag/api/TezConfiguration
I fix it by following : stackoverflow.com - 29602670
Here the properties i added to fix my issues :
person Raymond access_time 4 years ago
Hi,
From the screenshot, I could not identify any issues.
Are you using derby as metastore or external database like SQL Server or MySQL as metastore?
For Derby, it only allows maximum one concurrent session. If you are using Derby, can I suggest you to change that to a remote metastore?
For example, Configure a SQL Server Database as Remote Hive Metastore - Kontext.
And also, assuming you already created the symbolic link as mentioned in the article?
Hi,
From the screenshot, I could not identify any issues.
Are you using derby as metastore or external database like SQL Server or MySQL as metastore?
For Derby, it only allows maximum one concurrent session. If you are using Derby, can I suggest you to change that to a remote metastore?
For example, Configure a SQL Server Database as Remote Hive Metastore - Kontext.
And also, assuming you already created the symbolic link as mentioned in the article?
person R4F43L access_time 4 years ago
Hi, thanks a lot for this howto !
I have an error when i start hiveserver2 can someone help me please ?
WARN [main] server.HiveServer2 (HiveServer2.java:startHiveServer2(1064)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms
java.lang.RuntimeException: Error applying authorization policy on hive configuration: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hive.service.cli.CLIService.init(CLIService.java:118)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:230)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1036)
at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1305)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:932)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1893)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:964)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:924)
... 15 more
Hi, thanks a lot for this howto !
I have an error when i start hiveserver2 can someone help me please ?
WARN [main] server.HiveServer2 (HiveServer2.java:startHiveServer2(1064)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms
java.lang.RuntimeException: Error applying authorization policy on hive configuration: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hive.service.cli.CLIService.init(CLIService.java:118)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:230)
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1036)
at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1305)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:932)
at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1893)
at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:115)
... 12 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:964)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:924)
... 15 more
Did you configure a remote metastore for Hive? From the logs, it seems you are using derby as metastore? I recommend using SQL Server, MySQL or PostgreSQL as metadata store.
person Naseemuddin access_time 4 years ago
Hi Raymond, I was able to follow your instructions until the creation of the table. The insert step, however, does not seem to be working. It gets stuck with this output at the end
Starting Job = job_1615234468216_0003, Tracking URL = http://ADV075:8088/proxy/application_1615234468216_0003/
2021-03-08 21:46:27,779 INFO [34a2b056-4193-4c1f-9363-5117d5aa0607 main] exec.Task (SessionState.java:printInfo(1227)) - Starting Job = job_1615234468216_0003, Tracking URL = http://ADV075:8088/proxy/application_1615234468216_0003/
Kill Command = C:\hadoop-3.3.0\bin\mapred job -kill job_1615234468216_0003
2021-03-08 21:46:27,780 INFO [34a2b056-4193-4c1f-9363-5117d5aa0607 main] exec.Task (SessionState.java:printInfo(1227)) - Kill Command = C:\hadoop-3.3.0\bin\mapred job -kill job_1615234468216_0003
Also, I have been trying to use Spark on Hive tables. Here I get this output:
>>> spark.sql("show databases")
21/03/08 21:51:09 WARN NativeIO: NativeIO.getStat error (3): Das System kann den angegebenen Pfad nicht finden.
-- file path: tmp/hive
21/03/08 21:51:10 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/03/08 21:51:10 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/03/08 21:51:13 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/03/08 21:51:13 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN@192.168.178.60
21/03/08 21:51:13 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
DataFrame[namespace: string]Can you help me with that?
Hi Raymond, I was able to follow your instructions until the creation of the table. The insert step, however, does not seem to be working. It gets stuck with this output at the end
Starting Job = job_1615234468216_0003, Tracking URL = http://ADV075:8088/proxy/application_1615234468216_0003/
2021-03-08 21:46:27,779 INFO [34a2b056-4193-4c1f-9363-5117d5aa0607 main] exec.Task (SessionState.java:printInfo(1227)) - Starting Job = job_1615234468216_0003, Tracking URL = http://ADV075:8088/proxy/application_1615234468216_0003/
Kill Command = C:\hadoop-3.3.0\bin\mapred job -kill job_1615234468216_0003
2021-03-08 21:46:27,780 INFO [34a2b056-4193-4c1f-9363-5117d5aa0607 main] exec.Task (SessionState.java:printInfo(1227)) - Kill Command = C:\hadoop-3.3.0\bin\mapred job -kill job_1615234468216_0003
For Cygwin installation itself, please refer to its official documentation.
About editing user profile via Cygwin terminal, you should see something like the following screenshot where then you need to press Insert to start inserting new lines. If you re not familiar with vi editor and its command, refer to An introduction to the vi editor
Install Hive natively on Windows is not easy and you need to follow exactly all the steps I mentioned in the article. I would suggest you just use our WSL guide to install it: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux