Compile and Build Hadoop 3.2.1 on Windows 10 Guide
Hi Raymond,
I' trying to build hadoop 3.2.2 wintutils.
The build fails when building Apache Hadoop HDFS Native Client (hadoop-hdfs-native-client).
Reason: Compiler issue when compiling jni_helper.c. Header 'dirent.h' cannot be found.
[exec] ..\..\..\..\..\src\main\native\libhdfs\jni_helper.c(28): fatal error C1083: Cannot open include file: 'dirent.h':
No such file or directory [D:\hcp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-clet\target\native\main\native\libhdfs-tests\native_mini_dfs.vcxproj]
The environment is setup as you've decribed in your blog (MS Visual Studio 2010, Win SDK 8.1, ...)
Any idea what's going wrong?
Thank you very much.
Best regards,
Juergen
The screenshot is the info ones but not the error messages. The error messages appears before the summary of build result for each package. The texts will have something like ERROR in it.
person Alexander access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
HI Raymond,
Sure. I followed all the steps for 3.2.1 - and I got everything right up until the Maven build part - then I am getting this error message which I have been getting since the very beginning (I am building Hadoop from another guide at first - https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48aef
so I am trying many different things now. Also 3.3.0 through https://chocolatey.org/packages/hadoop and also your 3.3.0 guide.
Is this enough?
EDIT: I followed your 3.2.1 and 3..3.0 guides and also the compile and build 3.2.1 the very detailed one.
HI Raymond,
Sure. I followed all the steps for 3.2.1 - and I got everything right up until the Maven build part - then I am getting this error message which I have been getting since the very beginning (I am building Hadoop from another guide at first - https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48aef
so I am trying many different things now. Also 3.3.0 through https://chocolatey.org/packages/hadoop and also your 3.3.0 guide.
Is this enough?
EDIT: I followed your 3.2.1 and 3..3.0 guides and also the compile and build 3.2.1 the very detailed one.
person Raymond access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
The screenshot you provided doesn't include the detailed error message. Can you post it here?
BTW, could you please confirm whether you have followed all the exact steps in my article? It is not very easy to build Hadoop in Windows 10 and I only found the steps I provided are working when building Hadoop 3.2.1.
The screenshot you provided doesn't include the detailed error message. Can you post it here?
BTW, could you please confirm whether you have followed all the exact steps in my article? It is not very easy to build Hadoop in Windows 10 and I only found the steps I provided are working when building Hadoop 3.2.1.
person Alexander access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
i am getting the hadoop-common issue, but relinking it doesn't solve it..
i am getting the hadoop-common issue, but relinking it doesn't solve it..
I've replied to you in the CMake post. You need to install Microsoft vcpkg (Visual C++ package manager) to install the openssl package into your Windows system to meet the requirements of Hadoop HDFS C/C++ native client project. The GitHub documentation provides very detailed steps about how to install it. Let me know if you are still not clear and I can share a detailed guide about it.
person Hitendra access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
Hi,
Getting following error message !
main:
[exec] CMake Warning (dev) in CMakeLists.txt:
[exec] No project() command is present. The top-level CMakeLists.txt file must
[exec] contain a li-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.18363.
[exec] teral, direct call to the project() command. Add a line of
[exec] code such as
[exec]
[exec] project(ProjectName)
[exec]
[exec] near the top of the file, but after cmake_minimum_required().
[exec]
[exec] CMake is pretending there is a "project(Project)" command on the first
[exec] line.
[exec] This warning is for project developers. Use -Wno-dev to suppress it.
[exec]
[exec] -- The C compiler identification is MSVC 19.28.29336.0
[exec] -- The CXX compiler identification is MSVC 19.28.29336.0
[exec] -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
[exec] -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe -- works
[exec] -- Detecting C compiler ABI info
[exec] -- Detecting C compiler ABI info - done
[exec] -- Detecting C compile features
[exec] -- Detecting C compile features - done
[exec] -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
[exec] -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe -- works
[exec] -- Detecting CXX compiler ABI info
[exec] -- Detecting CXX compiler ABI info - done
[exec] -- Detecting CXX compile features
[exec] -- Detecting CXX compile features - done
[exec] CUSTOM_OPENSSL_PREFIX =
[exec] Cannot find a usable OpenSSL library. OPENSSL_LIBRARY=OPENSSL_LIBRARY-NOTFOUND, OPENSSL_INCLUDE_DIR=C:/Bigdata/openssl-1.1.1i/include, CUSTOM_OPENSSL_LIB=, CUSTOM_OPENSSL_PREFIX=, CUSTOM_OPENSSL_INCLUDE=
[exec] -- Performing Test THREAD_LOCAL_SUPPORTED
[exec] -- Performing Test THREAD_LOCAL_SUPPORTED - Success
[exec] -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
[exec] CMake Error at C:/Bigdata/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
[exec] OPENSSL_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec] C:/Bigdata/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
[exec] C:/Bigdata/CMake/share/cmake-3.16/Modules/FindOpenSSL.cmake:449 (find_package_handle_stan-- Configuring incomplete, errors occurred!
[exec] See also "C:/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/CMak
[exec] dard_args)
[exec] main/native/libhdfspp/CMakeLists.txt:44 (find_package)
[exec]
[exec]
[exec] eFiles/CMakeOutput.log".
[exec] See also "C:/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/CMakeFiles/CMakeError.log".
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.2.1:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.916 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 4.382 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.207 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 0.918 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.309 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.061 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.690 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.326 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 1.309 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.440 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 38.920 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 1.912 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 1.998 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.132 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 6.354 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 13.608 s]
[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [ 12.555 s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Yet Another Learning Platform ...................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:35 min
[INFO] Finished at: 2021-01-16T15:08:15+11:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 5:123 in C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hadoop-hdfs-native-client
C:\hadoop>
Hi,
Getting following error message !
main:
[exec] CMake Warning (dev) in CMakeLists.txt:
[exec] No project() command is present. The top-level CMakeLists.txt file must
[exec] contain a li-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.18363.
[exec] teral, direct call to the project() command. Add a line of
[exec] code such as
[exec]
[exec] project(ProjectName)
[exec]
[exec] near the top of the file, but after cmake_minimum_required().
[exec]
[exec] CMake is pretending there is a "project(Project)" command on the first
[exec] line.
[exec] This warning is for project developers. Use -Wno-dev to suppress it.
[exec]
[exec] -- The C compiler identification is MSVC 19.28.29336.0
[exec] -- The CXX compiler identification is MSVC 19.28.29336.0
[exec] -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
[exec] -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe -- works
[exec] -- Detecting C compiler ABI info
[exec] -- Detecting C compiler ABI info - done
[exec] -- Detecting C compile features
[exec] -- Detecting C compile features - done
[exec] -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
[exec] -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe -- works
[exec] -- Detecting CXX compiler ABI info
[exec] -- Detecting CXX compiler ABI info - done
[exec] -- Detecting CXX compile features
[exec] -- Detecting CXX compile features - done
[exec] CUSTOM_OPENSSL_PREFIX =
[exec] Cannot find a usable OpenSSL library. OPENSSL_LIBRARY=OPENSSL_LIBRARY-NOTFOUND, OPENSSL_INCLUDE_DIR=C:/Bigdata/openssl-1.1.1i/include, CUSTOM_OPENSSL_LIB=, CUSTOM_OPENSSL_PREFIX=, CUSTOM_OPENSSL_INCLUDE=
[exec] -- Performing Test THREAD_LOCAL_SUPPORTED
[exec] -- Performing Test THREAD_LOCAL_SUPPORTED - Success
[exec] -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
[exec] CMake Error at C:/Bigdata/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
[exec] OPENSSL_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec] C:/Bigdata/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
[exec] C:/Bigdata/CMake/share/cmake-3.16/Modules/FindOpenSSL.cmake:449 (find_package_handle_stan-- Configuring incomplete, errors occurred!
[exec] See also "C:/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/CMak
[exec] dard_args)
[exec] main/native/libhdfspp/CMakeLists.txt:44 (find_package)
[exec]
[exec]
[exec] eFiles/CMakeOutput.log".
[exec] See also "C:/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/CMakeFiles/CMakeError.log".
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.2.1:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.916 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 4.382 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.207 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 0.918 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.309 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.061 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.690 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.326 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 1.309 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.440 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 38.920 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 1.912 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 1.998 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.132 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 6.354 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 13.608 s]
[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [ 12.555 s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Yet Another Learning Platform ...................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:35 min
[INFO] Finished at: 2021-01-16T15:08:15+11:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 5:123 in C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hadoop-hdfs-native-client
C:\hadoop>
I tested all the above steps in a shiny new Windows 10 virtual machine without any Maven packages installed thus I don’t think the step you mentioned is necessary.
Can you please confirm the following?
- Building the same version Hadoop as mentioned?
- Followed exact steps in this article? In the past, a few users didn’t follow all the instructions or were building a different version Hadoop.
- Can view all the projects built successfully as the following screenshot shows? And also in each project folder there is also successfully built artifacts?
person Jin access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
Thank you for your reply! I ran mvn clean (if that's what you meant by cleaning up the generated files) and then the code you gave (with the -X switch) and this time the build was successful (so, no detailed error log popped up that could help manage this issue) but it seems the build still isn't there in hadoop-dist/target.
I followed your steps to the T, though the only other thing that popped up was during the build initially I had a similar error ([ERROR]: Failed to execute goal org.codehaus.mojo:exec-maven-plugin) with hadoop-dist which i addressed as described in this stackoverflow, i.e., I pasted the following code into the POM with hadoop-dist.
<plugin><groupId>org.codehaus.mojo</groupId><artifactId>exec-maven-plugin</artifactId><version>1.3.1</version></plugin>Could this be why it's not working still? Thank you again for your help! This whole process is all very new to me...
Thank you for your reply! I ran mvn clean (if that's what you meant by cleaning up the generated files) and then the code you gave (with the -X switch) and this time the build was successful (so, no detailed error log popped up that could help manage this issue) but it seems the build still isn't there in hadoop-dist/target.
I followed your steps to the T, though the only other thing that popped up was during the build initially I had a similar error ([ERROR]: Failed to execute goal org.codehaus.mojo:exec-maven-plugin) with hadoop-dist which i addressed as described in this stackoverflow, i.e., I pasted the following code into the POM with hadoop-dist.
<plugin><groupId>org.codehaus.mojo</groupId><artifactId>exec-maven-plugin</artifactId><version>1.3.1</version></plugin>Could this be why it's not working still? Thank you again for your help! This whole process is all very new to me...
person Raymond access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
Hi Jin,
I don't think Maven clean is the approach to resolve a build problem.
mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true -X
Hi Juergen,
This is because that header file dirent.h doesn't exist in Visual C++ though it exists on Linux.
You can download that header file from this repository and place it into your Visual ++ include folder. For example, C:\Program Files (x86)\Microsoft Visual Studio XXX\VC\include. Replace XXX with your Visual Studio version.
GitHub - tronkko/dirent: C/C++ library for retrieving information on files and directories
person Juergen access_time 3 years ago
Re: Compile and Build Hadoop 3.2.1 on Windows 10 Guide
Hi Raymond,
I' trying to build hadoop 3.2.2 wintutils.
The build fails when building Apache Hadoop HDFS Native Client (hadoop-hdfs-native-client).
Reason: Compiler issue when compiling jni_helper.c. Header 'dirent.h' cannot be found.
[exec] ..\..\..\..\..\src\main\native\libhdfs\jni_helper.c(28): fatal error C1083: Cannot open include file: 'dirent.h':
No such file or directory [D:\hcp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-clet\target\native\main\native\libhdfs-tests\native_mini_dfs.vcxproj]
The environment is setup as you've decribed in your blog (MS Visual Studio 2010, Win SDK 8.1, ...)
Any idea what's going wrong?
Thank you very much.
Best regards,
Juergen