Tag - hdfs

hadoop yarn hdfs

Install Hadoop 3.0.0 in Windows (Single Node)

18,385   21   about 2 years ago

This page summarizes the steps to install Hadoop 3.0.0 in your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...

View detail
lite-log hadoop hdfs

Copy Files from Hadoop HDFS to Local

53   0   about 4 months ago

Copy file from HDFS to local Use the following command: hadoop fs [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] For example, copy a file from /hdfs-file.txt in HDFS to local /tmp/ using the following command: ...

View detail
lite-log hadoop hdfs

Hadoop datanode issue and resolution - ‘Incompatible clusterIDs’

313   0   about 4 months ago

Issue After finishing installation Hadoop 3.0.0 in my Windows: Install Hadoop 3.0.0 in Windows (Single Node) , I got the following error after I formated the name node several ti...

View detail
lite-log hadoop hdfs

Resolve Hadoop RemoteException - Name node is in safe mode

278   0   about 2 years ago

In Safe Mode, the HDFS cluster is read-only. After completion of block replication maintenance activity, the name node leaves safe mode automatically. If you try to delete files in safe mode, the following exception may raise: org.apache.hadoop.ipc.RemoteException(org.apac...

View detail
hadoop hdfs parquet sqoop

Configure Sqoop in a Edge Node of Hadoop Cluster

1,458   0   about 2 years ago

This page continues with the following documentation about configuring a Hadoop multi-nodes cluster via adding a new edge node to configure administration or client tools. ...

View detail
hadoop yarn hdfs

Configure Hadoop 3.1.0 in a Multi Node Cluster

4,753   0   about 2 years ago

Previously, I summarized the steps to install Hadoop in a single node Windows machine. Install Hadoop 3.0.0 in Windows (Single Node) In this page, ...

View detail
hadoop yarn hdfs

Default Ports Used by Hadoop Services (HDFS, MapReduce, YARN)

4,368   0   about 2 years ago

This page summarizes the default ports used by Hadoop services. It is useful when configuring network interfaces in a cluster. Hadoop 3.1.0 HDFS The secondary namenode http/https server address and port. ...

View detail
sql server spark hdfs parquet sqoop

Load Data into HDFS from SQL Server via Sqoop

1,452   0   about 2 years ago

This page shows how to import data from SQL Server into Hadoop via Apache Sqoop. Prerequisites Please follow the link below to install Sqoop in your machine if you don’t have one environment ready. ...

View detail
lite-log spark hdfs scala parquet

Write and Read Parquet Files in HDFS through Spark/Scala

5,944   0   about 2 years ago

In my previous post, I demonstrated how to write and read parquet files in Spark/Scala. The parquet file destination is a local folder. Write and Read Parquet Files in Spark/Scala In this page...

View detail
ssis hadoop hdfs

Use Hadoop File System Task in SSIS to Write File into HDFS

2,023   0   about 2 years ago

Context SQL Server Integration Service ( SSIS ) has tasks to perform operations against Hadoop, for example: Hadoop File System Task Hadoop Hive Task Hadoop Pig Task In Data Flow Task, you can also use: Hadoop HDFS Source ...

View detail