Tag - hadoop

lite-log hadoop sqoop

Password Security Solution for Sqoop

79   0   about 8 months ago

In Sqoop, there are multiple approaches to pass in passwords for RDBMS. Options Option 1 - clear password through --password argument sqoop [subcommand] --username user --password pwd This is the weakest approach as password is exposed directly...

View detail
lite-log hadoop hdfs

Resolve Hadoop RemoteException - Name node is in safe mode

294   0   about 2 years ago

In Safe Mode, the HDFS cluster is read-only. After completion of block replication maintenance activity, the name node leaves safe mode automatically. If you try to delete files in safe mode, the following exception may raise: org.apache.hadoop.ipc.RemoteException(org.apac...

View detail
hadoop hdfs parquet sqoop

Configure Sqoop in a Edge Node of Hadoop Cluster

1,598   0   about 2 years ago

This page continues with the following documentation about configuring a Hadoop multi-nodes cluster via adding a new edge node to configure administration or client tools. ...

View detail
hadoop yarn

Configure YARN and MapReduce Resources in Hadoop Cluster

1,602   0   about 2 years ago

When configuring YARN and MapReduce in Hadoop cluster, it is very important to configure the memory and virtual processors correctly. If the configurations are incorrect, the nodes may not be able to start properly and the applications may not be able to run successfully. For example...

View detail
hadoop yarn hdfs

Configure Hadoop 3.1.0 in a Multi Node Cluster

4,977   0   about 2 years ago

Previously, I summarized the steps to install Hadoop in a single node Windows machine. Install Hadoop 3.0.0 in Windows (Single Node) In this page, ...

View detail
hadoop yarn hdfs

Default Ports Used by Hadoop Services (HDFS, MapReduce, YARN)

4,948   0   about 2 years ago

This page summarizes the default ports used by Hadoop services. It is useful when configuring network interfaces in a cluster. Hadoop 3.1.0 HDFS The secondary namenode http/https server address and port. ...

View detail
zeppelin spark hadoop rdd

Read Text File from Hadoop in Zeppelin through Spark Context

4,367   0   about 2 years ago

Background This page provides an example to load text file from HDFS through SparkContext in Zeppelin (sc). Reference The details about this method can be found at: SparkContext.textFile ...

View detail
ssis hadoop hdfs

Use Hadoop File System Task in SSIS to Write File into HDFS

2,280   0   about 2 years ago

Context SQL Server Integration Service ( SSIS ) has tasks to perform operations against Hadoop, for example: Hadoop File System Task Hadoop Hive Task Hadoop Pig Task In Data Flow Task, you can also use: Hadoop HDFS Source ...

View detail
lite-log hadoop hdfs

Invoke Hadoop WebHDFS APIs in .NET Core

1,078   0   about 2 years ago

Background Apache doesn't provide native official .NET APIs for Hadoop HDFS. The HTTP REST API supports the complete FileSystem / ...

View detail