Copy Files from Hadoop HDFS to Local

access_time 2 years ago visibility1886 comment 0

Copy file from HDFS to local

Use the following command:

hadoop fs [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]

For example, copy a file from /hdfs-file.txt in HDFS to local /tmp/ using the following command:

hadoop fs -copyToLocal /hdfs-file.txt /tmp/hdfs-file.txt

HDFS CLI

If you forgot any HDFS commands, you can use the following command to list all of them:

hadoop fs

The output will generate all the commands (not all of them are implemented yet):

hadoop fs
Usage: hadoop fs [generic options]
         [-appendToFile <localsrc> ... <dst>]
         [-cat [-ignoreCrc] <src> ...]
         [-checksum <src> ...]
         [-chgrp [-R] GROUP PATH...]
         [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
         [-chown [-R] [OWNER][:[GROUP]] PATH...]
         [-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>]
         [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
         [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] <path> ...]
         [-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
         [-createSnapshot <snapshotDir> [<snapshotName>]]
         [-deleteSnapshot <snapshotDir> <snapshotName>]
         [-df [-h] [<path> ...]]
         [-du [-s] [-h] [-v] [-x] <path> ...]
         [-expunge]
         [-find <path> ... <expression> ...]
         [-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
         [-getfacl [-R] <path>]
         [-getfattr [-R] {-n name | -d} [-e en] <path>]
         [-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
         [-help [cmd ...]]
         [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]]
         [-mkdir [-p] <path> ...]
         [-moveFromLocal <localsrc> ... <dst>]
         [-moveToLocal <src> <localdst>]
         [-mv <src> ... <dst>]
         [-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
         [-renameSnapshot <snapshotDir> <oldName> <newName>]
         [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
         [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
         [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
         [-setfattr {-n name [-v value] | -x name} <path>]
         [-setrep [-R] [-w] <rep> <path> ...]
         [-stat [format] <path> ...]
         [-tail [-f] <file>]
         [-test -[defsz] <path>]
         [-text [-ignoreCrc] <src> ...]
         [-touchz <path> ...]
         [-truncate [-w] <length> <path> ...]
         [-usage [cmd ...]]

Generic options supported are:
-conf <configuration file>        specify an application configuration file
-D <property=value>               define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port>  specify a ResourceManager
-files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines

The general command line syntax is:
command [genericOptions] [commandOptions]

If you want to view the detailed syntax for any command, you can try the following command:

hadoop fs -help [command]

For example, run command ‘hadoop fs -help copyToLocal’ will generate the following output:

hadoop fs -help copyToLocal
-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst> :
   Identical to the -get command.

info Last modified by Raymond 2 years ago copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Follow Kontext

Get our latest updates on LinkedIn or Twitter.

Want to publish your article on Kontext?

Learn more

More from Kontext

visibility 7486
thumb_up 0
access_time 3 years ago

This page provides an example to load text file from HDFS through SparkContext in Zeppelin (sc). The details about this method can be found at: https://spark.apache.org/docs/2.2.1/api/java/org/apache/spark/SparkContext.html#textFile-java.lang.String-int- ...

Apache Hive 3.1.2 Installation on Windows 10
visibility 1536
thumb_up 1
access_time 6 months ago

Hive 3.1.2 was released on 26th Aug 2019. It is still the latest 3.x release and works with Hadoop 3.x.y releases. In this article, I’m going to provide step by step instructions about installing Hive 3.1.2 on Windows 10. * Logos are registered trademarks of Apache Hive and Microsoft Windows.

Install Hadoop 3.2.1 on Windows 10 Step by Step Guide
visibility 24415
thumb_up 19
access_time 2 years ago

This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).