Read here about Kontext's Cookie and Privacy policy. Dismiss

Invoke Hadoop WebHDFS APIs in .NET Core

98 views 0 comments last modified about 3 months ago Raymond Tang

lite-log hadoop hdfs

Background

Apache doesn't provide native official .NET APIs for Hadoop HDFS. The HTTP REST API supports the complete FileSystem/FileContext interface for HDFS.

Thus, we could use these web APIs to perform HDFS operations in other programming language like C#.

WebHDFS APIs reference

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

Examples

List files

The following code snippet retrieve the file list in the root directory in my local Hadoop node.:

static void Main(string[] args)
         {
             WebHdfsListStatusApi();
             Console.ReadLine();
         }

        static void WebHdfsListStatusApi()
         {

            var protocal = "http";
             var host = "127.0.0.1";
             var port = 9870;
             var hdfsFilePath = "\\";
             var operation = "LISTSTATUS";
             var url = $"{protocal}://{host}:{port}/webhdfs/v1/{hdfsFilePath}?op={operation}";
             var request = (HttpWebRequest)WebRequest.Create(url);
             var response = (HttpWebResponse)request.GetResponse();
             using (StreamReader reader = new StreamReader(response.GetResponseStream()))
             {
                 var result = reader.ReadToEnd();
                 Console.WriteLine(result);
             }
         }

The output looks like the following screenshot:

image

The following is the output in Postman:

image

Get file content

Similarly you can also get the content of a file through OPEN operation:

static void Main(string[] args)
         {
             WebHdfsGetFileContent();
             Console.ReadLine();
         }

        static void WebHdfsGetFileContent()
         {

            var protocal = "http";
             var host = "127.0.0.1";
             var port = 9870;
             var hdfsFilePath = "\\Sales.csv";
             var operation = "OPEN";
             var url = $"{protocal}://{host}:{port}/webhdfs/v1/{hdfsFilePath}?op={operation}";
             var request = (HttpWebRequest)WebRequest.Create(url);
             var response = (HttpWebResponse)request.GetResponse();
             using (StreamReader reader = new StreamReader(response.GetResponseStream()))
             {
                 var result = reader.ReadToEnd();
                 Console.WriteLine(result);
             }
         }

The following screenshot is the sample output:

image

Related pages

Tuples in C# (4.x, 7.0, 7.1)

61 views   0 comments last modified about 3 months ago

What is a tuple? Tuple is an ordered sequence, fixed-size, immutable and of heterogeneous objects. Tuple has been existing in other languages such as F#, Python, Perl and etc. for a long time. It was first introduced into C# from C# 4.0 and has been evolving over time. Since C# 7.1, tuple...

View detail

Tuples in C# (4.x, 7.0, 7.1)

61 views   0 comments last modified about 3 months ago

What is a tuple? Tuple is an ordered sequence, fixed-size, immutable and of heterogeneous objects. Tuple has been existing in other languages such as F#, Python, Perl and etc. for a long time. It was first introduced into C# from C# 4.0 and has been evolving over time. Since C# 7.1, tuple...

View detail

C# version history and core features

64 views   0 comments last modified about 4 months ago

C# is my favourite programming language and has been evolving overtime. I have been programming with many other languages but C# is always my favourite because it is simple and easy to use. Visual Studio is the most powerful tool I’ve ever used. To me, programming efficiency and peformance are th...

View detail

Logging configuration in .NET core

184 views   0 comments last modified about 8 months ago

.NET core introduces a logging API that works with a number of logging frameworks. The built-in providers are configurable and extensible to support different level loggings.

View detail

Sending Emails in .NET Core Applications

1059 views   0 comments last modified about 8 months ago

Sending emails are common in applications. For example, when user registers, we need to send account activation emails. This post summarize the approaches we can use to send emails in .NET Core 1.x and 2.x.

View detail

Retrieve Http client request metadata like IP address and languages in asp.net core

591 views   0 comments last modified about 8 months ago

IP Address In ASP.NET Core, Request.UserHostAddress has been removed though that attribute exists in the traditional ASP.NET applications. We can use HttpContext.Connection to retrieve the remove client IP address: var ipAddress = HttpContext.Connecti...

View detail

Add comment

Please login first to add comments.  Log in New user?  Register

Comments (0)

No comments yet.