Scenario Recently I created an Oozie workflow which contains one Spark action. The Spark action master is yarn and deploy mode is cluster. Each time when the job runs about 30 minutes, the application fails with errors like the following: Application applicatio...
View detailIn my previous post , I showed how to configure a single node Hadoop instance on Windows 10. The steps are not too difficult to follow if you have Java programming backgr...
View detailWhen creating Spark date frame using schemas, you may encounter errors about “field **: **Type can not accept object ** in type <class '*'>”. The actual error can vary, for instances, the following are some examples: field xxx: BooleanType can not accept object 100 in type ...
View detailIn Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...
View detailIn my article Connect to Teradata database through Python , I demonstrated about how to use Teradata python package or Teradata ODBC driver to connect to Teradata. In this article, I’m going to...
View detailIn one of my previous articles about Password Security Solution for Sqoop , I mentioned creating credential using hadoop credential command. The credentials are stored in JavaKey...
View detailThis page summarizes the installation guides about big data tools on Windows through Windows Subsystem for Linux (WSL). ...
View detailThis page summarizes the steps required to install Apache Sqoop (v1.4.7) in Windows 10 environment via Windows Subsystem for Linux (WSL). Prerequisites If you have already installed Hadoop 3.2.0 in WSL, ignore the following steps as you don’t need to install it again. Follow&...
View detailThis pages summarizes the steps to install the latest version 2.4.3 of Apache Spark on Windows 10 via Windows Subsystem for Linux (WSL). Prerequisites Follow either of the following pages to install WSL in a system or non-system drive on your Windows 10. ...
View detailThis page summarizes the steps to install Zeppelin version 0.7.3 on Windows 10 via Windows Subsystem for Linux (WSL). Version 0.8.1 When running Zeppelin in Ubuntu, the server may pick up one host address that is not accessible, for example 169.254.148.100, and the the remote interprete...
View detailPreviously, I demonstrated how to configured Apache Hive 3.0.0 on Windows 10. Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide...
View detailSince Hive 3.x, new authentication feature for HiveServer2 client is added. When starting HiveServer2 service (Hive version 3.0.0), you may encounter errors like: ‘HiveServer2 metastore.RetryingMetaStoreClient: RetryingMetaStoreClient trying reconnect as [username] (auth:S...
View detailIn one of my previous post, I showed how to configure Apache Hive 3.0.0 in Windows 10. Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide ...
View detailThis page shows how to install Windows Subsystem for Linux (WSL) system on a non-system drive manually. Enable Windows Subsystem for Linux system feature Open PowerShell as Administrator and run the following command to enable WSL feature: Enable-WindowsOptionalFea...
View detailIn the past months, this website has been using the following Email address to delivery all the notification messages to the website users such as registration confirmation email, comment email and so on. no-reply[at]kontext.tech However, I recently found that...
View detailWith Microsoft SQL Server JDBC driver, you can connect to the database through SQL Server Authentication or Kerberos Authentication. This post summarizes the configurations required for each authentication method with coding examples. *NTLM block in the following diagram represents pure Jav...
View detailI’ve been following Mobius project for a while and have been waiting for this day. .NET for Apache Spark v0.1.0 was just published on 2019-04-25 on GitHub. It provides high performance APIs for programming Apache Spark applications with C# and F#. It is .NET Standard complaint and can run in Wind...
View detailThis post shows you how to connect to HiveServer2 via Hive JDBC driver in Java. *The way to connect to HiveServer1 is very similar though the driver names are different: Version Drive...
View detailSELECT is one of the most commonly used statements. In this tutorial, I will cover the following items: Two of the principal query clauses—FROM and SELECT Data Types Built-in functions CASE expressions and variations like ISNULL and COALESCE. * The functio...
View detailThis page summarizes the steps to install Hadoop 3.0.0 in your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...
View detailIn this article, I am going to introduce how to install Teradata Express in virtual machines in Windows. Download software 1) Download VMware Player for Windows 32-bit and 64-bit from the following link (version 6.0): ...
View detailIn traditional asp.net applications, Server.MapPath is commonly used to generate absolute path in the web server. However, this has been removed from ASP.NET Core. So what is the equivalent way of doing it?
View detailSQL Server Compact 4.0 (CE 4.0) is a free SQL Server embedded database ideal for building standalone and occasionally connected applications for mobile devices, desktops, Web clients and others. In one of my projects, I used it as the database for logging errors, which assumes the errors will onl...
View detailInfosphere DataStage is adopted as ETL (Extract, Transform, Load) tool in many Teradata based data warehousing projects. With the Teradata ODBC and .NET data providers, you can also use the BI tools from Microsoft, i.e. SSIS. In my previous post, I demonstrated how to install Teradata Tool...
View detailIn this page, I’m going to demonstrate how to write and read parquet files in Spark/Scala by using Spark SQLContext class. Reference What is parquet format? Go the following project site to understand more about parquet. ...
View detailSSIS (SQL Server Integration Service) provides a number of convenient tasks to enable data integration. Exporting data from database to Excel file is a common task in ETL (Extract, Transform, Load) projects. Constantly the users/customers may raise format request regarding the Excel extract. To g...
View detailThe identity system in ASP.NET has evolved over time. If you are using ASP.NET Core, you probably found User property is an instance of ClaimsPrincipal in Controller or Razor views. Thus to retrieve the information, you need to utilize the claims.
View detailOverview Recently, I have been working on an ETL framework to load various source data (i.e. files, SQL Server, Oracle and Teradata) into Teradata. Due to some limitations, Java was chosen as the implementation language though IBM Infosphere DataStage is available to use. DataStage has p...
View detailIssue Context After installed Visual Studio 2017 15.3 preview and .net core 2.0 preview SDK, I upgraded one of my existing asp.net core project to 2.0. The project was created using ‘dotnet new angular’ SPA template. I also upgraded all the client app packages to the latest. For exa...
View detailIn my previous posts about Querying Teradata and SQL Server, I logged into the virtual machine graphic interface to manage the database. However, I constantly found it is resource intensive as there is only 4GB memory in my laptop. Instead, I will use text mode to start the virtual machine and co...
View detailTeradata published an official Python module which can be used in DevOps projects. More details can be found at the following GitHub site: https://github.com/Teradata/PyTd Install Teradata module ...
View detailIn my previous post, I demonstrated how to write and read parquet files in Spark/Scala. The parquet file destination is a local folder. Write and Read Parquet Files in Spark/Scala In this page...
View detailContext This pages demonstrates how to convert string to java.util.Date in Spark via Scala. Prerequisites If you have not installed Spark, follow the page below to install it: ...
View detailThis post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...
View detailSQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...
View detailIn this post, I will demonstrate how to connect to Teradata database via .NET Data Provider for Teradata using C#. Prerequisites Install the .NET Data Provider for Teradata from the following link: ...
View detailabout 20 days ago
about 20 days ago
Hi, please try this repository https://github.com/steveloughran/winutils
Unfortunately, this is only updated to Hadoop 3.0.0.
I'm not sure whether it works for 3.2.0. You can give it a try. Please pay attention to the README.md file about the author's declaration.
about 20 days ago
You mention "native Hadoop executable (winutils.exe) is not included in the official Hadoop distribution and needs to be downloaded separately [or built locally]." Do you happen to know where this can be downloaded for 3.2.0?
about 20 days ago
Hi,
Thank you for your post.
Do you know how to customize de query in power query?
Couldn't find it in any documentation.
Thanks.
Joana Barbosa
about 24 days ago
If it doesn’t exist, you can create one using the template file in the same directory : hive-site.xml.template.
If the template file didn’t exist either, you can create this file directly. The root element for this XML file is configuration:
<configuration>
...
</configuration>
about 24 days ago
Hi Mate, we couldn't find file "$HIVE_HOME/conf/hive-site.xml" in 3.1.1 package. Alternatively tried other versions couldn't find same file there too.
Please let me know how do I get/fix it.
Many Thanks
about 2 months ago
Thanks. I also got the issue when I was writing the post. However I was able to resolve it by running the following two commands only:
sudo apt-get install ssh
sudo service ssh restart
Just in case other people cannot resolve the problem, I've updated the post to include the link to your comment so that they can follow that guide to resolve the connection issue.
about 2 months ago
about 2 months ago
Have you tried the steps I mentioned in the post?
sudo apt-get install ssh
sudo service ssh restart
I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. I cannot reproduce this issue in my environment, so it will be hard to say where it goes wrong in your environment.
There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs?
Can you also please try the same approach to allow ssh connections?
The websites won't start successfully until you resolve the ssh issue. So make sure you can ssh localhost first.
about 2 months ago
Have you tried the solution I mentioned in the post? I got the same issue when it is first installed but after the following commands, it work. And also make sure you stop and restart hadoop daemons.
sudo apt-get install ssh
sudo service ssh restart
I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs? Can you also use IPv4 addresses for localhost instead of the IPv6 one?
Can you try to add firewall rule to allow TCP traffic to ssh port 22?
about 2 months ago
I don't have SSIS in my current PC any more. However it should work.
Are there are any objects in your database? And also does your account have access to list the tables or views in the database?
about 2 months ago
ssh localhost
Connection closed by 127.0.0.1 port 22
http://localhost:9870/dfshealth.html#tab-overview not working
about 2 months ago
I installed ssh and restarted it. Now 'ssh localhost' just says 'Connection closed by ::1 port 22.'
about 2 months ago
about 3 months ago
about 3 months ago
This was a great and easy to follow post. THANK YOU!
about 3 months ago
I'm glad it helped. :) It took me a while when I migrated my website to .NET Core a few years ago.
about 3 months ago
Thanks... perfect! New to Core (2.2) ... I was wondering where Server.MapPath() went!
Hello,
I'm not sure whether I understand your questions correctly or not. Once your import the data using Import or Direct Query, you can then customise through Power Query Editor in Power BI:
https://docs.microsoft.com/en-us/power-query/power-query-quickstart-using-power-bi