By using this site, you acknowledge that you have read and understand our Cookie and Privacy policy. Your use of Kontext website is subject to this policy. Accept

Analytics & BI

Data Analytics,Big Data,Data Storage and Business Intelligence.

Subscribe

teradata python

Connect to Teradata database through Python

4633 views   3 comments last modified about 2 years ago

Teradata published an official Python module which can be used in DevOps projects. More details can be found at the following GitHub site: https://github.com/Teradata/PyTd Install Teradata module ...

View detail
python lite-log spark pyspark

Debug PySpark Code in Visual Studio Code

21 views   0 comments last modified about 16 days ago

The page summarizes the steps required to run and debug PySpark (Spark for Python) in Visual Studio Code. Install Python and pip Install Python from the official website: https://...

View detail
python spark pyspark

Implement SCD Type 2 Full Merge via Spark Data Frames

307 views   0 comments last modified about 2 months ago

Overview For SQL developers that are familiar with SCD and merge statements, you may wonder how to implement the same in big data platforms, considering database or storages in Hadoop are not designed/optimised for record level updates and inserts. In this post, I’m going to demons...

View detail
lite-log hadoop sqoop

Password Security Solution for Sqoop

37 views   0 comments last modified about 3 months ago

In Sqoop, there are multiple approaches to pass in passwords for RDBMS. Options Option 1 - clear password through --password argument sqoop [subcommand] --username user --password pwd This is the weakest approach as password is exposed directly...

View detail
python spark

PySpark: Convert JSON String Column to Array of Object (StructType) in Data Frame

421 views   0 comments last modified about 3 months ago

This post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...

View detail
java bigquery gcp dataflow gcs

Load CSV File from Google Cloud Storage to BigQuery Using Dataflow

1713 views   0 comments last modified about 7 months ago

This page documents the detailed steps to load CSV file from GCS into BigQuery using Dataflow to demo a simple data flow creation using Dataflow Tools for Eclipse. However it doesn’t necessarily mean this is the right use case for DataFlow. Alternatively ...

View detail
azure power-bi

Advanced analytics on big data with Azure - Tutorial

495 views   0 comments last modified about 8 months ago

Microsoft Azure provides a number of data analytics related products and services. It allows users to tailor the solutions to meet different requirements, for example, architecture for modern data warehouse, advanced analytics with big data or real time analytics. The following diagram sho...

View detail
power-bi bigquery

Use Google Cloud BigQuery as Data Source in Power BI

1119 views   0 comments last modified about 9 months ago

BigQuery is Google’s serverless data warehouse in Google Cloud. Power BI can consume data from various sources including RDBMS, NoSQL, Could, Services, etc. It is also easy to get data from BigQuery in Power BI. In this article, I am going to demonstrate how to connect to BigQuery to create...

View detail
zeppelin spark

Install Zeppelin 0.7.3 in Windows

2456 views   6 comments last modified about 2 years ago

This post summarizes the steps to install Zeppelin 0.7.3 in Windows environment. Tools and Environment GIT Bash Command Prompt Windows 10 Download Binary Package Download the latest binary package from the following website: ...

View detail
hadoop yarn hdfs

Install Hadoop 3.0.0 in Windows (Single Node)

12863 views   14 comments last modified about 2 years ago

This page summarizes the steps to install Hadoop 3.0.0 in your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...

View detail

Contacts

  • enquiry[at]kontext.tech

Subscribe