val sql="select count(*) from test_db.test_table;";
//sql: String = select count(*) from test_db.test_table;
val df = spark.sql(sql);
//df: org.apache.spark.sql.DataFrame = [count(1): bigint]
df.show()
/*
+--------+
|count(1)|
+--------+
| 5|
+--------+
*/
val df1 = spark.sql("select * from test_db.test_table;");
//df1: org.apache.spark.sql.DataFrame = [id: int, attr: string]
df1.groupBy().count()
//res5: org.apache.spark.sql.DataFrame = [count: bigint]
println(df1.groupBy().count())
//[count: bigint]
df1.groupBy().count().show()
/*+-----+
|count|
+-----+
| 5|
+-----+
*/
articleArticles 80
codeCode 45
imageDiagrams 20
descriptionNotebooks 0
chat_bubble_outlineThreads 1
commentComments 7
loyaltyKontext Points 1137
account_circle Profile
visibility 85
comment 0
access_time 8 months ago
language English
Find Number of Rows of Hive Table via Scala
To find the number of rows/records in a Hive table, we can use Spark SQL
count aggregation function: Hive SQL - Aggregate Functions Overview with Examples.This code snippet provide example of Scala code to implement the same. spark-shell is used directly for simplicity. The code snippet can also run Jupyter Notebooks or Zeppelin with Spark kernel. Alternatively, it can be compiled to jar file and then submit as job via spark-submit.
Code snippet
copyright
This page is subject to Site terms.
Log in with external accounts
comment Comments
No comments yet.
warning Please login first to view stats information.
