Pandas Read from SQLite Database

access_time 6 months ago visibility1123 comment 0

In my previous posts, I showed how to use jaydebeapi or sqlite3 packages to read data from SQLite databases. The high level approach I followed are:

  • Create database connection
  • Create a cursor object via executing SQL SELECT command.
  • Fetch all the records via the cursor
  • Convert the returned list of records a pandas DataFrame object.

In fact, pandas framework provides APIs to directly read data from SQLite or other SQL databases. We just need to create a DBI connection. In fact, we both connections created via JDBC or sqlite3 can be directly used.

The following code snippets show you how to do that.

Create Pandas DataFrame using JayDeBeApi 

import jaydebeapi
import pandas as pd
database = "../example.sqlite"

conn = jaydebeapi.connect("org.sqlite.JDBC",
                          f"""jdbc:sqlite:{database}""",
                          None,
                          "sqlite-jdbc-3.30.1.jar")
df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()

The above code snippet use pandas.read_sql API to read data directly as a pandas dataframe.

The output looks like the following:

 python .\pandas-sqlite.py
    type      name  tbl_name  rootpage                                                sql
0  table  Customer  Customer         2  CREATE TABLE Customer (ID int, Name text, Age ...

Create Pandas DataFrame using sqlite3

Alternatively, we can also use the same function on the connection object created via sqlite3 API.

import sqlite3
import pandas as pd
database = "../example.sqlite"

conn = sqlite3.connect(database)

df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()

The outputs are exactly the same as the previous one.

Write data into SQLite database

We can use pandas.dataframe.to_sql function to write dataframe data into a table in SQLite or any other SQL databases such as Oracle, SQL Server, MySQL, Teradata, etc.

import sqlite3
import pandas as pd
database = "../example.sqlite"

conn = sqlite3.connect(database)
users = {'ID': [1, 2, 3], 'Value': ['A', 'B', 'C']}
df = pd.DataFrame(users, columns=['ID', 'Value'])
print(df)
df.to_sql(name='Users', con=conn)
df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()
The above code snippets creates a dataframe and then save it as table Users in the example.sqlite database.
Output:
python .\pandas-sqlite-sqlite3.py
   ID Value
0   1     A
1   2     B
2   3     C
    type            name  tbl_name  rootpage                                                sql
0  table        Customer  Customer         2  CREATE TABLE Customer (ID int, Name text, Age ...
1  table           Users     Users         3  CREATE TABLE "Users" (\n"index" INTEGER,\n  "I...
2  index  ix_Users_index     Users         4  CREATE INDEX "ix_Users_index"ON "Users" ("index")

As you can see, the table is created and also an index is created too. Read through the official documentation about why an index column is created. Let's me know if you cannot figure out the reason.

info Last modified by Administrator at 2 months ago copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Kontext Column

Created for everyone to publish data, programming and cloud related articles.
Follow three steps to create your columns.


Learn more arrow_forward

More from Kontext

local_offer python local_offer sqlite local_offer python-database

visibility 99
thumb_up 0
access_time 6 months ago

SQLite is one of the most commonly used embedded file databases. All the mainstream programming language/framework provides APIs to interact with SQLite database. In my previous article  SQLite in .NET Core with Entity Framework Core , code snippet is provided to interact with SQLite via ...

local_offer sqlite local_offer python local_offer Java local_offer python-database

visibility 176
thumb_up 0
access_time 6 months ago

To read data from SQLite database in Python, you can use the built-in sqlite3 package . Another approach is to use SQLite JDBC driver via  JayDeBeApi  python package. Download the JAR file from one of the online repositories: Maven Repository BitBucket or any other equivalent ...

Improve PySpark Performance using Pandas UDF with Apache Arrow

local_offer pyspark local_offer spark local_offer spark-2-x local_offer pandas local_offer spark-advanced

visibility 3381
thumb_up 4
access_time 10 months ago

Apache Arrow is an in-memory columnar data format that can be used in Spark to efficiently transfer data between JVM and Python processes. This currently is most beneficial to Python users that work with Pandas/NumPy data. In this article, I'm going to show you how to utilise Pandas UDF in ...

About column