Pandas Read from SQLite Database

In my previous posts, I showed how to use jaydebeapi or sqlite3 packages to read data from SQLite databases. The high level approach I followed are:

• Create database connection
• Create a cursor object via executing SQL SELECT command.
• Fetch all the records via the cursor
• Convert the returned list of records a pandas DataFrame object.

In fact, pandas framework provides APIs to directly read data from SQLite or other SQL databases. We just need to create a DBI connection. In fact, we both connections created via JDBC or sqlite3 can be directly used.

The following code snippets show you how to do that.

Create Pandas DataFrame using JayDeBeApi 

import jaydebeapi
import pandas as pd
database = "../example.sqlite"

conn = jaydebeapi.connect("org.sqlite.JDBC",
                          f"""jdbc:sqlite:{database}""",
                          None,
                          "sqlite-jdbc-3.30.1.jar")
df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()

The above code snippet use pandas.read_sql API to read data directly as a pandas dataframe.

The output looks like the following:

 python .\pandas-sqlite.py
    type      name  tbl_name  rootpage                                                sql
0  table  Customer  Customer         2  CREATE TABLE Customer (ID int, Name text, Age ...

Create Pandas DataFrame using sqlite3

Alternatively, we can also use the same function on the connection object created via sqlite3 API.

import sqlite3
import pandas as pd
database = "../example.sqlite"

conn = sqlite3.connect(database)

df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()

The outputs are exactly the same as the previous one.

Write data into SQLite database

We can use pandas.dataframe.to_sql function to write dataframe data into a table in SQLite or any other SQL databases such as Oracle, SQL Server, MySQL, Teradata, etc.

import sqlite3
import pandas as pd
database = "../example.sqlite"

conn = sqlite3.connect(database)
users = {'ID': [1, 2, 3], 'Value': ['A', 'B', 'C']}
df = pd.DataFrame(users, columns=['ID', 'Value'])
print(df)
df.to_sql(name='Users', con=conn)
df = pd.read_sql("select * from sqlite_master", con=conn)
print(df)
conn.close()

python .\pandas-sqlite-sqlite3.py
   ID Value
0   1     A
1   2     B
2   3     C
    type            name  tbl_name  rootpage                                                sql
0  table        Customer  Customer         2  CREATE TABLE Customer (ID int, Name text, Age ...
1  table           Users     Users         3  CREATE TABLE "Users" (\n"index" INTEGER,\n  "I...
2  index  ix_Users_index     Users         4  CREATE INDEX "ix_Users_index"ON "Users" ("index")

As you can see, the table is created and also an index is created too. Read through the official documentation about why an index column is created. Let's me know if you cannot figure out the reason.