Skip to content

Add SQLFetchScroll support for fetchall() and fetchmany() - Fixes #1012#1059

Merged
bimalkjha merged 1 commit intoibmdb:masterfrom
bchoudhary6415:ibmdb_sqlFetchScroll
May 7, 2026
Merged

Add SQLFetchScroll support for fetchall() and fetchmany() - Fixes #1012#1059
bimalkjha merged 1 commit intoibmdb:masterfrom
bchoudhary6415:ibmdb_sqlFetchScroll

Conversation

@bchoudhary6415
Copy link
Copy Markdown
Collaborator

Added rowset-based fetching using SQLFetchScroll(SQL_FETCH_NEXT) for ibm_db.fetchall() and ibm_db.fetchmany() APIs. This replaces the previous row-by-row SQLFetch() approach, improving performance by fetching multiple rows per ODBC/CLI call using array-bound column buffers.

Fixes #1012


What's New

APIs to support SQLFetchScroll

  • ibm_db.fetchall(stmt) — Fetches all remaining rows from the result set as a list of tuples.
  • ibm_db.fetchmany(stmt, size) — Fetches up to size rows from the result set as a list of tuples.

Both APIs also work through the DBI layer: cursor.fetchall() and cursor.fetchmany(size).

SQL_ATTR_ROW_ARRAY_SIZE Support

A new statement attribute SQL_ATTR_ROW_ARRAY_SIZE controls how many rows are fetched per SQLFetchScroll call (i.e., the rowset size).

Feature Details
Default value 1000
Minimum 1
Maximum Automatically capped if total buffer exceeds 64 MB
Set via ibm_db.set_option(stmt, {ibm_db.SQL_ATTR_ROW_ARRAY_SIZE: N}, 0)
Get via ibm_db.get_option(stmt, ibm_db.SQL_ATTR_ROW_ARRAY_SIZE, 0)

How it works:

  • SQL_ATTR_ROW_ARRAY_SIZE tells the ODBC driver how many rows to fetch in a single SQLFetchScroll call.
  • The driver binds column buffers large enough to hold N rows, then fetches N rows at once into those buffers.
  • Fewer round-trips to the CLI layer = better performance, especially for large result sets.
  • If not set by the user, defaults to 1000 rows per fetch call.

Example:

import ibm_db

conn = ibm_db.connect(conn_str, "", "")
stmt = ibm_db.exec_immediate(conn, "SELECT * FROM LARGE_TABLE")

# Use default rowset size (1000)
rows = ibm_db.fetchall(stmt)

# Or set custom rowset size for fine-tuning
stmt2 = ibm_db.exec_immediate(conn, "SELECT * FROM LARGE_TABLE")
ibm_db.set_option(stmt2, {ibm_db.SQL_ATTR_ROW_ARRAY_SIZE: 5000}, 0)
rows = ibm_db.fetchall(stmt2)

# fetchmany - fetch in batches
stmt3 = ibm_db.exec_immediate(conn, "SELECT * FROM LARGE_TABLE")
batch = ibm_db.fetchmany(stmt3, 100)  # fetch 100 rows  

ibmdb_tests.py Fix

Fixed ValueError: Empty module name that occurred when running test files with SINGLE_PYTHON_TEST environment variable set to blank/empty. The load_tests() function now filters out empty module names before attempting import:
tests = [ t for t in tests if t ] # Filter out any empty module names

Note: LOB Column Handling

When any column in the result set is a LOB type (BLOB, CLOB, DBCLOB, XML), the fetch automatically falls back to single-row SQLFetch() + SQLGetData(). This is because:
LOB columns cannot be array-bound with SQLBindCol()
SQLGetData() is required for LOB data retrieval and only works on the current row
This fallback is transparent to the user — fetchall() and fetchmany() return the same results regardless of whether the table contains LOB columns.

…() APIs

Signed-off-by: Balram Choudhary <bchoudhary@rocketsoftware.com>
@bchoudhary6415 bchoudhary6415 requested a review from bimalkjha May 7, 2026 06:00
@bchoudhary6415 bchoudhary6415 self-assigned this May 7, 2026
@bimalkjha bimalkjha merged commit e592324 into ibmdb:master May 7, 2026
8 checks passed
@bchoudhary6415 bchoudhary6415 deleted the ibmdb_sqlFetchScroll branch May 7, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for SqlFetchScroll()/SQLExtendFetch() to query large rowsets

2 participants