Skip to content

Fix TIMESTAMP(10..12) fetch performance issue(#1079) and drop Python 3.9/3.10 wheel builds #1080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bimalkjha merged 2 commits into from
Oct 20, 2026
Merged

Fix TIMESTAMP(10..12) fetch performance issue(#1079) and drop Python 3.9/3.10 wheel builds #1080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4cbc641
test workflow file to remove python 3.9 and python3.10
Earammak Oct 19, 2026
cc8acf2
Fix TIMESTAMP(10..12) fetch performance issue(#1079) and drop Python …
Earammak Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/bld_wheels_and_upload.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
uses: pypa/cibuildwheel@v4.2.0
env:
CIBW_BUILD: "*-win_amd64"
CIBW_SKIP: "*t-*"
CIBW_SKIP: "*t-* cp39-* cp310-* pp39-* pp310-*"
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "python -c \"import shutil,sys; shutil.copy2(sys.argv[1], sys.argv[2])\" {wheel} {dest_dir}"

- name: Inject ibm_db_dll.pth into wheels
Expand All @@ -48,7 +48,7 @@ jobs:
uses: pypa/cibuildwheel@v4.2.0
env:
CIBW_BUILD: "*-win32"
CIBW_SKIP: "*t-*"
CIBW_SKIP: "*t-* cp39-* cp310-* pp39-* pp310-*"
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "python -c \"import shutil,sys; shutil.copy2(sys.argv[1], sys.argv[2])\" {wheel} {dest_dir}"

- name: Inject ibm_db_dll.pth into wheels
Expand Down Expand Up @@ -77,7 +77,7 @@ jobs:
# so manylinux2014 (glibc 2.17) can no longer be used here.
CIBW_MANYLINUX_I686_IMAGE: manylinux_2_34
CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_34
CIBW_SKIP: "*t-* *-musllinux_* *-*linux_{aarch64,ppc64le}"
CIBW_SKIP: "*t-* cp39-* cp310-* pp39-* pp310-* *-musllinux_* *-*linux_{aarch64,ppc64le}"
CIBW_REPAIR_WHEEL_COMMAND_LINUX:
auditwheel repair
--disable-isa-ext-check
Expand Down Expand Up @@ -110,7 +110,7 @@ jobs:
- name: Build wheels
uses: pypa/cibuildwheel@v4.2.0
env:
CIBW_SKIP: "*t-*"
CIBW_SKIP: "*t-* cp39-* cp310-* pp39-* pp310-*"
MACOSX_DEPLOYMENT_TARGET: 14.0

- uses: actions/upload-artifact@v6
Expand All @@ -131,7 +131,7 @@ jobs:
uses: pypa/cibuildwheel@v4.2.0
env:
CIBW_ARCHS: "x86_64"
CIBW_SKIP: "*t-*"
CIBW_SKIP: "*t-* cp39-* cp310-* pp39-* pp310-*"
MACOSX_DEPLOYMENT_TARGET: 10.15

- uses: actions/upload-artifact@v6
Expand Down
12 changes: 9 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,18 @@ https://github.com/ibmdb/python-ibmdb/wiki/APIs

For the asyncio APIs, please refer to [ASYNC_APIs_WIKI.md](ASYNC_APIs_WIKI.md).

## Timestamp Precision Notes

- Python `datetime` stores microseconds, so `ibm_db` returns timestamp values with up to 6 fractional digits in Python objects.
- Fetching Db2 `TIMESTAMP(10..12)` values now uses an extended CLI timestamp bind path internally to avoid per-row `CLI0182W` (`SQLSTATE 01S07`) diagnostic overhead during fetch.
- Application-visible behavior remains the same: timestamp values are returned as Python `datetime.datetime`, with fractional precision mapped to microseconds.

<a name="prereq"></a>

## Pre-requisites

Install Python 3.9 <= 3.14. The minimum python version supported by driver is python 3.9 and the latest version supported is python 3.14.
MacOS arm64 is supported Python 3.9 onwards.
Install Python 3.11 <= 3.15. The minimum python version supported by driver is python 3.11 and the latest version supported is python 3.15.
MacOS arm64 is supported Python 3.11 onwards.

When using an external clidriver, **ibm_db_dbi** must be compiled against the headers and libraries of that clidriver to function properly.
Therefore, it is not recommended to use the precompiled `ibm_db` wheel with an external clidriver.
Expand Down Expand Up @@ -89,7 +95,7 @@ To install ibm_db from source code after clone from git, or to know the detialed

## Installation

- Python **Wheels** are built for Linux, MacOS and Windows operating systems for multiple python versions (from python version 3.7 to 3.13). For other platforms, package gets installed from source distribution. For MaxOS arm64, python wheels are available from version 3.9 onwards.
- Python **Wheels** are built for Linux, MacOS and Windows operating systems for multiple python versions (from python version 3.11 to 3.15). For other platforms, package gets installed from source distribution. For MaxOS arm64, python wheels are available from version 3.11 onwards.

You can install the driver using pip as:

Expand Down
85 changes: 85 additions & 0 deletions asyncio_testsuite/test_37_async_timestamp_precision_fetch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
from __future__ import print_function
import asyncio
import datetime
import unittest
import config
from ibm_db_dbi import AsyncConnection
from testfunctions import IbmDbTestFunctions


class IbmDbTestCase(unittest.TestCase):

def test_37_async_timestamp_precision_fetch(self):
obj = IbmDbTestFunctions()
obj.assert_expect(self.run_test_37)

def run_test_37(self):
async def main():
conn = await AsyncConnection.connect(
"DATABASE=%s;HOSTNAME=%s;PORT=%d;PROTOCOL=TCPIP;UID=%s;PWD=%s;" % (
config.database, config.hostname, config.port,
config.user, config.password),
'', '')
cursor = await conn.cursor()

for p in (6, 9, 10, 11, 12):
ts = "CAST('2024-01-02 03:04:05.123456789012' AS TIMESTAMP(%d))" % p

await cursor.execute("VALUES " + ts)
row = await cursor.fetchone()
print("%d fetchone %s %s" % (p, isinstance(row[0], datetime.datetime), row[0]))

await cursor.execute("VALUES (%s), (%s)" % (ts, ts))
rows = await cursor.fetchall()
print("%d fetchall %d %s" % (p, len(rows), all(r[0] == row[0] for r in rows)))

await cursor.close()
await conn.close()

asyncio.run(main())

#__END__
#__LUW_EXPECTED__
#6 fetchone True 2024-01-02 03:04:05.123456
#6 fetchall 2 True
#9 fetchone True 2024-01-02 03:04:05.123456
#9 fetchall 2 True
#10 fetchone True 2024-01-02 03:04:05.123456
#10 fetchall 2 True
#11 fetchone True 2024-01-02 03:04:05.123456
#11 fetchall 2 True
#12 fetchone True 2024-01-02 03:04:05.123456
#12 fetchall 2 True
#__ZOS_EXPECTED__
#6 fetchone True 2024-01-02 03:04:05.123456
#6 fetchall 2 True
#9 fetchone True 2024-01-02 03:04:05.123456
#9 fetchall 2 True
#10 fetchone True 2024-01-02 03:04:05.123456
#10 fetchall 2 True
#11 fetchone True 2024-01-02 03:04:05.123456
#11 fetchall 2 True
#12 fetchone True 2024-01-02 03:04:05.123456
#12 fetchall 2 True
#__SYSTEMI_EXPECTED__
#6 fetchone True 2024-01-02 03:04:05.123456
#6 fetchall 2 True
#9 fetchone True 2024-01-02 03:04:05.123456
#9 fetchall 2 True
#10 fetchone True 2024-01-02 03:04:05.123456
#10 fetchall 2 True
#11 fetchone True 2024-01-02 03:04:05.123456
#11 fetchall 2 True
#12 fetchone True 2024-01-02 03:04:05.123456
#12 fetchall 2 True
#__IDS_EXPECTED__
#6 fetchone True 2024-01-02 03:04:05.123456
#6 fetchall 2 True
#9 fetchone True 2024-01-02 03:04:05.123456
#9 fetchall 2 True
#10 fetchone True 2024-01-02 03:04:05.123456
#10 fetchall 2 True
#11 fetchone True 2024-01-02 03:04:05.123456
#11 fetchall 2 True
#12 fetchone True 2024-01-02 03:04:05.123456
#12 fetchall 2 True
108 changes: 81 additions & 27 deletions ibm_db.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,7 @@ typedef union
SQLREAL r_val;
SQLWCHAR *w_val;
TIMESTAMP_STRUCT *ts_val;
TIMESTAMP_STRUCT_EXT *ts_ext_val;
TIMESTAMP_STRUCT_EXT_TZ *tstz_val;
DATE_STRUCT *date_val;
TIME_STRUCT *time_val;
Expand Down Expand Up @@ -867,7 +868,15 @@ static void _python_ibm_db_free_result_struct(stmt_handle *handle)
}
break;
case SQL_TYPE_TIMESTAMP:
if (handle->row_data[i].data.ts_val != NULL)
if (handle->column_info[i].scale > 9)
{
if (handle->row_data[i].data.ts_ext_val != NULL)
{
PyMem_Del(handle->row_data[i].data.ts_ext_val);
handle->row_data[i].data.ts_ext_val = NULL;
}
}
else if (handle->row_data[i].data.ts_val != NULL)
{
snprintf(messageStr, sizeof(messageStr), "Freeing row_data[%d].data.ts_val=%p", i, handle->row_data[i].data.ts_val);
LogMsg(DEBUG, messageStr);
Expand Down Expand Up @@ -2038,21 +2047,37 @@ static int _python_ibm_db_bind_column_helper(stmt_handle *stmt_res)
case SQL_TYPE_TIMESTAMP:
snprintf(messageStr, sizeof(messageStr), "Case SQL_TYPE_TIMESTAMP for column index %d", i);
LogMsg(DEBUG, messageStr);
row_data->ts_val = ALLOC(TIMESTAMP_STRUCT);
if (row_data->ts_val == NULL)
SQLPOINTER timestamp_buffer;
SQLSMALLINT timestamp_c_type;
SQLLEN timestamp_buffer_size;
if (stmt_res->column_info[i].scale > 9)
{
row_data->ts_ext_val = ALLOC(TIMESTAMP_STRUCT_EXT);
timestamp_buffer = row_data->ts_ext_val;
timestamp_c_type = SQL_C_TIMESTAMP_EXT;
timestamp_buffer_size = sizeof(TIMESTAMP_STRUCT_EXT);
}
else
{
row_data->ts_val = ALLOC(TIMESTAMP_STRUCT);
timestamp_buffer = row_data->ts_val;
timestamp_c_type = SQL_C_TYPE_TIMESTAMP;
timestamp_buffer_size = sizeof(TIMESTAMP_STRUCT);
}
if (timestamp_buffer == NULL)
{
LogMsg(EXCEPTION, "Failed to Allocate Memory for ts_val");
LogMsg(EXCEPTION, "Failed to Allocate Memory for timestamp value");
PyErr_SetString(PyExc_Exception, "Failed to Allocate Memory");
return -1;
}
snprintf(messageStr, sizeof(messageStr), "Allocated memory for ts_val with size %zu", sizeof(TIMESTAMP_STRUCT));
snprintf(messageStr, sizeof(messageStr), "Allocated memory for timestamp value with size %zu", (size_t)timestamp_buffer_size);
LogMsg(DEBUG, messageStr);
snprintf(messageStr, sizeof(messageStr), "Calling SQLBindCol with parameters: hstmt=%p, col=%d, sql_type=SQL_C_TYPE_TIMESTAMP, buffer_size=%zu, out_length=%p",
stmt_res->hstmt, i + 1, sizeof(TIMESTAMP_STRUCT), &stmt_res->row_data[i].out_length);
snprintf(messageStr, sizeof(messageStr), "Calling SQLBindCol with parameters: hstmt=%p, col=%d, sql_type=%d, buffer_size=%zu, out_length=%p",
stmt_res->hstmt, i + 1, timestamp_c_type, (size_t)timestamp_buffer_size, &stmt_res->row_data[i].out_length);
LogMsg(DEBUG, messageStr);
Py_BEGIN_ALLOW_THREADS;
rc = SQLBindCol((SQLHSTMT)stmt_res->hstmt, (SQLUSMALLINT)(i + 1),
SQL_C_TYPE_TIMESTAMP, row_data->time_val, sizeof(TIMESTAMP_STRUCT),
timestamp_c_type, timestamp_buffer, timestamp_buffer_size,
(SQLINTEGER *)(&stmt_res->row_data[i].out_length));
Py_END_ALLOW_THREADS;
snprintf(messageStr, sizeof(messageStr), "SQLBindCol returned %d for column %d", rc, i);
Expand Down Expand Up @@ -2436,10 +2461,20 @@ static rowset_col_buffer *_python_ibm_db_bind_rowset_columns(
memset(bufs[i].data, 0, sizeof(TIME_STRUCT) * row_array_size);
break;
case SQL_TYPE_TIMESTAMP:
bufs[i].elem_size = sizeof(TIMESTAMP_STRUCT); bufs[i].ctype = SQL_C_TYPE_TIMESTAMP;
bufs[i].data = (void *)ALLOC_N(TIMESTAMP_STRUCT, row_array_size);
if (bufs[i].data == NULL) goto alloc_error;
memset(bufs[i].data, 0, sizeof(TIMESTAMP_STRUCT) * row_array_size);
if (stmt_res->column_info[i].scale > 9)
{
bufs[i].elem_size = sizeof(TIMESTAMP_STRUCT_EXT); bufs[i].ctype = SQL_C_TIMESTAMP_EXT;
bufs[i].data = (void *)ALLOC_N(TIMESTAMP_STRUCT_EXT, row_array_size);
if (bufs[i].data == NULL) goto alloc_error;
memset(bufs[i].data, 0, sizeof(TIMESTAMP_STRUCT_EXT) * row_array_size);
}
else
{
bufs[i].elem_size = sizeof(TIMESTAMP_STRUCT); bufs[i].ctype = SQL_C_TYPE_TIMESTAMP;
bufs[i].data = (void *)ALLOC_N(TIMESTAMP_STRUCT, row_array_size);
if (bufs[i].data == NULL) goto alloc_error;
memset(bufs[i].data, 0, sizeof(TIMESTAMP_STRUCT) * row_array_size);
}
break;
case SQL_TYPE_TIMESTAMP_WITH_TIMEZONE:
bufs[i].elem_size = sizeof(TIMESTAMP_STRUCT_EXT_TZ); bufs[i].ctype = SQL_C_TYPE_TIMESTAMP_EXT_TZ;
Expand Down Expand Up @@ -2805,9 +2840,10 @@ static PyObject *_python_ibm_db_build_row_from_rowset(
}
case SQL_TYPE_TIMESTAMP:
{
TIMESTAMP_STRUCT *tsv = (TIMESTAMP_STRUCT *)base_ptr;
value = PyDateTime_FromDateAndTime(tsv->year, tsv->month, tsv->day,
tsv->hour % 24, tsv->minute, tsv->second, tsv->fraction / 1000);
TIMESTAMP_STRUCT timestamp_value;
memcpy(&timestamp_value, base_ptr, sizeof(timestamp_value));
value = PyDateTime_FromDateAndTime(timestamp_value.year, timestamp_value.month, timestamp_value.day,
timestamp_value.hour % 24, timestamp_value.minute, timestamp_value.second, timestamp_value.fraction / 1000);
break;
}
case SQL_TYPE_TIMESTAMP_WITH_TIMEZONE:
Expand Down Expand Up @@ -3201,7 +3237,11 @@ static PyObject *_python_ibm_db_bind_fetch_rowset_helper(
if (stmt_res->row_data[i].data.tstz_val != NULL) { PyMem_Del(stmt_res->row_data[i].data.tstz_val); stmt_res->row_data[i].data.tstz_val = NULL; }
break;
case SQL_TYPE_TIMESTAMP:
if (stmt_res->row_data[i].data.ts_val != NULL) { PyMem_Del(stmt_res->row_data[i].data.ts_val); stmt_res->row_data[i].data.ts_val = NULL; }
if (stmt_res->column_info[i].scale > 9)
{
if (stmt_res->row_data[i].data.ts_ext_val != NULL) { PyMem_Del(stmt_res->row_data[i].data.ts_ext_val); stmt_res->row_data[i].data.ts_ext_val = NULL; }
}
else if (stmt_res->row_data[i].data.ts_val != NULL) { PyMem_Del(stmt_res->row_data[i].data.ts_val); stmt_res->row_data[i].data.ts_val = NULL; }
break;
case SQL_TYPE_DATE:
if (stmt_res->row_data[i].data.date_val != NULL) { PyMem_Del(stmt_res->row_data[i].data.date_val); stmt_res->row_data[i].data.date_val = NULL; }
Expand Down Expand Up @@ -15404,16 +15444,23 @@ static PyObject *_python_ibm_db_bind_fetch_helper(PyObject *args, int op)
break;

case SQL_TYPE_TIMESTAMP:
value = PyDateTime_FromDateAndTime(row_data->ts_val->year, row_data->ts_val->month, row_data->ts_val->day,
row_data->ts_val->hour % 24, row_data->ts_val->minute, row_data->ts_val->second,
row_data->ts_val->fraction / 1000);
{
TIMESTAMP_STRUCT timestamp_value;
if (stmt_res->column_info[column_number].scale > 9)
memcpy(&timestamp_value, row_data->ts_ext_val, sizeof(timestamp_value));
else
timestamp_value = *row_data->ts_val;
value = PyDateTime_FromDateAndTime(timestamp_value.year, timestamp_value.month, timestamp_value.day,
timestamp_value.hour % 24, timestamp_value.minute, timestamp_value.second,
timestamp_value.fraction / 1000);
snprintf(messageStr, sizeof(messageStr),
"Column data converted to timestamp: year=%d, month=%d, day=%d, hour=%d, minute=%d, second=%d, fraction=%d",
row_data->ts_val->year, row_data->ts_val->month, row_data->ts_val->day,
row_data->ts_val->hour % 24, row_data->ts_val->minute, row_data->ts_val->second,
row_data->ts_val->fraction / 1000);
timestamp_value.year, timestamp_value.month, timestamp_value.day,
timestamp_value.hour % 24, timestamp_value.minute, timestamp_value.second,
timestamp_value.fraction / 1000);
LogMsg(DEBUG, messageStr);
break;
}

case SQL_TYPE_TIMESTAMP_WITH_TIMEZONE:
value = format_timestamp_pystr(row_data->tstz_val);
Expand Down Expand Up @@ -20430,16 +20477,19 @@ static PyObject* ibm_db_fetch_callproc(PyObject* self, PyObject* args)

case SQL_TYPE_TIMESTAMP:
case SQL_TYPE_TIMESTAMP_WITH_TIMEZONE:{
const char *timestamp_type = curr->data_type == SQL_TYPE_TIMESTAMP_WITH_TIMEZONE
? "SQL_TYPE_TIMESTAMP_WITH_TIMEZONE" : "SQL_TYPE_TIMESTAMP";
if(curr->svalue){
LogMsg(DEBUG, "Processing SQL_TYPE_TIMESTAMP array element using svalue");
snprintf(messageStr, sizeof(messageStr), "Processing %s array element using svalue", timestamp_type);
LogMsg(DEBUG, messageStr);
char *elem_ptr = curr->svalue + i * curr->param_size;
size_t max_len = curr->param_size;
size_t actual_len;
#ifndef __MVS__
actual_len = strnlen(elem_ptr, max_len);
#endif
snprintf(messageStr, sizeof(messageStr),
"SQL_TYPE_TIMESTAMP svalue element %d: actual_len=%zu, max_len=%zu", i, actual_len, max_len);
"%s svalue element %d: actual_len=%zu, max_len=%zu", timestamp_type, i, actual_len, max_len);
LogMsg(DEBUG, messageStr);
elem = PyUnicode_Decode(elem_ptr, actual_len, "utf-8", "strict");
if (!elem) {
Expand All @@ -20464,7 +20514,8 @@ static PyObject* ibm_db_fetch_callproc(PyObject* self, PyObject* args)
}
}
else if (curr->uvalue) {
LogMsg(DEBUG, "Processing SQL_TYPE_TIMESTAMP/SQL_TIMESTAMP_WITH_TIMEZONE array element using uvalue");
snprintf(messageStr, sizeof(messageStr), "Processing %s array element using uvalue", timestamp_type);
LogMsg(DEBUG, messageStr);
SQLWCHAR *elem_ptr = (SQLWCHAR *)((char *)curr->uvalue + i * curr->param_size);
/* bind_indicator_array[i] holds the byte length of the output data;
* cap it to param_size (the buffer stride) to avoid over-read. */
Expand Down Expand Up @@ -20856,7 +20907,9 @@ static PyObject* ibm_db_fetch_callproc(PyObject* self, PyObject* args)
PyErr_Clear();
pyStr = PyBytes_FromStringAndSize((const char *)ts_wstr,ind_len * sizeof(SQLWCHAR));
}
snprintf(messageStr, sizeof(messageStr),"Scalar TIMESTAMP parameter using uvalue, length=%ld",(long)ind_len);
snprintf(messageStr, sizeof(messageStr), "Scalar %s parameter using uvalue, length=%ld",
curr->data_type == SQL_TYPE_TIMESTAMP_WITH_TIMEZONE ? "TIMESTAMP WITH TIME ZONE" : "TIMESTAMP",
(long)ind_len);
LogMsg(DEBUG, messageStr);
if (curr->data_type == SQL_TYPE_TIMESTAMP_WITH_TIMEZONE) {
/* Return as string to preserve timezone information */
Expand Down Expand Up @@ -20896,7 +20949,8 @@ static PyObject* ibm_db_fetch_callproc(PyObject* self, PyObject* args)
}
}
if (curr->ts_value) {
snprintf(messageStr, sizeof(messageStr), "Scalar timestamp_value = %04d-%02d-%02d %02d:%02d:%02d.%03d",
snprintf(messageStr, sizeof(messageStr), "Scalar %s value = %04d-%02d-%02d %02d:%02d:%02d.%03d",
curr->data_type == SQL_TYPE_TIMESTAMP_WITH_TIMEZONE ? "TIMESTAMP WITH TIME ZONE" : "TIMESTAMP",
curr->ts_value->year, curr->ts_value->month, curr->ts_value->day,
curr->ts_value->hour % 24, curr->ts_value->minute, curr->ts_value->second,
(int)(curr->ts_value->fraction / 1000));
Expand Down
Loading
Loading