Skip to content
7 changes: 7 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,13 @@ History
such as a ``gzip.GzipFile``. ``maxminddb.types.SupportsRead`` describes
this type.

* ``MODE_AUTO`` now accepts a binary file object. Previously, it raised
``TypeError``. It reads the file object into memory with the pure Python
reader, as ``MODE_FD`` does. The C extension needs a path, so lookups are
slower than with a path when the extension is installed.
* The pure Python reader raises ``TypeError`` when the database argument does
not suit the mode.

3.2.0 (2026-09-10)
++++++++++++++++++

Expand Down
17 changes: 10 additions & 7 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,21 +39,24 @@ provide `free GeoLite databases
files must be decompressed with ``gunzip``.

After you have obtained a database and imported the module, call
``open_database`` with a path, or file descriptor (in the case of ``Mode.FD``),
``open_database`` with a path, or binary file object (with ``Mode.FD`` or ``Mode.AUTO``),
to the database as the first argument. Optionally, you may pass a mode as the
second argument. The modes are available from ``maxminddb.Mode``. Valid modes are:

* ``Mode.MMAP_EXT`` - use the C extension with memory map.
* ``Mode.MMAP`` - read from memory map. Pure Python.
* ``Mode.FILE`` - read database as standard file. Pure Python.
* ``Mode.MEMORY`` - load database into memory. Pure Python.
* ``Mode.FD`` - load database into memory from a file descriptor. Pure Python.
* ``Mode.FD`` - load database into memory from a binary file object. Pure Python.
* ``Mode.AUTO`` - try ``Mode.MMAP_EXT``, ``Mode.MMAP``, ``Mode.FILE`` in that
order. Default.

**NOTE**: When using ``Mode.FD``, it is the *caller's* responsibility to be
sure that the file descriptor gets closed properly. The caller may close the
file descriptor immediately after the ``Reader`` object is created.
order. A file object is read into memory with the pure Python reader, as
with ``Mode.FD``. Pass a path to use the faster C extension. Default.

**NOTE**: When using a file object, it is the *caller's* responsibility to be
sure that the file object gets closed properly. The caller may close the
file object immediately after the ``Reader`` object is created. The pure
Python reader closes a file descriptor that you pass to it. This includes
``Mode.AUTO`` when the C extension is not installed.

The ``open_database`` function returns a ``Reader`` object. To look up an IP
address, use the ``get`` method on this object. The method will return the
Expand Down
51 changes: 35 additions & 16 deletions maxminddb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
Mode,
)
from .errors import InvalidDatabaseError
from .reader import Reader
from .reader import _PATH_TYPES, Reader, _is_descriptor, _type_name

if TYPE_CHECKING:
from .types import DatabaseSource
Expand Down Expand Up @@ -48,16 +48,27 @@ def open_database(

Arguments:
database: A path to a valid MaxMind DB file such as a GeoIP database
file, or a file descriptor in the case of MODE_FD.
file, or a binary file object for MODE_FD or MODE_AUTO.
MODE_MMAP, MODE_FILE and MODE_MEMORY also accept the file
descriptor of a regular file. MODE_MEMORY reads it from its
current offset, the others from the start. Without the C
extension, MODE_AUTO accepts one too. Once the reader opens
a file descriptor, it closes it, even when the file is not
a valid database. If an error comes first, for example for
a mode that takes no descriptor or for a directory, the
caller still owns it. The reader does not close a file
object, but it closes a buffer that read() returns if the
buffer has close(), such as an mmap.
mode: mode to open the database with. Valid mode are:
* MODE_MMAP_EXT - use the C extension with memory map.
* MODE_MMAP - read from memory map. Pure Python.
* MODE_FILE - read database as standard file. Pure Python.
* MODE_MEMORY - load database into memory. Pure Python.
* MODE_FD - the param passed via database is a file descriptor, not
a path. This mode implies MODE_MEMORY.
* MODE_FD - the param passed via database is a binary file
object, not a path. This mode implies MODE_MEMORY.
* MODE_AUTO - tries MODE_MMAP_EXT, MODE_MMAP, MODE_FILE in that
order. Default mode.
order. Reads a file object into memory, as MODE_FD
does. Default mode.

"""
if mode not in (
Expand All @@ -72,23 +83,31 @@ def open_database(
raise ValueError(msg)

has_extension = _extension and hasattr(_extension, "Reader")
use_extension = has_extension if mode == MODE_AUTO else mode == MODE_MMAP_EXT

if not use_extension:
return Reader(database, mode)

if not has_extension:
if mode == MODE_MMAP_EXT and not has_extension:
msg = "MODE_MMAP_EXT requires the maxminddb.extension module to be available"
raise ValueError(
msg,
)

# The C type exposes the same API as the Python Reader, so for type
# checking purposes, pretend it is one. (Ideally this would be a subclass
# of, or share a common parent class with, the Python Reader
# implementation.) The extension accepts only a path. It raises TypeError
# for a file descriptor or a file object.
return cast("Reader", _extension.Reader(database, mode)) # type: ignore[arg-type]
# The extension accepts only a path, so MODE_AUTO gives a file object to
# the pure Python reader. It still refuses a file descriptor, as before.
# The cast pretends the C reader is the pure Python Reader, which has the
# same API.
if mode in (MODE_AUTO, MODE_MMAP_EXT) and has_extension:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code checks the database type in two places. open_database and Reader._load_buffer each test isinstance(database, _PATH_TYPES), _is_descriptor() and callable(getattr(database, "read", None)), and each builds its own hints.

The messages already differ: one says "(X given). Use MODE_FD for a file object." and the other says "Unsupported database type (X). ...". One shared helper that returns the kind (path, descriptor, file object or other) would remove the repeated checks and keep the messages the same.

馃 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent reply on behalf of @oschwald.

Not changed. _is_descriptor is the one shared check, and both places use it. Each place sorts the argument for its own reader. The extension takes only a path, but the pure Python reader also takes a descriptor or a file object. So each one gives the hint that fits its own reader.

if isinstance(database, _PATH_TYPES):
return cast("Reader", _extension.Reader(database, mode))
# Leave a bool to the pure Python reader, which refuses it.
is_descriptor = _is_descriptor(database)
if mode == MODE_MMAP_EXT or is_descriptor:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In MODE_AUTO, a file descriptor works or fails depending on whether the C extension is installed.

  • Without the extension, open_database(os.open(p, os.O_RDONLY)) works and the reader closes the fd.
  • With the extension, it raises TypeError and the fd stays open.

So code that works in one environment fails (and leaks fds) in the other. This PR adds new code that keeps this split. Can we send a descriptor in MODE_AUTO to Reader(database, mode), as we do for a file object? The other option is to refuse a descriptor in both readers.

馃 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent reply on behalf of @oschwald.

Not changed. 3.2.0 already had both behaviors. With the extension, open_database(fd) raised TypeError. Without it, the pure Python reader opened the fd and closed it. This PR keeps both.

  • If we refuse a descriptor in both readers, PyPy and pure Python code that works today breaks.
  • If we send it to Reader with the extension installed, a call that raised before would now take ownership of the fd and close it, and it would use the slower reader without warning.

The docstrings describe both cases.

msg = f"The C extension requires a path ({_type_name(database)} given)."
if is_descriptor:
msg += " Use MODE_MMAP for a file descriptor."
elif callable(getattr(database, "read", None)):
msg += " Use MODE_FD for a file object."
raise TypeError(msg)

return Reader(database, mode)


__version__ = version("maxminddb")
7 changes: 5 additions & 2 deletions maxminddb/const.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ class Mode(IntEnum):
"""

AUTO = 0
"""Try MODE_MMAP_EXT, MODE_MMAP, MODE_FILE in that order. Default mode."""
"""Try MODE_MMAP_EXT, MODE_MMAP, MODE_FILE in that order. Default mode.

A file object uses MODE_FD.
"""

MMAP_EXT = 1
"""Use the C extension with memory map."""
Expand All @@ -25,7 +28,7 @@ class Mode(IntEnum):
"""Load database into memory. Pure Python."""

FD = 16
"""Database is a file descriptor, not a path. This mode implies MODE_MEMORY."""
"""Database is a binary file object, not a path. This mode implies MODE_MEMORY."""


# Backward compatibility: export both enum members and old-style constants
Expand Down
4 changes: 2 additions & 2 deletions maxminddb/decoder.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ class Decoder:

def __init__(
self,
database_buffer: FileBuffer | mmap.mmap | bytes,
database_buffer: FileBuffer | mmap.mmap | bytes | bytearray,
pointer_base: int = 0,
pointer_test: bool = False, # noqa: FBT001, FBT002
) -> None:
Expand Down Expand Up @@ -116,7 +116,7 @@ def _decode_bytes(
size: int,
offset: int,
budget: _DecodeBudget,
) -> tuple[bytes, int]:
) -> tuple[bytes | bytearray, int]:
# Charge the payload before copying so a crafted size cannot force a
# large allocation, and so pointers reusing one target recharge.
remaining = budget.payload_left - size
Expand Down
7 changes: 5 additions & 2 deletions maxminddb/file.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,21 @@
from __future__ import annotations

import os
from typing import overload
from typing import TYPE_CHECKING, overload

try:
from multiprocessing import Lock
except ImportError:
from threading import Lock # type: ignore[assignment]

if TYPE_CHECKING:
from maxminddb.types import StrOrBytesPath


class FileBuffer:
"""A slice-able file reader."""

def __init__(self, database: str) -> None:
def __init__(self, database: StrOrBytesPath | int) -> None:
"""Create FileBuffer."""
self._handle = open(database, "rb") # noqa: SIM115
self._size = os.fstat(self._handle.fileno()).st_size
Expand Down
Loading
Loading