---

Table of Contents

• Introduction
• Development Workflow of an Algorithm in streamPAMI
• Inputs and Outputs of an Algorithm in streamPAMI
• Accuracy Guarantees
• Evaluation
• Features
• Maintenance
• Try your first streamPAMI program
• Mining a live stream
• Testing
• License
• Background
• Getting Help
• Contribution to streamPAMI
• Algorithms
• References

---

Introduction

streamPAMI (stream PAttern MIning) is a Python library of algorithms that discover user interest-based patterns in data streams: unbounded sequences of transactions that arrive over time and cannot be stored or rescanned in full.

It is the stream counterpart of PAMI and follows the same package layout and the same method names, so PAMI users can start without learning a new interface.

	PAMI	streamPAMI
Data	a static database, read in full	a stream, read one batch at a time
Memory	the whole database	a compact summary of the stream
Answers	exact, for the whole database	for any time period; exact or with a stated error bound
Live data	no	yes: processBatch() / queryPatterns()

---

Development Workflow of an Algorithm in streamPAMI

Every algorithm passes a literature check, brute-force oracle tests after every batch, and mutation checks before it is documented. Every change is then independently reviewed in a clean environment before it is merged.

---

Inputs and Outputs of an Algorithm in streamPAMI

Inputs (example: FP-Stream)

Parameter	Meaning
iFile	Input stream: a file path, a URL, or a pandas DataFrame with a Transactions column (same formats as PAMI). Use None when feeding batches yourself with processBatch().
minSup	Minimum support σ as a fraction in (0, 1) of the transactions in the queried period. Unlike PAMI, absolute counts are not accepted, because the size of a period depends on the query.
epsilon	Maximum support error ε, in (0, minSup). Smaller is more accurate but uses more memory.
batchSize	Number of transactions per batch when reading iFile.
sep	Item separator (default: tab).

Outputs

Method	Returns
mine()	Reads iFile batch by batch and stores the patterns for the whole stream.
getPatterns()	{(item, ...): support} from the last mine().
getPatternsAsDataFrame()	The same, as a DataFrame with columns Patterns and Support.
save(oFile)	Writes pattern:support lines.
processBatch(transactions)	Adds one batch to the summary.
queryPatterns(startFrame, endFrame)	Patterns for a period made of time frames (0 = newest). Can be called at any time.
getTimeFrames()	The batch range (start, end) covered by each time frame, newest first.
getRuntime(), getMemoryRSS(), getMemoryUSS(), printResults()	As in PAMI.

---

Accuracy Guarantees

Approximate stream algorithms trade accuracy for memory. Every algorithm in streamPAMI states its guarantee, and the test suite checks that guarantee against a brute-force miner.

FP-Stream (Giannella et al., 2003). The guarantee holds only when σ > 2·ε and the queried period starts at the newest time frame. For such a period with W transactions and true support f:

1. Every pattern with f ≥ σ·W is returned (no false negatives).
2. Each returned support lies in [f − ε·W, f].
3. Patterns with f between (σ − ε)·W and σ·W may also be returned.

Known limitations. With ε ≥ σ/2, false negatives are possible, and the constructor emits a UserWarning. For periods that start at an older frame, a frequent pattern can be missed when recent batches are much larger than older ones. A reproducible case is recorded in tests/test_FPStream.py::test_false_negative_older_period_uneven_batches.

DSTree is exact over any contiguous range of batches in its sliding window of windowSize batches. Empty transactions supplied to processBatch count towards the support denominator; blank lines in files and URLs are skipped. Its tree retains the window's transactions, compressed by shared prefixes, so memory grows with the window and its per-node batch counts; it has no epsilon-bounded summary.

Lossy Counting (Manku and Motwani, 2002) uses a landmark window: queries cover the whole processed stream of N transactions. Every itemset with true count at least minSup * N is returned; no output has true count below (minSup - epsilon) * N; each reported count is between the true count minus epsilon * N and the true count. batchSize must be a multiple of ceil(1/epsilon). Trailing transactions that do not fill a bucket are held unprocessed, excluded from N and the guarantees, and reported with a warning and getHeldCount(). The paper's Buffer/Trie/SetGen data structures and SetGen pruning are not reproduced; every tracked itemset is counted exactly each batch.

---

Evaluation

The stream is queried after every batch of 10,000 transactions. FP-Stream queries the whole stream so far, with epsilon = 0.1 * minSup; DSTree queries its sliding window of three batches. Each is compared with exact FP-growth from PAMI, re-mined after every batch over the same horizon with integer support threshold ceil(minSup * W). We use minSup values 0.006, 0.007, 0.008, 0.009 and 0.010, with five repetitions in fresh subprocesses per method and configuration. All 100 runs are randomised together with fixed seed 20261020. The plots show final pattern counts, median total runtime across all ten batches, and median process peak memory. Runtime and memory error bars span the minimum to maximum of the five repetitions. These measurements describe one laptop. We call a method faster or slower only when its min–max runtime range does not overlap the other method's; overlapping or touching ranges are described as similar.

Dataset: T10I4D100K. The supplied file contains 99,935 transactions, so the final batch contains 9,935; all thresholds use actual transaction counts. Dataset reading is excluded from runtime. Baseline runtime includes preparing its temporary input file at each batch. Memory includes imports, the preloaded dataset, tree state and retained answers for validation, in decimal MB.

Machine / software	Captured value
OS	Windows-10-10.0.26200-SP0
CPU	Intel(R) Core(TM) i7-14650HX, 24 logical CPUs
RAM	34.049 GB
Python	3.11.9
pandas	3.0.6
PAMI	2026.10.13.1
streamPAMI commit measured	ea79897f34a2829051bcaf62faca44184a9dd91b
Run-order seed	20261020

FP-Stream: whole stream

FP-Stream returned more patterns than the exact baseline at every setting, as its query threshold also admits candidates below minSup. Runtimes were similar to PAMI's at all five settings because the observed ranges overlapped. Its peak-memory ranges were below PAMI's at every setting on this laptop.

DSTree: last three batches

streamPAMI's DSTree is an eager-maintenance variant (see its docstring), so runtime and memory describe this implementation, not the original authors' design.

DSTree's pattern sets and supports matched PAMI after every batch in all repetitions. Runtimes were similar at minSup 0.006. DSTree was slower at 0.007, 0.008, 0.009 and 0.010, where the runtime ranges did not overlap PAMI's. Its peak-memory ranges were lower at 0.006–0.008, overlapped PAMI's at 0.009, and were higher at 0.010. The CSVs retain every repetition.

FP-Stream precision and recall

Final recall was 1.0 at every minSup in every repetition. Precision ranged from 0.8257 to 0.9145; the additional patterns account for the difference from 1.0.

Reproduce from the repository root with the project's .venv activated and the bench extra installed:

python benchmarks/run_benchmarks.py --data benchmarks/data/Transactional_T10I4D100K.csv && python benchmarks/plot_benchmarks.py

See benchmarks/README.md for the pilot command, measurement boundaries, machine details, raw CSV results and real console output.

---

Features

• ✅ Every algorithm is checked against a brute-force oracle after every batch, on random and drifting streams
• 📏 Stated accuracy guarantees, with exact (fractional) threshold arithmetic
• 🌊 Works on live streams, not only on files replayed from disk
• 🧠 Memory follows the algorithm: a compact summary or a compressed sliding window
• 🔁 Same interface as PAMI
• 🖥️ Operating system independent

---

Maintenance

Installation

streamPAMI is not on PyPI yet. Install it from GitHub:

pip install git+https://github.com/udayRage/streamPAMI.git

For development (includes pytest):

git clone https://github.com/udayRage/streamPAMI.git
cd streamPAMI
pip install -e ".[dev]"

Upgradation

pip install --upgrade git+https://github.com/udayRage/streamPAMI.git

Uninstallation

pip uninstall streamPAMI

Information

pip show streamPAMI

---

Try your first streamPAMI program

$ python

# first import streamPAMI
from streamPAMI.frequentPattern.basic import FPStream as alg
fileURL = "https://u-aizu.ac.jp/~udayrage/datasets/transactionalDatabases/Transactional_T10I4D100K.csv"
obj = alg.FPStream(iFile=fileURL, minSup=0.01, epsilon=0.002, batchSize=10000, sep='\t')
obj.mine()
obj.save('frequentPatternsAtMinSup0.01.txt')
frequentPatternsDF = obj.getPatternsAsDataFrame()
print('Total No of patterns: ' + str(len(frequentPatternsDF)))
print('Runtime: ' + str(obj.getRuntime()))
print('Memory (RSS): ' + str(obj.getMemoryRSS()))
print('Memory (USS): ' + str(obj.getMemoryUSS()))

Output:
<!-- TODO(Ojash): run the code above and paste the REAL output here. -->

---

Mining a live stream

Feed batches as they arrive and ask for the current answer at any time:

from streamPAMI.frequentPattern.basic import FPStream as alg

obj = alg.FPStream(iFile=None, minSup=0.4, epsilon=0.1, batchSize=4)

stream = [
    [["bread", "milk"], ["bread", "butter"], ["milk"], ["bread", "milk", "butter"]],
    [["bread", "milk"], ["milk"], ["milk", "eggs"], ["bread", "milk"]],
    [["eggs"], ["milk", "eggs"], ["eggs", "butter"], ["milk", "eggs"]],
]
for batch in stream:
    obj.processBatch(batch)                      # update the summary with one batch
    print("Time frames (newest first):", obj.getTimeFrames())
    print("Frequent now:", obj.queryPatterns())  # answer for the whole stream so far

print("Most recent batch only:", obj.queryPatterns(startFrame=0, endFrame=1))

Output:
Time frames (newest first): [(0, 1)]
Frequent now: {('bread',): 3, ('butter',): 2, ('milk',): 3, ('bread', 'butter'): 2, ('bread', 'milk'): 2}
Time frames (newest first): [(1, 2), (0, 1)]
Frequent now: {('bread',): 5, ('milk',): 7, ('bread', 'milk'): 4}
Time frames (newest first): [(2, 3), (0, 2)]
Frequent now: {('bread',): 5, ('milk',): 9, ('bread', 'milk'): 4, ('eggs',): 5}
Most recent batch only: {('milk',): 2, ('eggs',): 4, ('eggs', 'milk'): 2}

Notice how eggs becomes frequent once it appears in the third batch, and how batches 0 and 1 were merged into one time frame (0, 2): older data is kept at coarser granularity.

---

Testing

pytest -v

Each algorithm is compared after every batch against tests/bruteForce.py, which counts every itemset exactly. Exact algorithms must match it; approximate algorithms must satisfy their stated guarantee. Known limitations are recorded as strict expected failures (xfail), so they cannot be silently forgotten or silently "fixed".

---

License

streamPAMI is released under the GNU General Public License v3.0.

---

Background

streamPAMI is developed at UdayLab, University of Aizu, as the stream mining companion to PAMI.

---

Getting Help

Please open an issue at GitHub Issues.

---

Contribution to streamPAMI

Contributions are welcome: bug reports, fixes, documentation and new algorithms. A new algorithm must:

1. follow the PAMI layout: streamPAMI/<patternFamily>/<variant>/<Algorithm>.py;
2. state its paper reference and accuracy guarantee in its docstring;
3. come with a test against the brute-force oracle that checks that guarantee after every batch.

---

Algorithms

1. Pattern mining in binary transactional streams

1.1. Frequent pattern mining

Basic	Closed	Maximal	Top-k
FP-Stream, DSTree, Lossy Counting

---

References

• C. Giannella, J. Han, J. Pei, X. Yan and P. S. Yu, "Mining Frequent Patterns in Data Streams at Multiple Time Granularities", Next Generation Data Mining, 2003. PDF
• C. K.-S. Leung and Q. I. Khan, "DSTree: A Tree Structure for the Mining of Frequent Sets from Data Streams", Proc. IEEE ICDM, 2006, pp. 928-932, doi:10.1109/ICDM.2006.62.
• G. S. Manku and R. Motwani, "Approximate Frequency Counts over Data Streams", Proc. 28th VLDB Conference, Hong Kong, 2002.

Go to Top