Repository navigation
Unbounded Archiving: Implement Treap and ND-tree archivers for C and Python. - #82
MLopez-Ibanez wants to merge 18 commits into
Conversation
aaaaa12 to
b74aa19
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #82 +/- ##
==========================================
+ Coverage 78.50% 79.23% +0.72%
==========================================
Files 70 82 +12
Lines 7608 9377 +1769
Branches 1477 1840 +363
==========================================
+ Hits 5973 7430 +1457
- Misses 1262 1436 +174
- Partials 373 511 +138
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
15e31e2 to
71ac7d9
Compare
e851134 to
9061670
Compare
71ac7d9 to
b2fcf27
Compare
9061670 to
4a8dfc6
Compare
b2fcf27 to
18d204a
Compare
|
Thanks for the work @MLopez-Ibanez and @fieldsend! A few comments:
|
Yes, this can definitely be improved and we have some ideas. However, it will also be interesting to understand what you mean by "impossible". It takes less than Could you suggest how to extend https://github.com/multi-objective/moocore/blob/18d204abb1eccb09c8b448488e473d4112d51817/python/benchmarks/bench_archive_unbounded.py to also benchmark If you tell me how to benchmark it, I can run the benchmark and add the results to this PR.
Sure, there are many improvements that can be done to make the code faster. I will add a TODO-archiving file. I am also happy for people to fork the code and experiment with it. As mentioned above, there is a benchmark file that should be very easy to extend and run. However, it is pretty fast already, so before making it even faster, we would like to know if it is fast enough for your purposes and if the API is OK or should be renamed/extended.
Good point! I think I know how to handle this by adding a "revision" field to the tree and iterator and if there is a mismatch, the iterator is invalid. I'll implement this next. |
18d204a to
2a5aa79
Compare
This is now implemented and tested. Modifying the archive while iterating raises an exception. |
It was a bad decision on my part regarding datastructures. We were using polars dataframes in other parts of the framework so I decided to keep things consistent and used them for archivng purposes too. After doing some basic profiling, most of the time is actually being spent on constructing these. I also tried comparing our implementation against yours using the benchmarking script you pointed out. I had to lower the budget because I didn't want to wait too long for DESDEO. Based on the result, I don't think it is necessary to add this comparison to the official docs of moocore. We will just switch our implementation to moocore's as soon as it's merged to main.
Minor suggestion for the plots/docs. Change the line and marker style for |
Did you normalise the time by the number of points like it is done for the other archives, so that the time shown is the average time to add a single point? |
4a8dfc6 to
775a67f
Compare
8187235 to
ee74988
Compare
775a67f to
d759153
Compare
d759153 to
4412f9b
Compare
4412f9b to
1a21f8c
Compare
fdc4c07 to
086ef14
Compare
086ef14 to
9183af0
Compare
0ce6b27 to
4b546a8
Compare
|
@light-weaver Hi Bhupinder, does the archive in DESDEO accept duplicated points or discards them? |
|
The default in |
ed86475 to
b7f7268
Compare
* treap2d.h: New. * treap2d_dlnode.h: New. * treap2d_fixed_size.c: New. * treap2d_fixed_size.h: New. * pareto.c (pareto_rank_3d): Replace AVL-tree with Treap. * nondominated_kung.h (find_nondominated_3d_impl_sorted, kung_merge_dim3): Likewise. * hv3d_priv.h (hv3d_preprocessing): Likewise.
* avl_tiny.h: Delete. * Makefile: Update. * NEWS.md: Update. * hv3d_priv.h: Delete AVL-tree code. * nondominated.c: Likewise. * pareto.c: Likewise.
* r/inst/REFERENCES.bib: Regenerate. * python/doc/source/REFERENCES.bib: Regenerate.
…_iter__ to special-members.
and hide the class/module prefixes for objects in sidebar.
* TODO-archiving: New. * c/archive.h: New. * c/archiving.h: New. * c/archiving_priv.h: New. * c/common.h (squared_distance, which_max): New. * c/main-archiver.c: New. * c/nd_tree.h: New. * c/nd_tree_iterator.h: New. * c/treap_archive.c: New. * c/treap_archive.h: New. * c/treap_iterator.h: New. * c/flex_bucket.h: New. * python/doc/source/reference/functions.archiving.rst: New. * python/doc/source/whatsnew/index.rst: Document it. * python/src/moocore/__init__.py: Export UnboundedArchive and BaseUnboundedArchive. * python/src/moocore/_archive.py: New. * python/src/moocore/_docsubstitute.py (COMMON_PARAMS): Add dim. * python/src/moocore/_moocore.py (_parse_maximise): Move to ... * python/src/moocore/_utils.py: ... here. (_as_1d_float_array, _as_2d_float_array): New. * python/tests/conftest.py (AutoIncrement): New class. (count): New fixture. * python/tests/test_archiving.py: New. * r/src/install.libs.R: Add archiver.
* benchmarks/bench.py (check_array_equal): Use np.testing.assert_array_equal(). (check_bool_array_equal): New. (Bench.__init__): Check setup. Print timestamp with datetime. * benchmarks/bench_archive_unbounded.py: New. * doc/source/benchmarks_archiving.rst: New. * doc/source/benchmarks.inc.rst: Link to previous. * doc/source/_static/bench/unbarchive_*.png. New files.
b7f7268 to
1fd65f3
Compare
for more information, see https://pre-commit.ci

Hi @Dvermetten @olafmersmann @ttusar @gaplopes @light-weaver @adbjesus @apguerreiro
Since our meeting in Coimbra, @fieldsend and myself have been slowly working on implementing the archiver API that we discussed in the meeting.
We have an incomplete WIP implementation that does the most basic operations.
We are looking for feedback on the API, testcases, examples, suggestions for fixes and help in improving / completing it.
API and examples
Benchmarks