build: Package examples outside of the distribution

[cadenmyers13]

Problem

See issues/discussion here: scikit-package/scikit-package#667

Proposed solution

include it in MANIFEST.in and remove this from pyproject.toml

[tool.setuptools]
include-package-data = true

[tool.setuptools.package-data]
"diffpy.cmi" = ["../docs/examples/**/*"]

After doing this, build the sdist and whl files, install the package through them and make sure the cmi copy <example> still works

[cadenmyers13]

@sbillinge This is proving a bit more challenging than i thought. It was probably a bad idea to package examples with diffpy.cmi like we've done. This could get pretty heavy really quick. All of our CLI depends on requirements and examples being packaged with cmi. if you remove the setup.py (see here: https://github.com/diffpy/diffpy.cmi/blob/main/setup.py), code cannot find the necessary information when cmi commands are ran.

One idea for a fix would be to house all examples in a separate github repo, then when the user wants to get an example, the example is pulled from that repo. This would require internet though. Just an idea, im open to other options

[sbillinge]

One idea for a fix would be to house all examples in a separate github repo, then when the user wants to get an example, the example is pulled from that repo. This would require internet though. Just an idea, im open to other options

This would be fine. I can see occasions where it would be inconvenient (live demo in a room with bad internet for example), but in general, I would say it is a good tradeoff. If we have an "install all" kind of functionality it would be quick and easy to get all the examples before the demo for example.

What would be involved in making this switch?

[cadenmyers13]

@sbillinge Well first we would obviously need a repo in diffpy namespace where examples can be added. I would also have to update a bit of documentation but overall this should be easy.

The heaviest lift would be reworking how these examples are accessed. This involves changing source code and tests. My naive guess is that it wouldn't be too difficult.

Additionally, we package the requirements/ dir to be used in the cli info and install commands. It might be okay to leave this since these are extremely light txt or yml files. We could also try to figure out how to put these in a .py file in src/ using something like,

req_file = """
numpy
matplotlib
"""

I think this style has a name but i dont know it. I think this might be the more difficult part especially if users want to add there own packs/profiles tbh.

[sbillinge]

OK, sounds good. We may want to work on some UCs of users adding their own packs. Whatever we come up with, we want it to be fairly extensible (easily extended) but also maintainable (things don't break if a pack breaks for example). There will be some tradeoffs, but a couple of UCs (one where the user makes a "pack" but only uses it locally and doesn't merge it, one where the user wants to merge it into the codebase, maybe a third where the user makes their own repo with the pack so it is publicly available but not merged). Then we can chase a design that satisfies these as best as possible.

[sbillinge]

@cadenmyers13 what is the status of this one? did we separate out the examples in the end?

[cadenmyers13]

@sbillinge not done yet. We need to think about how to handle this