`Extension error` when both `"myst_nb"` and `"myst_parser"` are in `extensions`

[NickleDave]

Describe the bug

Hi, thanks so much for MyST-NB, I am very happy I can include .md notebooks in my build, and switch over from nbsphinx fairly painlessly.

I am not even sure if this is a bug per se but it is behavior that was a bit surprising so I want to raise an issue about it, since I would guess that other people that switch to myst-parser and then add myst-nb might run into the same situation.

context
I added MyST-NB to a project after switching from .rst to MyST parser, because I wanted to additionally execute notebooks during builds, which double as tutorial material in the docs.

This gave me something like the following in my conf.py

extensions = [
    "myst_nb",
    "myst_parser",
    ...
]

expectation
I expected to be able to just build and execute notebooks after this.

bug
But instead I get:

$ sphinx-build -nW --keep-going -b html docs/ docs/_build/html
Running Sphinx v4.5.0

Extension error:
source_suffix '.md' is already registered

(I get the same error when using sphinx-autobuild, if it matters.)

problem

I think this might be intended behavior, judging from this note in the docs?

Loading the myst_nb extension also activates the myst_parser extension, for enabling the MyST flavour of Markdown. It is not required to add this explicitly in the list of extensions.

But it's still surprising, at least to me.

I would guess one of the following behaviors might be a bit less confusing:

• a very explicit error, like: there is no need to add myst_parser when you are using myst_nb, please remove myst_parser from extensions in conf.py
  • by the same token, if this is the usage pattern, then maybe the docs should be revised to an admonition that very clearly states "you will need to remove myst_parser from extensions if you add myst_nb"
• or make it so that MyST-NB and MyST parser "know" about each other and can gracefully co-exist as extensions -- I realize this might be easier said than done depending on the plumbing involved

Reproduce the bug

I am able to get this to happen with a minimal build so I'm pretty sure it's not something unique to my setup.

1. make a venv
2. pip install sphinx + myst-parser
3. make a dummy index.md like the one attached
4. additionally pip install myst-nb
5. modify conf.py as above so extensions includes myst_nb and myst_parser
6. ???
7. profit
8. er no I mean get the same error

$ sphinx-build -nW --keep-going -b html docs/ docs/_build/html
Running Sphinx v4.5.0

Extension error:
source_suffix '.md' is already registered

index.md

List your environment

• Not using jupyter-book
• Python 3.10
• Ubuntu-ish

 pip freeze
alabaster==0.7.12
asttokens==2.0.5
attrs==21.4.0
Babel==2.10.1
backcall==0.2.0
certifi==2022.5.18.1
charset-normalizer==2.0.12
click==8.1.3
debugpy==1.6.0
decorator==5.1.1
docutils==0.17.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
greenlet==1.1.2
idna==3.3
imagesize==1.3.0
importlib-metadata==4.11.4
ipykernel==6.13.0
ipython==8.4.0
jedi==0.18.1
Jinja2==3.1.2
jsonschema==4.6.0
jupyter-cache==0.5.0
jupyter-client==7.3.1
jupyter-core==4.10.0
markdown-it-py==2.1.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mdit-py-plugins==0.3.0
mdurl==0.1.1
myst-nb==0.15.0
myst-parser==0.17.2
nbclient==0.5.13
nbformat==5.4.0
nest-asyncio==1.5.5
packaging==21.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.29
psutil==5.9.1
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
pyzmq==23.1.0
requests==2.27.1
six==1.16.0
snowballstemmer==2.2.0
Sphinx==4.5.0
sphinx-togglebutton==0.3.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
SQLAlchemy==1.4.37
stack-data==0.2.0
tabulate==0.8.9
tornado==6.1
traitlets==5.2.2.post1
typing_extensions==4.2.0
urllib3==1.26.9
wcwidth==0.2.5
zipp==3.8.0

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[Timost]

Hi ! +1 on this issue.

I think warning about this behaviour should be made more visible, maybe in the https://myst-nb.readthedocs.io/en/latest/quickstart.html

[NickleDave]

I noticed when googling that there used to be a {warning} callout but it must have been removed at some point?

From this page:
https://myst-nb.readthedocs.io/en/v0.9.0/_sources/use/myst.md.txt

:::{warning}
If you are using MyST-NB in your documentation, do not activate `myst-parser`. It will
be automatically activated by `myst-nb`.
:::

This is more direct (and possibly not the friendliest of language, maybe why it was removed) but still doesn't state directly that activating both will cause an error

[CsatiZoltan]

@NickleDave If I replace 'myst_parser' with 'myst_nb', in the extensions variable, while writing in Markdown syntax

source_suffix = {'.md': 'markdown'`}

I get the error

Sphinx error:
Source parser for markdown not registered
make: *** [html] Error 2

It seems to me that the myst_parser is required to write in Markdown.

[ahasha]

I have the same issue -- I have non-notebook documentation written in myst-markdown, and wanted to switch from nbsphinx to myst-nb for my notebook-driven documentation pages to avoid their pandoc dependency and get fully into the executablebooks ecosystem. However, it seems I cannot have both because of this issue.

FYI -- I have installed myst_nb from the current main branch, rev 3d6a5d1, in order to use it with sphinx 6.2.1

[pwuertz]

I'm new to this and quite confused. Maybe the general idea is to pipe everything via jupytext into myst_nb because every .md file is potentially a Notebook file?

https://docs.readthedocs.io/en/stable/guides/jupyter.html#using-notebooks-in-other-formats suggests:

# conf.py
nb_custom_formats = {
    ".md": ["jupytext.reads", {"fmt": "mystnb"}],
}

In which case myst-nb handles both plain .md files and Notebooks in .md format? Is there any downside to this (handling .md via myst-nb instead of using myst-parser)?

[maxstrobel]

+1 same issue here 😅 are already workarounds available?

[NickleDave]

Sounds like the question of whether to specify both myst-nb and myst-parser in extension is confusing in general.

But we're now at a point on this issue where people are chiming in with multiple things that might or might not be related.

@chrisjsewell @choldgraf apologies in advance for adding to your notifications but what do you think about clarifying here and/or adding info in the docs about when and how to use these extensions together?

[michaelweinold]

Related:

• DOC: hiding cells with LaTeX #622

[MikeOpenHWGroup]

FWIW, I had recommonmark previously defined in my extensions and encountered this issue after adding myst_parser. Getting rid of the former worked for me:

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.todo',
#    'recommonmark',
    'sphinx.ext.autosectionlabel',
    'sphinx_markdown_tables',
    'myst_parser',
]

I have not noticed any ill-effects after making this change and I am now able to use the "admonition" feature from myst_parser.

[Bain-OS]

I'm new to this and quite confused. Maybe the general idea is to pipe everything via jupytext into myst_nb because every .md file is potentially a Notebook file?

https://docs.readthedocs.io/en/stable/guides/jupyter.html#using-notebooks-in-other-formats suggests:

conf.py

nb_custom_formats = {
".md": ["jupytext.reads", {"fmt": "mystnb"}],
}

In which case myst-nb handles both plain .md files and Notebooks in .md format? Is there any downside to this (handling .md via myst-nb instead of using myst-parser)?

not quite, there are some things myst-nb does not support. eg code nesting