Skip to content

Storage plugin initialization loads heavy dependencies (e.g., XRootD) during CLI startup #3847

@Hocnonsense

Description

@Hocnonsense

Snakemake version
latest (9.13.8)

Describe the bug
I encountered a severe startup slowdown caused by the plugin loading mechanism in snakemake_interface_storage_plugins. I believe this reveals a general issue:

Storage plugins are eagerly imported during CLI initialization, and their top-level imports may bring in very expensive dependencies, even when the user never intends to use those plugins.

This is caused by plugin auto-discovery eagerly imports all storage plugins:

https://github.com/snakemake/snakemake-interface-common/blob/1b3449f5c7444f2eb070e4315d247263661c3d8d/src/snakemake_interface_common/plugin_registry/__init__.py#L82-L98

    def collect_plugins(self) -> None:
        """Collect plugins and call register_plugin for each."""
        self.plugins = dict()

        # Executor plugins are externally installed plugins named
        # "snakemake_executor_<name>".
        # They should follow the same convention if on pip,
        # snakemake-executor-<name>.
        # Note that these will not be detected installed in editable
        # mode (pip install -e .).
        for moduleinfo in pkgutil.iter_modules():
            if not moduleinfo.ispkg or not moduleinfo.name.startswith(
                self.module_prefix
            ):
                continue
            module = importlib.import_module(moduleinfo.name)
            self.register_plugin(moduleinfo.name, module)

Logs

$ time pixi run --environment py313 snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
... 
real 0m41.253s 
user 0m2.558s 
sys 0m0.272s
$ PYTHONPROFILEIMPORTTIME=1 time snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
...
import time:       167 |      44651 | boto3
import time:       118 |        118 | XRootD
import time:       495 |        495 |       pyxrootd
import time:  40048126 |   40048126 |       pyxrootd.client
import time:       134 |        134 |         XRootD.client.url
...
$ pip uninstall snakemake_storage_plugin_xrootd
Found existing installation: snakemake-storage-plugin-xrootd 0.4.1
Uninstalling snakemake-storage-plugin-xrootd-0.4.1:
  Would remove:
    .pixi/envs/py313/lib/python3.13/site-packages/snakemake_storage_plugin_xrootd-0.4.1.dist-info/*
    .pixi/envs/py313/lib/python3.13/site-packages/snakemake_storage_plugin_xrootd/*
Proceed (Y/n)? y
  Successfully uninstalled snakemake-storage-plugin-xrootd-0.4.1
$ time snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
real    0m0.984s
user    0m2.368s
sys     0m0.141s

Minimal example
For my case, snakemake_storage_plugin_xrootd triggers import pyxrootd.client, which is extremely slow to load (~40 seconds on my system).
As a result, running even a small dry-run takes ~40 seconds solely because the XRootD client library is imported — even though my workflow does not use XRootD at all.
After uninstalling the storage plugin only, startup time drops from 41s to 0.6s.

Additional context
The current plugin loading mechanism does eager import of all plugins, regardless of:

  • whether the user uses the plugin,
  • whether storage backends are referenced,
  • whether users even have the backend installed.

For storage plugins with heavy dependencies, this is unavoidable performance loss.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions