-
Notifications
You must be signed in to change notification settings - Fork 617
Description
Snakemake version
latest (9.13.8)
Describe the bug
I encountered a severe startup slowdown caused by the plugin loading mechanism in snakemake_interface_storage_plugins. I believe this reveals a general issue:
Storage plugins are eagerly imported during CLI initialization, and their top-level imports may bring in very expensive dependencies, even when the user never intends to use those plugins.
This is caused by plugin auto-discovery eagerly imports all storage plugins:
def collect_plugins(self) -> None:
"""Collect plugins and call register_plugin for each."""
self.plugins = dict()
# Executor plugins are externally installed plugins named
# "snakemake_executor_<name>".
# They should follow the same convention if on pip,
# snakemake-executor-<name>.
# Note that these will not be detected installed in editable
# mode (pip install -e .).
for moduleinfo in pkgutil.iter_modules():
if not moduleinfo.ispkg or not moduleinfo.name.startswith(
self.module_prefix
):
continue
module = importlib.import_module(moduleinfo.name)
self.register_plugin(moduleinfo.name, module)Logs
$ time pixi run --environment py313 snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
...
real 0m41.253s
user 0m2.558s
sys 0m0.272s
$ PYTHONPROFILEIMPORTTIME=1 time snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
...
import time: 167 | 44651 | boto3
import time: 118 | 118 | XRootD
import time: 495 | 495 | pyxrootd
import time: 40048126 | 40048126 | pyxrootd.client
import time: 134 | 134 | XRootD.client.url
...
$ pip uninstall snakemake_storage_plugin_xrootd
Found existing installation: snakemake-storage-plugin-xrootd 0.4.1
Uninstalling snakemake-storage-plugin-xrootd-0.4.1:
Would remove:
.pixi/envs/py313/lib/python3.13/site-packages/snakemake_storage_plugin_xrootd-0.4.1.dist-info/*
.pixi/envs/py313/lib/python3.13/site-packages/snakemake_storage_plugin_xrootd/*
Proceed (Y/n)? y
Successfully uninstalled snakemake-storage-plugin-xrootd-0.4.1
$ time snakemake -s 'tests/test_jupyter_notebook_nbconvert/Snakefile' -d test_module_rule_redef -n
real 0m0.984s
user 0m2.368s
sys 0m0.141sMinimal example
For my case, snakemake_storage_plugin_xrootd triggers import pyxrootd.client, which is extremely slow to load (~40 seconds on my system).
As a result, running even a small dry-run takes ~40 seconds solely because the XRootD client library is imported — even though my workflow does not use XRootD at all.
After uninstalling the storage plugin only, startup time drops from 41s to 0.6s.
Additional context
The current plugin loading mechanism does eager import of all plugins, regardless of:
- whether the user uses the plugin,
- whether storage backends are referenced,
- whether users even have the backend installed.
For storage plugins with heavy dependencies, this is unavoidable performance loss.