Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configparser.NoOptionError: No option 'storage-backend' in section: 'mirror' #1702

Closed
epifeny opened this issue Apr 11, 2024 · 2 comments · Fixed by #1740
Closed

configparser.NoOptionError: No option 'storage-backend' in section: 'mirror' #1702

epifeny opened this issue Apr 11, 2024 · 2 comments · Fixed by #1740
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@epifeny
Copy link

epifeny commented Apr 11, 2024

Hello,

I'm using v6.5.0 and when I attempt to use bandersnatch verify --delete I get a key error.
I read the docs, and from what I understand, storage-backend defaults to filesystem, so I didn't think it would be required to add it into the configuration file.
https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html#storage-backend
image

The mirror section looks like this:

[mirror]
directory = /opt/pip/external
hash-index = false
json = true
master = https://pypi.org
stop-on-error = false
timeout = 10
verifiers = 5
workers = 1

/ # bandersnatch --version
bandersnatch 6.5.0

/ # bandersnatch --config /etc/bandersnatch.conf verify --delete
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/configparser.py", line 805, in get
    value = d[option]
            ~^^^^^^^^
  File "/usr/local/lib/python3.11/collections/__init__.py", line 1006, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/collections/__init__.py", line 998, in __missing__
    raise KeyError(key)
KeyError: 'storage-backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/bandersnatch", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bandersnatch/main.py", line 225, in main
    return asyncio.run(async_main(args, config))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bandersnatch/main.py", line 161, in async_main
    return await bandersnatch.verify.metadata_verify(config, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bandersnatch/verify.py", line 241, in metadata_verify
    backend=config.get("mirror", "storage-backend"),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/configparser.py", line 808, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'storage-backend' in section: 'mirror'

/ # pip freeze
aiohttp==3.9.3
aiohttp-socks==0.8.4
aiohttp-xmlrpc==1.5.0
aiosignal==1.3.1
async-timeout==4.0.3
attrs==23.2.0
bandersnatch==6.5.0
filelock==3.13.3
frozenlist==1.4.1
humanfriendly==10.0
idna==3.6
lxml==5.2.1
multidict==6.0.5
packaging==24.0
python-socks==2.4.4
yarl==1.9.4
@cooperlees
Copy link
Contributor

Thanks for the report. @flyinghyrax is working on making our defaults better as there are some gaps. This is one.

#1694 is part of it.

@cooperlees cooperlees added bug Something isn't working help wanted Extra attention is needed labels Apr 12, 2024
@flyinghyrax
Copy link
Contributor

Indeed! I'm hoping to work more on it this weekend.

flyinghyrax added a commit to flyinghyrax/bandersnatch that referenced this issue May 27, 2024
This splits the existing 'default.conf' config file shipped with the package into two
similar files: "defaults.conf" and "example.conf". "example.conf" is an exact copy of the
previous "default.conf". The new "defaults.conf" is a stripped-down version containing
only default values for all mirror configuration options except "mirror.directory".

BandersnatchConfig is changed to *always* read defaults.conf, then read the user config
file if one is specified. This leaves the ConfigParser populated with default values for
any mirror options that aren't set by the user (except mirror.directory).

Notable ripple effects for this include:
- It is no longer meaningful to check ConfigParser.has_option with the 'mirror' section.
  Instead, you have to check whether the options value is empty or None.
- Specifying a default/fallback value when calling .get on the 'mirror' section will
  have no effect, because the option will already be present in the ConfigParser mappings.

As (mostly) an implementation detail, BandersnatchConfig is changed to be a subclass
of ConfigParser. The BandersnatchConfig singleton can be used anywhere a ConfigParser
instance is expected without having to use '.config' to access a nested ConfigParser.

Fixes pypa#1702
Fixes pypa#990
flyinghyrax added a commit to flyinghyrax/bandersnatch that referenced this issue May 28, 2024
This splits the existing 'default.conf' config file shipped with the package into two
similar files: "defaults.conf" and "example.conf". "example.conf" is an exact copy of the
previous "default.conf". The new "defaults.conf" is a stripped-down version containing
only default values for all mirror configuration options except "mirror.directory".

BandersnatchConfig is changed to *always* read defaults.conf, then read the user config
file if one is specified. This leaves the ConfigParser populated with default values for
any mirror options that aren't set by the user (except mirror.directory).

Notable ripple effects for this include:
- It is no longer meaningful to check ConfigParser.has_option with the 'mirror' section.
  Instead, you have to check whether the options value is empty or None.
- Specifying a default/fallback value when calling .get on the 'mirror' section will
  have no effect, because the option will already be present in the ConfigParser mappings.

As (mostly) an implementation detail, BandersnatchConfig is changed to be a subclass
of ConfigParser. The BandersnatchConfig singleton can be used anywhere a ConfigParser
instance is expected without having to use '.config' to access a nested ConfigParser.

Fixes pypa#1702
Fixes pypa#990
cooperlees added a commit that referenced this issue May 29, 2024
* Provide default values for mirror config options

This splits the existing 'default.conf' config file shipped with the package into two
similar files: "defaults.conf" and "example.conf". "example.conf" is an exact copy of the
previous "default.conf". The new "defaults.conf" is a stripped-down version containing
only default values for all mirror configuration options except "mirror.directory".

BandersnatchConfig is changed to *always* read defaults.conf, then read the user config
file if one is specified. This leaves the ConfigParser populated with default values for
any mirror options that aren't set by the user (except mirror.directory).

Notable ripple effects for this include:
- It is no longer meaningful to check ConfigParser.has_option with the 'mirror' section.
  Instead, you have to check whether the options value is empty or None.
- Specifying a default/fallback value when calling .get on the 'mirror' section will
  have no effect, because the option will already be present in the ConfigParser mappings.

As (mostly) an implementation detail, BandersnatchConfig is changed to be a subclass
of ConfigParser. The BandersnatchConfig singleton can be used anywhere a ConfigParser
instance is expected without having to use '.config' to access a nested ConfigParser.

Fixes #1702
Fixes #990

* Update mirror configuration documentation page

Add default values for options that are no longer required.

* Fix unnecessary concatenation in a string literal

Co-authored-by: Cooper Lees <[email protected]>

---------

Co-authored-by: Cooper Lees <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants