Can't ignore an unknown acronym only when it's written in upper case

[vadz]

I have a lot of occurrences of the word SEH in my code and I'd like codespell to leave it alone, but this doesn't seem to work, using neither -I nor -L option, e.g.:

% echo SEH | codespell -L SEH -
1: SEH
        SEH ==> SHE

The problem seems to be the case, i.e. if I use -L seh, it does recognize "SEH", but I'd prefer not to have to do it, as "seh" is much more likely to be a misspelling of "she" than anything else -- "SEH" itself is always written in upper case.

I'm not sure if it's the same as #1128, as it happens for camel case identifier there, while my problem doesn't seem to be related to camel case (or an identifier, for that matter, as the problematic acronym only occurs inside comments), sorry in advance if it's the same bug and so an already known problem.

P.S. This is using 1.17.1 from Debian unstable.

[peternewman]

Hi @vadz ,

Currently codespell is case-insensitive, so as per the readme:
https://github.com/codespell-project/codespell/blob/master/README.rst

The -I flag can be used for a list of certain words to allow that are in the codespell dictionaries. The format of the file is one word per line. Invoke using: codespell -I path/to/file.txt to execute codespell referencing said list of allowed words. Important note: The list passed to -I is case-sensitive based on how it is listed in the codespell dictionaries.

So you need to use -L seh.

Our dictionary checking needs to be case-insensitive because for example a C define is normally written in upper case so we'd want to catch SPELING as being an error without the dictionary needing to have both speling AND SPELING in it. I guess we could theoretically make the ignores match the case of the word in the code, but if you're using a term which codespell thinks is a typo you might have it as a define in capitals, a variable and possible also in camel case, so I think it would get rather complicated. I'll leave this open though, as it is different to the camel case request and another possible feature request.

[Holzhaus]

I have the same problem. I want to exclude InOut, but not inout (because that's possibly a typo and supposed to say input). Is there some way?

[Holzhaus]

How about filtering matches after looking them up in the dictionary?

[peternewman]

I have the same problem. I want to exclude InOut, but not inout (because that's possibly a typo and supposed to say input). Is there some way?

So you want to do it based on the case of the typo?

Currently -x FILE, --exclude-file FILE would be your only option.

How about filtering matches after looking them up in the dictionary?

Do you mean based on the correction? I don't really follow.

[Holzhaus]

I have the same problem. I want to exclude InOut, but not inout (because that's possibly a typo and supposed to say input). Is there some way?

So you want to do it based on the case of the typo?

Currently -x FILE, --exclude-file FILE would be your only option.

That would ignore the whole line and I'd also have to specify every line that this word occurs in, right?

How about filtering matches after looking them up in the dictionary?

Do you mean based on the correction? I don't really follow.

1. My code has something like char* pInOut;
2. I run codespell --ignore-regex "\\Wp(?=[A-Z]\\w*\\W)" to strip out the p prefix for pointers
3. codespell warns InOut => input, in out. This is where we could filter using a case-sensitive whiltelist (containing InOut in this case)

Using -L InOut does not work and -L inout is not feasible because I only want to ignore words with that that exact case.

Right now I worked around it by writing a small wrapper that reads a .codespellignore file and creates an ignore-regex pattern based on that, but that's probably not really performant and seems "hacky" to me.

[peternewman]

That would ignore the whole line and I'd also have to specify every line that this word occurs in, right?

Yes, not ideal but quite easy with grep. I do it quite a lot:
https://github.com/OpenLightingProject/ola/blob/master/.codespellignorelines

It's not perfect, but for a lot of edge cases, like uint/unit, even case sensitivity wouldn't help.

Actually for this specific use case, the ignore regex is probably perfect:
https://github.com/codespell-project/codespell/blob/master/codespell_lib/_codespell.py#L277-L283

Doh, just realised you're doing that already! You can vanish pInOut before you even start matching things. Nice trick with stripping the p's off the front by the way!

Care to share your wrapper @Holzhaus as it might help others.

@vadz I think the above would work for you too?

3. codespell warns `InOut => input, in out`. This is where we could filter using a case-sensitive whiltelist (containing `InOut` in this case)

Understood. So you're talking about filtering based on the case of the MATCH rather than the dictionary entry. You probably either want two whitelists, or some way to flag whether each entry should be case sensitive or not. It's certainly possible!

[Holzhaus]

Care to share your wrapper @Holzhaus as it might help others.

mixxxdj/mixxx#3066

The wrapper also parses the git diff and filters out all results that are not coming from added lines (because our legacy codebase is quite large, and fixing other peoples typos in merge commits is annoying).

[vadz]

@peternewman It looks like it ought to work, but I couldn't test it because of this:

% pip install git+https://github.com/codespell-project/codespell@293f12152f7a6cef6e87cd6c78d8aee69db7af28
Collecting git+https://github.com/codespell-project/codespell@293f12152f7a6cef6e87cd6c78d8aee69db7af28
  Cloning https://github.com/codespell-project/codespell (to revision 293f12152f7a6cef6e87cd6c78d8aee69db7af28) to /tmp/pip-req-build-8HAgKz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-8HAgKz/setup.py", line 10, in <module>
        from codespell_lib import __version__
      File "codespell_lib/__init__.py", line 1, in <module>
        from ._codespell import main, _script_main, VERSION as __version__  # noqa
      File "codespell_lib/_codespell.py", line 24, in <module>
        import configparser
    ImportError: No module named configparser

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-8HAgKz/

I didn't look into it in details, so it's perfectly possible that I'm doing something wrong, but I couldn't actually test your suggestion...

[larsoner]

This is a builtin module going back to 2.7:

https://docs.python.org/2.7/library/configparser.html

Are you on 2.6? It's unsupported

[Holzhaus]

You need at least Python 3 I think. In python 2.7 the module has another name (ConfigParser instead of configparser) and Python 2 has reached end-of-life months ago anyway.

[vadz]

Oops, sorry, pip indeed uses 2.7.16 by default on my Debian 10 (Buster) system, I keep forgetting that I need to use pip3 explicitly. Its output was rather weird anyhow:

Collecting git+https://github.com/codespell-project/codespell@293f12152f7a6cef6e87cd6c78d8aee69db7af28
  Cloning https://github.com/codespell-project/codespell (to revision 293f12152f7a6cef6e87cd6c78d8aee69db7af28) to /tmp/pip-req-build-5la8ce8k
Building wheels for collected packages: codespell
  Running setup.py bdist_wheel for codespell ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-5la8ce8k/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-wkwq0mmi --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for codespell
  Running setup.py clean for codespell
Failed to build codespell
Installing collected packages: codespell
  Running setup.py install for codespell ... done
Successfully installed codespell-2.0.dev0

with the "Failed" line in ominous red, but it seems to have been installed successfully and I can confirm that adding --ignore-regex '\WSEH\W' to the command line (and removing the suppression lines) works for me, thanks.

[Holzhaus]

You need to install wheel before installation:

pip3 install wheel

[peternewman]

Oops, sorry, pip indeed uses 2.7.16 by default on my Debian 10 (Buster) system, I keep forgetting that I need to use pip3 explicitly.

I think this should do the trick for you:
sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1

[peterjc]

This works OK to ignore three letter upper case words (i.e. many acronyms):

codespell --ignore-regex '\W[A-Z][A-Z][A-Z]\W'

Or being more clever, 2 or 3 letter acronyms:

codespell --ignore-regex '\W([A-Z]{2,3})\W'

However, this fails with consecutive words to be ignored via the regex, see #2049 for a potential solution.

[LilithHafner]

I think this is a pretty reasonable workflow:

1. run codespell
2. see a false positive of the form SEH ==> SHE
3. add SEH to the ignore list
4. run codespell
5. see a false positive of the form SEH ==> SHE
6. be sad

I'd like steps 5 and 6 to be different. In order of preference:

1. SEH in the ignore list matches SEH in code
2. SEH in the ignore list does nothing, but a warning/error is produced that there is a useless entry in the ignore list (which says why it's useless)
3. SEH in the ignore list does nothing and there is no warning/error at all (current situation)

[DimitriPapadopoulos]

Pull requests are welcome, but make sure you address components such as #1578 (comment).

[vEnhance]

Pull requests are welcome, but make sure you address components such as #1578 (comment).

Just to be clear, would you consider a pull request where:

• when a word is written in all lowercase in the ignore list, the behavior remains the same (so seh would match seh, Seh, SEH, etc.), but
• if a word is written with a specific casing in the ignore list, then it matches only that specific case, (so SEH matches only SEH), rather than the current behavior of doing nothing with no warning.

I believe this is also the suggestion in #1578 (comment)

[DimitriPapadopoulos]

I am not the maintainer, but I think that would be a good, long-awaited change. I am only worried about compatibility. I don't have a clear idea (yet?) of what could possibly go wrong in existing codebases when we release such a change. Comments on that?

Also, what about words in the dictionary themselves that are not all lowercase? Such as Feburary? Should the case have some special meaning? I am referring to the typo, of course, not to the suggested fixes which should keep the uppercase as required.

grep '^[A-Z]'

$ grep '^[A-Z]' codespell_lib/data/dictionary*.txt
codespell_lib/data/dictionary_code.txt:MSDOS->MS-DOS
codespell_lib/data/dictionary.txt:ACI->ACPI
codespell_lib/data/dictionary.txt:ACII->ASCII
codespell_lib/data/dictionary.txt:Amercia->America
codespell_lib/data/dictionary.txt:Ardiuno->Arduino
codespell_lib/data/dictionary.txt:Australien->Australian
codespell_lib/data/dictionary.txt:Austrlaian->Australian
codespell_lib/data/dictionary.txt:Berkley->Berkeley
codespell_lib/data/dictionary.txt:Bernouilli->Bernoulli
codespell_lib/data/dictionary.txt:Blitzkreig->Blitzkrieg
codespell_lib/data/dictionary.txt:Bonnano->Bonanno
codespell_lib/data/dictionary.txt:Brasillian->Brazilian
codespell_lib/data/dictionary.txt:Britian->Britain
codespell_lib/data/dictionary.txt:Brittish->British
codespell_lib/data/dictionary.txt:Buddah->Buddha
codespell_lib/data/dictionary.txt:Buddist->Buddhist
codespell_lib/data/dictionary.txt:Caesarian->Caesarean
codespell_lib/data/dictionary.txt:Cambrige->Cambridge
codespell_lib/data/dictionary.txt:Capetown->Cape Town
codespell_lib/data/dictionary.txt:Carmalite->Carmelite
codespell_lib/data/dictionary.txt:Carnagie->Carnegie
codespell_lib/data/dictionary.txt:Carnagie-Mellon->Carnegie-Mellon
codespell_lib/data/dictionary.txt:Carnigie->Carnegie
codespell_lib/data/dictionary.txt:Carnigie-Mellon->Carnegie-Mellon
codespell_lib/data/dictionary.txt:Carribbean->Caribbean
codespell_lib/data/dictionary.txt:Carribean->Caribbean
codespell_lib/data/dictionary.txt:Carthagian->Carthaginian
codespell_lib/data/dictionary.txt:Cataline->Catiline, Catalina,
codespell_lib/data/dictionary.txt:Ceasar->Caesar
codespell_lib/data/dictionary.txt:Celcius->Celsius
codespell_lib/data/dictionary.txt:Cgywin->Cygwin
codespell_lib/data/dictionary.txt:Champange->Champagne
codespell_lib/data/dictionary.txt:Cincinatti->Cincinnati
codespell_lib/data/dictionary.txt:Cincinnatti->Cincinnati
codespell_lib/data/dictionary.txt:Conneticut->Connecticut
codespell_lib/data/dictionary.txt:Cyrllic->Cyrillic
codespell_lib/data/dictionary.txt:Dardenelles->Dardanelles
codespell_lib/data/dictionary.txt:DCHP->DHCP
codespell_lib/data/dictionary.txt:Decemer->December
codespell_lib/data/dictionary.txt:Dravadian->Dravidian
codespell_lib/data/dictionary.txt:EBCIDC->EBCDIC
codespell_lib/data/dictionary.txt:EDCDIC->EBCDIC
codespell_lib/data/dictionary.txt:Enlish->English, enlist,
codespell_lib/data/dictionary.txt:Europian->European
codespell_lib/data/dictionary.txt:Europians->Europeans
codespell_lib/data/dictionary.txt:Eurpean->European
codespell_lib/data/dictionary.txt:Eurpoean->European
codespell_lib/data/dictionary.txt:Fahrenheight->Fahrenheit
codespell_lib/data/dictionary.txt:Farenheight->Fahrenheit
codespell_lib/data/dictionary.txt:Farenheit->Fahrenheit
codespell_lib/data/dictionary.txt:Febuary->February
codespell_lib/data/dictionary.txt:Feburary->February
codespell_lib/data/dictionary.txt:Flemmish->Flemish
codespell_lib/data/dictionary.txt:Formalhaut->Fomalhaut
codespell_lib/data/dictionary.txt:Foundland->Newfoundland
codespell_lib/data/dictionary.txt:Fransiscan->Franciscan
codespell_lib/data/dictionary.txt:Fransiscans->Franciscans
codespell_lib/data/dictionary.txt:FTBS->FTBFS
codespell_lib/data/dictionary.txt:Galations->Galatians
codespell_lib/data/dictionary.txt:Gameboy->Game Boy
codespell_lib/data/dictionary.txt:Ghandi->Gandhi
codespell_lib/data/dictionary.txt:Godounov->Godunov
codespell_lib/data/dictionary.txt:Gothenberg->Gothenburg
codespell_lib/data/dictionary.txt:Gottleib->Gottlieb
codespell_lib/data/dictionary.txt:Guaduloupe->Guadalupe, Guadeloupe,
codespell_lib/data/dictionary.txt:Guadulupe->Guadalupe, Guadeloupe,
codespell_lib/data/dictionary.txt:Guatamala->Guatemala
codespell_lib/data/dictionary.txt:Guatamalan->Guatemalan
codespell_lib/data/dictionary.txt:Guilia->Giulia
codespell_lib/data/dictionary.txt:Guilio->Giulio
codespell_lib/data/dictionary.txt:Guiness->Guinness
codespell_lib/data/dictionary.txt:Guiseppe->Giuseppe
codespell_lib/data/dictionary.txt:Habsbourg->Habsburg
codespell_lib/data/dictionary.txt:Hallowean->Hallowe'en, Halloween,
codespell_lib/data/dictionary.txt:Hatian->Haitian
codespell_lib/data/dictionary.txt:Heidelburg->Heidelberg
codespell_lib/data/dictionary.txt:Hyperldger->Hyperledger
codespell_lib/data/dictionary.txt:I'sd->I'd
codespell_lib/data/dictionary.txt:Ihaca->Ithaca
codespell_lib/data/dictionary.txt:ISNB->ISBN
codespell_lib/data/dictionary.txt:Israelies->Israelis
codespell_lib/data/dictionary.txt:Istambul->Istanbul
codespell_lib/data/dictionary.txt:Japanes->Japanese
codespell_lib/data/dictionary.txt:Johanine->Johannine
codespell_lib/data/dictionary.txt:Jospeh->Joseph
codespell_lib/data/dictionary.txt:Laotion->Laotian, lotion,
codespell_lib/data/dictionary.txt:Laotions->Laotians, lotions,
codespell_lib/data/dictionary.txt:Linix->Linux
codespell_lib/data/dictionary.txt:Lybia->Libya
codespell_lib/data/dictionary.txt:Malcom->Malcolm
codespell_lib/data/dictionary.txt:Massachusettes->Massachusetts
codespell_lib/data/dictionary.txt:Massachussets->Massachusetts
codespell_lib/data/dictionary.txt:Massachussetts->Massachusetts
codespell_lib/data/dictionary.txt:Mediteranean->Mediterranean
codespell_lib/data/dictionary.txt:Michagan->Michigan
codespell_lib/data/dictionary.txt:Micorsoft->Microsoft
codespell_lib/data/dictionary.txt:Micosoft->Microsoft
codespell_lib/data/dictionary.txt:Microfost->Microsoft
codespell_lib/data/dictionary.txt:Microsfoft->Microsoft
codespell_lib/data/dictionary.txt:Microsft->Microsoft
codespell_lib/data/dictionary.txt:Microsof->Microsoft
codespell_lib/data/dictionary.txt:Microsofot->Microsoft
codespell_lib/data/dictionary.txt:Micrsft->Microsoft
codespell_lib/data/dictionary.txt:Micrsoft->Microsoft
codespell_lib/data/dictionary.txt:MingGW->MinGW
codespell_lib/data/dictionary.txt:Miscrosoft->Microsoft
codespell_lib/data/dictionary.txt:Misouri->Missouri
codespell_lib/data/dictionary.txt:Missisipi->Mississippi
codespell_lib/data/dictionary.txt:Missisippi->Mississippi
codespell_lib/data/dictionary.txt:Mississipi->Mississippi
codespell_lib/data/dictionary.txt:Mocrosoft->Microsoft
codespell_lib/data/dictionary.txt:Monserrat->Montserrat
codespell_lib/data/dictionary.txt:Montnana->Montana
codespell_lib/data/dictionary.txt:Mythraic->Mithraic
codespell_lib/data/dictionary.txt:Naploeon->Napoleon
codespell_lib/data/dictionary.txt:Napolean->Napoleon
codespell_lib/data/dictionary.txt:Napoleonian->Napoleonic
codespell_lib/data/dictionary.txt:Nazereth->Nazareth
codespell_lib/data/dictionary.txt:Newyorker->New Yorker
codespell_lib/data/dictionary.txt:Novemer->November
codespell_lib/data/dictionary.txt:Novermber->November
codespell_lib/data/dictionary.txt:Nullabour->Nullarbor
codespell_lib/data/dictionary.txt:Nuremburg->Nuremberg
codespell_lib/data/dictionary.txt:Octobor->October
codespell_lib/data/dictionary.txt:Palistian->Palestinian
codespell_lib/data/dictionary.txt:Palistinian->Palestinian
codespell_lib/data/dictionary.txt:Palistinians->Palestinians
codespell_lib/data/dictionary.txt:Papanicalou->Papanicolaou
codespell_lib/data/dictionary.txt:Peloponnes->Peloponnese, Peloponnesus,
codespell_lib/data/dictionary.txt:Pennyslvania->Pennsylvania
codespell_lib/data/dictionary.txt:Philipines->Philippines
codespell_lib/data/dictionary.txt:Phillipine->Philippine
codespell_lib/data/dictionary.txt:Phillippines->Philippines
codespell_lib/data/dictionary.txt:Phonecian->Phoenecian
codespell_lib/data/dictionary.txt:POSIX-complient->POSIX-compliant
codespell_lib/data/dictionary.txt:Postdam->Potsdam
codespell_lib/data/dictionary.txt:Premonasterians->Premonstratensians
codespell_lib/data/dictionary.txt:Pucini->Puccini
codespell_lib/data/dictionary.txt:Puertorrican->Puerto Rican
codespell_lib/data/dictionary.txt:Puertorricans->Puerto Ricans
codespell_lib/data/dictionary.txt:Queenland->Queensland
codespell_lib/data/dictionary.txt:Rockerfeller->Rockefeller
codespell_lib/data/dictionary.txt:Russion->Russian
codespell_lib/data/dictionary.txt:Sacremento->Sacramento
codespell_lib/data/dictionary.txt:Sagitarius->Sagittarius
codespell_lib/data/dictionary.txt:Sanhedrim->Sanhedrin
codespell_lib/data/dictionary.txt:Septemer->September
codespell_lib/data/dictionary.txt:Sionist->Zionist
codespell_lib/data/dictionary.txt:Sionists->Zionists
codespell_lib/data/dictionary.txt:Sixtin->Sistine, Sixteen,
codespell_lib/data/dictionary.txt:Skagerak->Skagerrak
codespell_lib/data/dictionary.txt:Soalris->Solaris
codespell_lib/data/dictionary.txt:Thse->these, this,
codespell_lib/data/dictionary.txt:Tolkein->Tolkien
codespell_lib/data/dictionary.txt:Toosday->Tuesday
codespell_lib/data/dictionary.txt:Tosday->Tuesday, today,
codespell_lib/data/dictionary.txt:Tuscon->Tucson
codespell_lib/data/dictionary.txt:Twosday->Tuesday
codespell_lib/data/dictionary.txt:Ukranian->Ukrainian
codespell_lib/data/dictionary.txt:Unicde->Unicode
codespell_lib/data/dictionary.txt:UnitesStates->UnitedStates
codespell_lib/data/dictionary.txt:UTF8ness->UTF-8-ness
codespell_lib/data/dictionary.txt:Voight->Voigt
codespell_lib/data/dictionary.txt:Vulacn->Vulcan
codespell_lib/data/dictionary.txt:Vulakn->Vulkan
codespell_lib/data/dictionary.txt:Yementite->Yemenite, Yemeni,

@larsoner Would you agree to something like #1578 (comment)?

[larsoner]

Sure, and if we're really worried about backward compat we can make the behavior opt in with --case-sensitive or something

[vEnhance]

Interestingly, it seems like --uri-ignore-words-list has a completely different behavior from the "normal" words list:

$ codespell --version
2.2.6

$ cat hi.txt
https://en.wikipedia.org/wiki/acsii
https://en.wikipedia.org/wiki/ACSII

$ codespell hi.txt
hi.txt:1: acsii ==> ascii
hi.txt:2: ACSII ==> ASCII

$ codespell hi.txt --uri-ignore-words-list acsii
hi.txt:2: ACSII ==> ASCII

$ codespell hi.txt --uri-ignore-words-list ACSII
hi.txt:1: acsii ==> ascii

$ codespell hi.txt --uri-ignore-words-list ACSII,acsii

$ codespell hi.txt -L acsii

$ codespell hi.txt -L ACSII
hi.txt:1: acsii ==> ascii
hi.txt:2: ACSII ==> ASCII

Meanwhile, the documentation under codespell --help doesn't seem to hint at this difference:

  -I FILE, --ignore-words FILE
                        file that contains words that will be ignored by codespell.
                        File must contain 1 word per line. Words are case sensitive
                        based on how they are written in the dictionary file
  -L WORDS, --ignore-words-list WORDS
                        comma separated list of words to be ignored by codespell.
                        Words are case sensitive based on how they are written in
                        the dictionary file
  --uri-ignore-words-list WORDS
                        comma separated list of words to be ignored by codespell in
                        URIs and emails only. Words are case sensitive based on how
                        they are written in the dictionary file. If set to "*", all
                        misspelling in URIs and emails will be ignored.

I'm going to focus on the "normal" ignore list for this issue/PR and leave the behavior for URI unchanged I guess, but maybe the help should be reworded too?

[DimitriPapadopoulos]

For now, let's focus on improving the behaviour of codespell. Then we can improve the documentation.

It seems impossible to provide decent, comprehensible documentation with the current behaviour. See discussion in #3210 and specifically #3210 (comment).