Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore HeLI models (use 2.0) #5

Merged
merged 15 commits into from
Oct 3, 2024
Merged

Restore HeLI models (use 2.0) #5

merged 15 commits into from
Oct 3, 2024

Commits on Oct 2, 2024

  1. Bring back HeLI language models (HeLI 2.0)

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    8f04dd7 View commit details
    Browse the repository at this point in the history

  2. Better handling of CJK 

    Only considered CJK chars the ones in the CJK_* blocks, which are only
Chinese and do not include Kana or Hangul. Not sure if it was the
intention, but it is the way HeLI does it.

Fixed a bug that was repeating the first non CJK character after CJK
group of characters.

More accurately compute the pct of cjk characters
    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7d220ec View commit details
    Browse the repository at this point in the history

  3. Load conficencethresholds

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    dab695a View commit details
    Browse the repository at this point in the history

  4. Use confidence values to alter prediction 

    Fix identify python func name
    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    3e85f16 View commit details
    Browse the repository at this point in the history

  5. Fix compilation for Rust < 1.80

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    143d05c View commit details
    Browse the repository at this point in the history

  6. Convert anyhow::Err's to PyErr

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    afcb6b7 View commit details
    Browse the repository at this point in the history

  7. Add TODO's for missing features in Python

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    75c21fd View commit details
    Browse the repository at this point in the history

  8. Collapse to macro to obtain confidence thresholds

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    6799a8b View commit details
    Browse the repository at this point in the history

  9. Print input text without modifications in debug

    @ZJaume
    ZJaume committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    a0b3dfe View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Update README with differences with HeLI

    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c9c46f7 View commit details
    Browse the repository at this point in the history

  2. Enable confidence by default. 

    Change option use_confidence by ignore_confidence
    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    d9525a0 View commit details
    Browse the repository at this point in the history

  3. Update identifier tests

    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c3288ce View commit details
    Browse the repository at this point in the history

  4. Compute confidence only with macrolangs collapsed

    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c6821f3 View commit details
    Browse the repository at this point in the history

  5. Allow loading model text from all languages or only a subset

    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9a4764f View commit details
    Browse the repository at this point in the history

  6. Update languagemodel tests

    @ZJaume
    ZJaume committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c7f14df View commit details
    Browse the repository at this point in the history