Note: ULTRA is developed under Python 3.6.
Welcome! ULTRA is a community project that aims to support research on unbiased learning to rank and online learning to rank. Your experience and what you can contribute are important to the project's success.
If you run into unexpected behavior in ULTRA, have trouble on applying it to your code, or find a bug or a feature you would like to get, please do not hesitate to contact us!
The main forum for discussion is the project's GitHub issue tracker. This is the right place to start a discussion of any of the above or most any other topic concerning the project.
ULTRA appreciates your contribution! If you are interested in contributing to ULTRA, there are several ways to get started:
- Work on new data simulation/preparation paradigm.
- Work on new unbiased/online learning algorithm.
- Work on new learning-to-rank model.
- Work on new ranking metric/utils.
- Work on documentation.
- Try to answer questions on the issue tracker.
We use the standard GitHub pull-request flow, which may be familiar to you if you've contributed to other projects on GitHub -- see blow.
Anyone interested in ULTRA may review your code. One of the core developers will check and merge your pull request when they think it's ready. If you go a few days without a reply, please feel free to ping the thread by adding a new comment.
A list of ULTRA core developers can be found in Readme.
- Fork the latest version of ULTRA into your repo.
- Create an issue under ULTR-Community/ULTRA/, write description about the bug/enhancement.
- Clone your forked ULTRA into your machine, add your changes.
- Create json files in tests/test_settings/ (e.g.,tests/test_settings/test.json) to test models that have been changed. Please create separate json files to test each changed model.
- Run
make test
to ensure all tests passed on your computer. - Run
make format
to use autopep8 to format your code. - Push to your forked repo, then send the pull request to the official repo. In pull request, you need to create a link to the issue you created using
#[issue_id]
, and describe what has been changed. - We'll assign reviewers to review your code.
Your PR will be merged if:
- Funcitonally benefit for the project.
- With proper docstrings, see codebase as examples.
- All reviewers approved your changes.
Thanks and let's improve ULTRA together!