fix: evaluate missing splits #1268
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isaac-chung
requested changes
Oct 1, 2024
There was a problem hiding this comment.
@thivyanth Thanks for taking a first stab at it! For any new functionality, we usually add a test / test cases to help confirm that the added code works. Could you please add a test for this? Specifically the following cases:
- Results exist, but no missing splits
- Results exist, and 1 missing split.
I can review the PR afterwards.
KennethEnevoldsen
requested changes
Oct 3, 2024
| if save_path.exists() and not overwrite_results: | ||
| logger.info( | ||
| f"{task.metadata.name} results already exists. Loading results from disk. Set overwrite_results=True to overwrite." | ||
| existing_results = self.load_existing_results(save_path) |
There was a problem hiding this comment.
you should use
MTEBResults.from_disk(path)
to load results.
(see line 354)
| logger.info( | ||
| f"{task.metadata.name} results exist but missing splits: {missing_splits}. Running evaluation for missing splits." | ||
| ) | ||
| task_eval_splits = missing_splits |
There was a problem hiding this comment.
This will overwrite the existing file further down resulting in a new file where the splits are missing. You will have to merge the results objects.
Comment on lines
+506
to
+510
| def load_existing_results(self, save_path): | ||
| if save_path.exists(): | ||
| with open(save_path) as f: | ||
| return json.load(f) | ||
| return None |
There was a problem hiding this comment.
Suggested change
| def load_existing_results(self, save_path): | |
| if save_path.exists(): | |
| with open(save_path) as f: | |
| return json.load(f) | |
| return None |
not needed see comment above
Comment on lines
+512
to
+517
| def compare_splits_and_subsets(self, existing_results, task_eval_splits): | ||
| missing_splits = [] | ||
| for split in task_eval_splits: | ||
| if split not in existing_results: | ||
| missing_splits.append(split) | ||
| return missing_splits |
There was a problem hiding this comment.
Suggested change
| def compare_splits_and_subsets(self, existing_results, task_eval_splits): | |
| missing_splits = [] | |
| for split in task_eval_splits: | |
| if split not in existing_results: | |
| missing_splits.append(split) | |
| return missing_splits | |
| @staticmethod | |
| def compare_splits_and_subsets(existing_results: MTEBResults, task_eval_splits: list[str]) -> list[str]: | |
| missing_splits = [] | |
| for split in task_eval_splits: # this will need to be adapted to MTEBResults object | |
| if split not in existing_results: | |
| missing_splits.append(split) | |
| return missing_splits |
isaac-chung
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #1260
Checklist
@Muennighoff @isaac-chung
make test.944 passed, 236 skipped, 55 warnings in 232.13s (0:03:52)make lint.