Stop swallowing errors when paginating #2903
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2394
Previously,
paginated_api_call
would swallow all exceptions (after logging a warning). Now, it raises, them, and let's the caller decide whether to swallow or re-raise them._remote_validation()
: re-raises (seems important to fail validation if invalid)fetch_attachments()
: logs warnings, but does not raisefetch_page_blog_documents()
: logs warnings, but does not raisefetch_spaces()
: re-raises (seems important to not skip whole spaces - that should fail the sync)search_by_query()
: re-raises (seems important to let you know if your advanced sync rules are incorrect)Also, I took a pass through the file to make sure that we were using
exc_info=exception
everywhere we were logging about an exception, instead of stuff likef"Exception: {exception}"
Checklists
Pre-Review Checklist
config.yml.example
)v7.13.2
,v7.14.0
,v8.0.0
)Related Pull Requests
Release Note
Improved the Confluence connector by ensuring that Confluence pagination API request errors bubble up when appropriate. This may result in more syncs ending in an error state, but these would not be new errors/issues, they will just be more visible and actionable.