Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Outlook] Feature: Office365 multi-user support #2844

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

ilyasabdellaoui
Copy link

Introduced BaseOffice365User abstract base class to standardize Office 365 user handling. Added MultiOffice365Users to manage multiple emails from config. Added client_emails (comma-separated) in OutlookDataSource config. Resolved issue with fetching too many users causing SMTP server not found error.

Closes #2843

Checklists

Pre-Review Checklist

  • this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • this PR has a thorough description
  • Covered the changes with automated tests
  • Tested the changes locally
  • Considered corresponding documentation changes
  • Contributed any configuration settings changes to the configuration reference
  • if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

@ilyasabdellaoui ilyasabdellaoui requested a review from a team as a code owner September 23, 2024 02:07
Copy link

cla-checker-service bot commented Sep 23, 2024

💚 CLA has been signed

@ilyasabdellaoui ilyasabdellaoui marked this pull request as draft September 23, 2024 02:14
@ilyasabdellaoui ilyasabdellaoui marked this pull request as ready for review September 23, 2024 02:15
Introduced BaseOffice365User abstract base class to standardize Office 365 user handling.
Added MultiOffice365Users to manage multiple emails from config.
Added client_emails (comma-separated) in OutlookDataSource config.
Resolved issue with fetching too many users causing SMTP server not found error.
@ilyasabdellaoui ilyasabdellaoui force-pushed the feature/office365-multi-user-support branch from 37f7483 to 897fed8 Compare September 23, 2024 02:32
@seanstory
Copy link
Member

buildkite test this

Copy link
Member

@seanstory seanstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thanks for submitting a patch! I've left a few comments, but overall I think this is a pretty clean approach and solves a clear ask.

Outside of the below comments, I'd also request that you change the

Closes #2843

in your PR description to use "Relates to" or "Part of", since another part of the issue you filed is that the current solution is limited to 999 users. I expect that's something we can better control with the graph API and another separate configuration, for folks who want "all users", and have more than 999.

connectors/sources/outlook.py Show resolved Hide resolved
connectors/sources/outlook.py Show resolved Hide resolved
):
async for response in source.client._get_user_instance.get_users():
user_mails = [user["mail"] for user in response["value"]]
users.extend(user_mails)
assert users == ["[email protected]", "[email protected]"]

client_emails = "[email protected]"
async with create_outlook_source(client_emails=client_emails) as source:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest making these their own test functions. That way, any failures are more easily identifyable, and they can have more focused test names to help the reader understand what's being tested.

connectors/sources/outlook.py Outdated Show resolved Hide resolved
connectors/sources/outlook.py Outdated Show resolved Hide resolved
connectors/sources/outlook.py Outdated Show resolved Hide resolved
@seanstory
Copy link
Member

Screenshot 2024-09-24 at 3 45 50 PM

The tests pass, but the linter is currently failing. You can fix this by running make autoformat when you're done making changes.

@ilyasabdellaoui ilyasabdellaoui requested a review from a team as a code owner September 29, 2024 12:24
@ilyasabdellaoui
Copy link
Author

ilyasabdellaoui commented Sep 29, 2024

in your PR description to use "Relates to" or "Part of", since another part of the issue you filed is that the current solution is limited to 999 users. I expect that's something we can better control with the graph API and another separate configuration, for folks who want "all users", and have more than 999.

@seanstory After rechecking the code, I realized I was mistaken in my initial description. The code isn't limited to fetching the top 999 users—it handles pagination using the @odata.nextLink property from the Microsoft Graph API, which retrieves all users. I missed that detail. I'll update my issue description to reflect the correct behavior.

async def get_users(self):
access_token = await self._fetch_token()
url = f"https://graph.microsoft.com/v1.0/users?$top={TOP}"
while True:
try:
async with self._get_session.get(
url=url,
headers={
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json",
},
) as response:
json_response = await response.json()
yield json_response
url = json_response.get("@odata.nextLink")
if url is None:
break
except Exception:
raise

@ilyasabdellaoui ilyasabdellaoui marked this pull request as draft September 29, 2024 13:25
…Cloud connector

- Refactored the  method to utilize Microsoft Graph's JSON batching feature.
- The method now processes client emails in batches of up to 20 to optimize API requests.
- Added error handling for individual user fetch errors within the batch response.
- If any batch request fails, the method collects the errors and raises an exception with details.
@seanstory
Copy link
Member

buildkite test this

@ilyasabdellaoui
Copy link
Author

ilyasabdellaoui commented Sep 30, 2024

buildkite test this

Lint still fails due to async-related issues. Autoformat doesn't solve the lint errors. I'll investigate further after completing test functions. If unable to resolve, I'll seek assistance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enhancement: Allow Outlook Cloud connector to specify client emails instead of defaulting to all users
2 participants