M365 troubleshooting.
Step 1: Verify prerequisites.
Usually when we have issues with the M365 integration, it is because the partner did not set up the integration correctly. Read over the prerequisites document and watch the video.
Common mistakes are:
- Not having MFA set up for the service account, or not using Microsoft Authenticator for MFA. The MFA for this user cannot be performed through a third-party application, such as Duo.
- Not having the correct roles and groups:
- The integrating service account must be assigned the Global Administrator role for initial setup.
- The integrating service account must be part of two security groups: the AdminAgents security group and the security group assigned to the GDAP relationship
- GDAP relationship being made from the tenant page. The relationship must be made from Home > Customers > Administer.
- GDAP relationship not having the group assigned to it, or not having one of the following roles: Global Admin, Cloud Application Admin, or Application Admin.
Message to partner:
Usually when we see tenant issues, it is due to the granular delegated admin privileges (GDAP) not being set up correctly within Microsoft Partner Center. The CSP tenant must have a valid granular delegated admin privileges (GDAP) relationship set up for each of its client tenants. Each GDAP relationship must have a security group assigned, and the integrating service account must be part of that security group. The group must be granted one of the following roles: Global Admin, Cloud Application Admin, or Application Admin.
Please make sure all the steps within https://rmm.datto.com/help/en/Content/3NEWUI/Setup/Integrations/Microsoft365Integration.htm#Prerequisites are completed. Our documentation team has produced a new video embedded within that page to help our partners set up the Microsoft 365 Integration. The video is only 5 minutes, it will take you through all the requirements and prerequisites to synchronize your tenants.
Should you require more information about GDAP, here are some articles that you may find useful:
https://learn.microsoft.com/en-us/partner-center/gdap-introduction
https://tminus365.com/adding-gdap-relationships/
https://tminus365.com/vendor-integrations-break-with-gdap-the-fix/
Note: The relationship needs to be made from the Administer page, not from the tenant page:
Step 2: Reauthenticate the integration.
The steps below are from the 12.5 release notes.
Message to partner:
Please reauthenticate the integration using the steps below:
- Navigate to Setup > Integrations > Microsoft 365.
- In the Authenticate tab selected by default, click Turn Off.
- Turn the integration back on, insert a dummy value in the Tenant ID field, and click Save. You will receive a message stating, "Unable to save Microsoft 365 configuration details. Please try again."
- Enter your real tenant ID and click Save. Initially, the Tenants tab will be empty, but the list of tenants should appear within five minutes.
- Re-sync your tenants. For instructions, refer to Sync Microsoft 365 client tenants to Datto RMM.
- Return to the All Tenants and All Users pages in the Microsoft 365 menu, which may also require up to five minutes to populate results.
Step 3: Recreate the GDAP.
Because the GDAP is the main cause of issues, for troubleshooting, it is best to have the partner recreate the GDAP.
Message to the partner:
Please try removing the GDAP entirely from the tenant within the Microsoft portal and add a new one back in. Note, the GDAP needs to be created under Microsoft Partner Center > Customers > Administer > Request admin relationship, and not under the tenant itself.
Then, resync the tenants that are showing as pending. You can do so with the following instructions: https://rmm.datto.com/help/en/Content/3NEWUI/Setup/Integrations/Microsoft365Integration.htm#Unsync_Microsoft_365_client_tenants_from_Datto_RMM
Step 4: Reset the integration.
Message to the partner:
Can you please reset your M365 integration? You can do so with the following steps:
- Turn off the integration in Datto RMM
- Log into Microsoft Entra for your client tenant
- Locate "Applications" in the left hand menu and open the blade and select "Enterprise applications"
- Remove the preset filter in the table and search for "Datto RMM"
- Select the application named "Datto RMM integration"
- Go to the "Properties" Page and select "Delete" at the top of that page
- Repeat Steps 2-6 for each tenant
- Turn the integration back on, insert a dummy value in the Tenant ID field, and click Save. You will receive a message stating, "Unable to save Microsoft 365 configuration details. Please try again."
- Enter your real tenant ID and click Save. Initially, the Tenants tab will be empty, but the list of tenants should appear within five minutes.
- Sync the tenants
Step 5: Check Kibana for errors.
Within Kibana perform the following searches for the timeframe of the failed syncs:
- “<account ID>” AND “M365”
- “<Tenant ID>”
You can get the account ID from the partner’s support access, and the tenant ID from the tenants page. If the tenants page is not populating, you can ask the partner for it. This applies to tenant sync, missing tenants, or missing users.
Look for any errors in the results. Here are several example error messages:
Message: Unable to find tenant info. ApiError (Forbidden, { Error = { Code = "Authorization_RequestDenied" Message = "Insufficient privileges to complete the operation." Details = None InnerError = Some { Date = 5/3/2024 1:57:04 AM RequestId = "cf0247be-0d10-4480-9fd6-67d079d25e34" ClientRequestId = "cf0247be-0d10-4480-9fd6-67d079d25e34" } } })
Message: Unable to find Risk Detections. ApiError (Forbidden, { Error = { Code = "AccessDenied" Message = "Your account does not have access to this report or data. Please contact your global administrator to request access. One of the following roles is required: Company Administrator, Security Administrator, Security Reader, Security Operator, Global Reader." Details = None InnerError = Some { Date = 5/3/2024 1:57:06 AM RequestId = "897cbd15-0122-439b-abe0-417571738751" ClientRequestId = "897cbd15-0122-439b-abe0-417571738751" } } })
Message: Failed to create access token Unauthorized "AADSTS65001: The user or administrator has not consented to use the application with ID '82b6e081-6213-4500-b348-791ee3e24389' named 'Datto RMM integration'. Send an interactive authorization request for this user and resource. Trace ID: 522f1d72-1f3e-4b66-9cd5-2deaaa142600 Correlation ID: 6861590a-f055-46e8-90e4-fb3cf1ca1deb Timestamp: 2024-05-03 02:02:44Z"
Message: Unable to request consent. DecoderError "The following errors were found:Error at: `$`Expecting an object but instead got:nullError at: `$`Expecting an object but instead got:nullError at: `$`Expecting an object but instead got:nullError at: `$`Expecting an object but instead got:nullError at: `$`Expecting an object but instead got:null"
Message: Unable to request consent. PartnerCenterApiError (Forbidden, { Code = 403 Message = "Exception of type 'Providers.Common.V1.CoreException' was thrown." Description = "Exception of type 'Providers.Common.V1.CoreException' was thrown." ErrorName = "Forbidden" IsRetryable = false }, "27052412-bfad-4c22-ad60-26e1c64c59e1")
Message: Unable to request consent. PartnerCenterApiError (BadRequest, { Code = 400 Message = "{ "error": { "code": "Request_BadRequest", "message": "Unsupported token. Unable to initialize the authorization context.", "innerError": { "date": "2024-02-09T09:00:54", "request-id": "e387efa9-8854-4939-8d5c-058dcc20c50b", "client-request-id": "e387efa9-8854-4939-8d5c-058dcc20c50b" } }}" Description = "{ "error": { "code": "Request_BadRequest", "message": "Unsupported token. Unable to initialize the authorization context.", "innerError": { "date": "2024-02-09T09:00:54", "request-id": "e387efa9-8854-4939-8d5c-058dcc20c50b", "client-request-id": "e387efa9-8854-4939-8d5c-058dcc20c50b" } }}" ErrorName = "BadRequest" IsRetryable = false }, "d3e84692-7a4e-431e-8683-8fcd36aa67b7")
Message: Unable to request consent. PartnerCenterApiError (BadRequest, { Code = 400 Message = "{ "type": "MsalServiceException", "error_code": "invalid_request", "error_description": "AADSTS530034: A delegated administrator was blocked from accessing the tenant due to account risk. Trace ID: f2719bb6-6a35-461f-938c-ad0877bbf500 Correlation ID: f7bf1316-d944-46e1-9f74-fb607ea69790 Timestamp: 2024-03-28 07:29:24Z",
Note, you can also view surrounding documents to see message that came in at the same time as the error, but not part of your search term.
Step 6: Check if the issue matches current problem cases.
Current M365 problem cases are:
M365 Open Problem cases:
- Problem #5315045 - [Datto RMM] M365 - Some tenants not syncing successfully
- Problem #5268330 - [Datto RMM] M365 - Syncing tenant does not show the number of users expected
- Problem #5314989 - [Datto RMM][M365] All Tenants page does not show any Tenants (none found) but shows the correct number under Integrations page
- Problem #5315070 - [Datto RMM] Microsoft 365 page > All Users > hyperlink to open M365 - only the original user who configured the integration is able to use hyperlink
- Problem #5323186 - [Datto RMM][M365] Tenants intermittently fail to sync when using the Cloud Application Admin role in the GDAP relationship
- Problem #5196102 - [Datto RMM][M365] Microsoft 365 Integration not displaying user last sign in date
Check each of these problem cases for the steps to reproduce and validation criteria. And gather for your case.
Step 7: Prepare to send up to Elevated support.
In the sendup we need the following information
- Partners support access.
- Screenshots:
- Screenshots of the service account showing it assigned the Global Administrator role, and MFA enabled.
- Screenshots showing the service account is in both the AdminAgents security group and the security group that is assigned to the GDAP relationship
- Screenshot of GDAP Admin relationship of a tenant
- Screenshot of the API permissions. In Azure portal: App registrations > All applications > Application used with integration > API permissions
- Screenshots showing that the tenants were unsynced and synced again
- Timestamps of when M365 was reset or tenants were resynced
- Tenant Name and Tenant ID
- Links to Kibana searches. (Remember to change timeframe to Absolute for the shared link)
- (If applicable) Screenshots of the tenant pages and user’s pages
Message to the partner:
In order to further investigate the M365 sync issues, we will need the following information:
- Screenshots of the service account showing it assigned the Global Administrator role, and MFA enabled.
- Screenshots showing the service account is in both the AdminAgents security group and the security group that is assigned to the GDAP relationship
- Tenant name (if not already provided)
- Screenshot of GDAP Admin relationship of a tenant from within your Microsoft Portal: Microsoft Partner Center > Customers > Administer > select a customer > admin relationship. Please include the security groups showing the GDAP has the security group that the service account is in, and the group had one of the following roles: Global Admin, Cloud Application Admin, or Application Admin.
- Screenshot of the API permissions. In Azure portal: App registrations > All applications > Application used with integration > API permissions
- Screenshots showing that the tenants were unsynced and synced again. You can unsync devices using the steps outlined here: https://rmm.datto.com/help/en/Content/3NEWUI/Setup/Integrations/Microsoft365Integration.htm#Unsync_Microsoft_365_client_tenants_from_Datto_RMM
- Date and time of when the sync was attempted.
Once you get us this information, we can further investigate the issue.
Step 8: Send up to Elevated Support
Email partner using the Support::RMM::TE L2 Email macro.
Depending on if your case matches any of the problem cases, send the case up using the following macros:
- Support::RMM::Tech Escalation: L2: RMM
- Support::RMM::Incidents::Suspected Known Issue
Related File: Support M365 Troubleshooting.pdf