Conditional Access policies in Report-only Mode Failure

Conditional Access policies in Report-only Mode. Now what?

Last Updated on June 20, 2022 by Oktay Sari

Conditional Access policies in Report-only Mode…Now what?

Conditional access policies in Report-only mode allow you to evaluate the impact of Conditional Access policies before you enable them. For instance, you can see conditional access policies in Report-only mode in the Azure AD sign-in logs, but there’s more to it and that’s what this post is all about.

Most of the time, Conditional Access Policies are straightforward to configure. However, sometimes they can be complex and you might want to know how new policies affect your users.  Therefore, consider enabling new Conditional Access policies you create, in report-only mode to evaluate the results, but not enforce the policy. The Results are logged in the Conditional Access and Report-only tabs of the Sign-in log details, but more interestingly, you can use Azure Monitor (Log Analytics) to see even more insights and Reporting options. Once the impact has been assessed, policies can either be modified as required or enabled (switched to “On“).

You can read about complex Conditional Access policies in previous posts I wrote;

 

Here’s an overview of some conditional access policies in report-only mode:

Conditional access policies in Report-only

How to enable Conditional Access policies in Report-only mode

If you’re new to Report-only mode, I suggest you start by reading the docs for more information. In short, it comes down to this;

Require MFA for Guest Access

 

  • Create a new policy (or select an existing one)
  • Under Enable policy set the toggle to Report-only mode.
  • Save your policy

Please note:

Before enabling Conditional Access policies that require compliant devices in report-only mode, you might want to exclude Mac, iOS, and Android devices. These policies may prompt users to select a device certificate during policy evaluation, even though compliance isn’t enforced. In addition, these prompts may repeat until the device is made compliant.

Also Note that report-only mode is not applicable for Conditional Access policies with “User Actions” scopes Register security information and Register or join devices

 

 

 

 

 

 

 

 

 

Conditional Access insights and reporting

The Conditional Access insights and reporting workbook enables you to visualize Conditional Access queries and see the impact of a policy over time. You can examine the impact of an individual policy or all policies. Conditional Access insights and reporting uses Azure Monitor (Log Analytics) to store data over time, so you’ll have to set-up a Log Analytics workspace first.

Please Note:

If you don’t setup a Log Analytics workspace, you won’t be able to use Conditional Access Insights and Reports, but can still use the sign-in logs to individually analyze sign-in attempts and see which conditional access policies actually apply or not.

Conditional access policies in Report-only Sign-in Details

Also remember that the default retention of (sign-in) logs in Azure AD is quite short. Check out the MS documentation on retention for more info, but here are some numbers;

Azure ad data retention

By archiving the sign-in and audit logs to a storage account, you can save it indefinitely if you want to. But that’s outside the scope of this post. I recommend you read the docs to learn more.

Conditional access policies in Report-only mode Prerequisites

In short, here are the prerequisites;

  • You’ll need a Log Analytics workspace to retain sign-in logs data
  • You will have to stream Azure AD logs to Log Analytics
  • AAD role to access Insights and reporting
  • AAD role to access Log Analytics Workspace

Create a Log Analytics workspace

You’ve created some Conditional access policies in report-only mode, and now what? Let’s create a Azure Log Analytics workspace (if you haven’t done that already).

  • In the Azure portal, enter Log Analytics in the search box.
  • Select Log Analytics workspaces.
  • In the Log Analytics workspaces menu, click Create

log-analytics-workspaces

  • Select a subscription
  • Select of create a new Resource group
  • Name your Workspace
  • Select a region
  • Click on Review + Create
  • When the validation passes, Click Create once again

Log analytics project details

When your deployment is ready, you can go to the resource directly.

deploy log analytics workspace

Stream Azure AD Sign-in logs to your Log Analytics Workspace

We’re almost there. Let’s stream some of the Azure sign-in logs, to Log Analytics. This is also very straight forward.

  1. In the Azure AD blade, scroll down to the Monitoring section.
  2. Click on Diagnostics settings
  3. Click on Add diagnostic setting

add diagnostic setting

  1. First, Give your Diagnostic settings an name! As long as you don’t give it a name, the save button will be grayed out.
  2. Select the sign-in logs you want to stream to your workspace.
  3. I recommend you select SigninLogs and ServicePrincipalSigninLogs
  4. Select Send to Log Analytics workspace as the destination
  5. Select a subscription
  6. Select your workspace
  7. Finally click on Save

Azure AD diagnostic setting

Now you’re ready to collect data, but first give it some time. Depending on the size of the organization and complexity of Conditional Access policies, this can be any number of days. Most of the time, I’ll collect data for a week of two, up to a month at the most.

Conditional Access Insights and reporting breakdown

You can use the insights and reporting dashboard to see the impact of one or more Conditional Access policies over a specified period. The default time range is 24 hours. In the example below, I’ve set the time range to the last 28 days. You can go as far back as 90 days.

Conditional access policies in Report-only Insights and reporting

Conditional access policies in Report-only mode Impact summary

From here on, you can further drill down the reports. Let’s click on Failure and see what happens. Remember that the default selection is for all enabled policies, all users and all apps.

Conditional Access policies in Report-only Mode Failure

Now let’s select one policy and analyze the results. The first row, is the impact summary. It shows how many users or sign-ins during the selected time range resulted in either “Success”, “Failure”, ”User action required” or “Not applied” upon policy evaluation. When I click on Failure, in the example below, it shows 4 users where the selected policy denied access and the required controls were not satisfied.

Conditional Access policies in Report-only Mode Impact summary

By the way, If you open the ellipsis (three dots) of one of the pie charts you’ll be able to export this specific data to Excel format or open the query into the Log Analytics query. More on that later…

Note: You can only export data to Excel format

The example below, shows the Failure for Client apps used.

client app failure

Finally, scroll down a little more, to see which users would have failed to pass the conditional access policy upon evaluation. Click on one of the users on the left, to drill down on a particular user.

sign-in details-reportonlyfailure

Open the last run query in the Logs view

Now let’s click on the ellipsis on the top right corner, and then click on “Open the last run query in the Logs view.

Open the last run query in the Logs view

This will open Log Analytics, and from here on, you can work your magic…

Conditional access policies in Report-only data in Log Analytics

Now let’s say, I wanted to see the Client App Used in the results. I’ll simply edit the Query and add ClientAppUsed to the top part of the query (Project)

update log analytics query2

And to the bottom part, where the results are showed.

update log analytics query

And there you have it. The query now shows the Client App too. In this example it shows the Exchange Web Services, but it also could have been Mobile Apps and Desktop clients.

log-analytics-clientappused

Unlike the data on the Insights and Reports dashboard, you now how the option to export the data, to CSV, Power BI, or Excel. What ever you prefer most.

log-analytics-export

And Yes, you can still go to Azure Sign-in logs, and look-up the same information there. In the example below, I’ve searched for the Correlation ID of one of the log entries.

sign-in logs report-only

User Action required

Here’s on other example. Let’s have a look at User action required. This metric shows the number of users where the selected report-only policy applied but user action would be required if the policy were to be enabled. In the example below, I’m looking at a policy that would have required MFA for all guest users. Since the policy is in Report-only mode, for now no MFA.

The Impact summary shows that over the past 30 days, 28 guest users would have been required to sign-in using MFA, if this policy was set to ON.

Conditional access policies in Report-only User action required

When I scroll down, I can see all guest accounts and the number of Sign-in counts per user. From here, I have the option to export these users to excel, and before enabling the policy.  I can use the export to send out an e-mail and inform guests about upcoming changes.

Sign-in counts per user

Final thoughts

It’s always a good idea to understand how your Conditional Access Policies will apply to end-users and when you apply them. For instance, will users be required to do MFA? Do they need to sign-in on a compliant device? Or are users restricted to the browser on unmanaged devices? You might want to start with your CA policies configured in Report-only mode for few days or weeks, to get a good view on the impact your CA Policies will have in the end.

0 0 votes
Article Rating

Oktay Sari

#Microsoft365 | #EMS |#MEM | #Intune | Father | #Diver | #RC Pilot & #Magician in spare time | former Microsoft WI MVP

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments