Last Updated on June 20, 2022 by Oktay Sari

Conditional Access policies in Report-only Mode…Now what?

Conditional access policies in Report-only mode allow you to evaluate the impact of Conditional Access policies before you enable them. For instance, you can see conditional access policies in Report-only mode in the Azure AD sign-in logs, but there’s more to it and that’s what this post is all about.

Most of the time, Conditional Access Policies are straightforward to configure. However, sometimes they can be complex and you might want to know how new policies affect your users. Therefore, consider enabling new Conditional Access policies you create, in report-only mode to evaluate the results, but not enforce the policy. The Results are logged in the Conditional Access and Report-only tabs of the Sign-in log details, but more interestingly, you can use Azure Monitor (Log Analytics) to see even more insights and Reporting options. Once the impact has been assessed, policies can either be modified as required or enabled (switched to “On“).

You can read about complex Conditional Access policies in previous posts I wrote;

Here’s an overview of some conditional access policies in report-only mode:

How to enable Conditional Access policies in Report-only mode

If you’re new to Report-only mode, I suggest you start by reading the docs for more information. In short, it comes down to this;

Create a new policy (or select an existing one)
Under Enable policy set the toggle to Report-only mode.
Save your policy

Please note:

Before enabling Conditional Access policies that require compliant devices in report-only mode, you might want to exclude Mac, iOS, and Android devices. These policies may prompt users to select a device certificate during policy evaluation, even though compliance isn’t enforced. In addition, these prompts may repeat until the device is made compliant.

Also Note that report-only mode is not applicable for Conditional Access policies with “User Actions” scopes Register security information and Register or join devices

Conditional Access insights and reporting

The Conditional Access insights and reporting workbook enables you to visualize Conditional Access queries and see the impact of a policy over time. You can examine the impact of an individual policy or all policies. Conditional Access insights and reporting uses Azure Monitor (Log Analytics) to store data over time, so you’ll have to set-up a Log Analytics workspace first.

Please Note:

If you don’t setup a Log Analytics workspace, you won’t be able to use Conditional Access Insights and Reports, but can still use the sign-in logs to individually analyze sign-in attempts and see which conditional access policies actually apply or not.

Also remember that the default retention of (sign-in) logs in Azure AD is quite short. Check out the MS documentation on retention for more info, but here are some numbers;

By archiving the sign-in and audit logs to a storage account, you can save it indefinitely if you want to. But that’s outside the scope of this post. I recommend you read the docs to learn more.

Conditional access policies in Report-only mode Prerequisites

In short, here are the prerequisites;

You’ll need a Log Analytics workspace to retain sign-in logs data
You will have to stream Azure AD logs to Log Analytics
AAD role to access Insights and reporting
AAD role to access Log Analytics Workspace

Create a Log Analytics workspace

You’ve created some Conditional access policies in report-only mode, and now what? Let’s create a Azure Log Analytics workspace (if you haven’t done that already).

In the Azure portal, enter Log Analytics in the search box.
Select Log Analytics workspaces.
In the Log Analytics workspaces menu, click Create

Select a subscription
Select of create a new Resource group
Name your Workspace
Select a region
Click on Review + Create
When the validation passes, Click Create once again

When your deployment is ready, you can go to the resource directly.

Stream Azure AD Sign-in logs to your Log Analytics Workspace

We’re almost there. Let’s stream some of the Azure sign-in logs, to Log Analytics. This is also very straight forward.

In the Azure AD blade, scroll down to the Monitoring section.
Click on Diagnostics settings
Click on Add diagnostic setting

First, Give your Diagnostic settings an name! As long as you don’t give it a name, the save button will be grayed out.
Select the sign-in logs you want to stream to your workspace.
I recommend you select SigninLogs and ServicePrincipalSigninLogs
Select Send to Log Analytics workspace as the destination
Select a subscription
Select your workspace
Finally click on Save

Now you’re ready to collect data, but first give it some time. Depending on the size of the organization and complexity of Conditional Access policies, this can be any number of days. Most of the time, I’ll collect data for a week of two, up to a month at the most.

Conditional Access Insights and reporting breakdown

You can use the insights and reporting dashboard to see the impact of one or more Conditional Access policies over a specified period. The default time range is 24 hours. In the example below, I’ve set the time range to the last 28 days. You can go as far back as 90 days.

Conditional access policies in Report-only mode Impact summary

From here on, you can further drill down the reports. Let’s click on Failure and see what happens. Remember that the default selection is for all enabled policies, all users and all apps.

Now let’s select one policy and analyze the results. The first row, is the impact summary. It shows how many users or sign-ins during the selected time range resulted in either “Success”, “Failure”, ”User action required” or “Not applied” upon policy evaluation. When I click on Failure, in the example below, it shows 4 users where the selected policy denied access and the required controls were not satisfied.

By the way, If you open the ellipsis (three dots) of one of the pie charts you’ll be able to export this specific data to Excel format or open the query into the Log Analytics query. More on that later…

Note: You can only export data to Excel format

The example below, shows the Failure for Client apps used.

Finally, scroll down a little more, to see which users would have failed to pass the conditional access policy upon evaluation. Click on one of the users on the left, to drill down on a particular user.

Open the last run query in the Logs view

Now let’s click on the ellipsis on the top right corner, and then click on “Open the last run query in the Logs view.”

This will open Log Analytics, and from here on, you can work your magic…

Now let’s say, I wanted to see the Client App Used in the results. I’ll simply edit the Query and add ClientAppUsed to the top part of the query (Project)

And to the bottom part, where the results are showed.

And there you have it. The query now shows the Client App too. In this example it shows the Exchange Web Services, but it also could have been Mobile Apps and Desktop clients.

Unlike the data on the Insights and Reports dashboard, you now how the option to export the data, to CSV, Power BI, or Excel. What ever you prefer most.

And Yes, you can still go to Azure Sign-in logs, and look-up the same information there. In the example below, I’ve searched for the Correlation ID of one of the log entries.

User Action required

Here’s on other example. Let’s have a look at User action required. This metric shows the number of users where the selected report-only policy applied but user action would be required if the policy were to be enabled. In the example below, I’m looking at a policy that would have required MFA for all guest users. Since the policy is in Report-only mode, for now no MFA.

The Impact summary shows that over the past 30 days, 28 guest users would have been required to sign-in using MFA, if this policy was set to ON.

When I scroll down, I can see all guest accounts and the number of Sign-in counts per user. From here, I have the option to export these users to excel, and before enabling the policy. I can use the export to send out an e-mail and inform guests about upcoming changes.

Final thoughts

It’s always a good idea to understand how your Conditional Access Policies will apply to end-users and when you apply them. For instance, will users be required to do MFA? Do they need to sign-in on a compliant device? Or are users restricted to the browser on unmanaged devices? You might want to start with your CA policies configured in Report-only mode for few days or weeks, to get a good view on the impact your CA Policies will have in the end.

0 0 votes

Article Rating