Report-only mode for conditional access

This week is, like last week, about a awareness for new feature that is introduced with conditional access. Last week was all about the recently introduced Conditional Access Insights workbook. In that post I already mentioned the Report-only mode for conditional access policies. In this post I want to focus on that Report-only mode. Report-only mode is a new state of a conditional access policy state that allows IT administrators to evaluate the impact of conditional access policies before enabling them in their environment. That enables the IT administrators to anticipate on the number and names of users impacted by common deployment initiatives such as blocking legacy authentication, requiring multi-factor authentication, or implementing sign-in risk policies. A great step forward.

In this post I’ll walk through the steps of configuring Report-only mode for conditional access policies, followed by looking at the end-user experience. I’ll end this post by looking at the administrator experience.

Configure report-only mode

Let’s start by having a look at the steps to configure the Report-only mode for a conditional access policy. These steps will walk through the creation of a new conditional access policy, with a focus on configuring the Report-only mode. The exact configuration of the conditional access policy assignments and conditions are not part of that focus. The following three steps walk through that configuration.

  1. Open the Azure portal and navigate to Azure Active Directory  > Security > Conditional access (or open the Microsoft 365 Device Management portal and navigate to Endpoint security Conditional access) to open the Conditional access – Policies blade
  2. On the Conditional access – Policies blade, click New policy to open the New blade
  3. On the New blade, configure the assignment and conditions to filter the users and cloud apps that should be targeted by the conditional access policy. After configuring the conditions it’s time to look at the access controls. The access controls are the configuration that eventually might impact the end-user. In the access controls, the grant control determines that behavior. In the grant control the IT administrator can configure the requirements that should be met for accessing the cloud app for the end-user. Depending on the configured requirements, there might be a minimal impact for the end-user (see Figure 1 and and Figure 2 about the messages that are shown about the impact of the conditional access policy based on the configured requirements). After configuring the grant control, select Report-only with Enable policy (also shown in Figure 1) and click Create.

End-user experience

Depending on the configuration that is used in the grant control, of the conditional access policy, the end-user might have a slight impact when using the Report-only mode. The table below is a summary of the available requirements in combination with the potential impact. This table is based on the information as shown during the configuration of the conditional access (see Figure 2), as I haven’t been able to get the mentioned experience on my test devices. I’ve tested with a Samsung Galaxy 10, iPad 2018 and iPhone X.

RequirementPotential user impact
Require multi-factor authentication No impact
Require device to be marked as compliantMay prompt users on macOS, iOS and Android devices to select a device certificate
Require Hybrid Azure AD joined device No impact
Require approved client appMay prompt users on macOS, iOS and Android devices to select a device certificate
Require app protection policyMay prompt users on macOS, iOS and Android devices to select a device certificate

Administrator experience

An interesting part to look at is the experience of the IT administrator. That can be achieved by looking at the Conditional Access Insights workbook (as shown last week). The Conditional Access Insights workbook can be used to get the insights of the different Report-only mode conditional access policies. The data in the workbook can be filtered to only show information about Report-only mode conditional access policies, or even only data of a specific conditional access policy.

Besides that workbook, the Sign-ins monitoring of Azure AD also provides a new tab in the details of a sign-in. That tab is the Report-only (Preview) tab. As shown below that tab provides information about the different Report-only mode conditional access policies that were applicable to the sign-in. Per conditional access policy, the result is shown of the sign-in. That result will show what the effect would be of that conditional access policy and that information will help with determining the impact of enabling that conditional access policy.

Below is an overview of the different result states of a Report-only conditional access policy. Almost all of these results are shown in Figure 3 above (with the exception of the user action required result).

Report only: FailureThe configured conditional access policy conditions were satisfied, but not all the required (non-interactive) controls were satisfied.
Report only: SuccessThe configured conditional access policy conditions and required (non-interactive) controls were satisfied. 
Report only: Not appliedNot all configured conditional access policy conditions were satisfied.
Report only: User action requiredThe configured conditional policy conditions were satisfied, but a user action would be required to satisfy the required controls.

More information

For more information regarding report-only, please refer to the following documents: