Office 365 – Distribution List Migrations Version 2.0 – Part 9

A common request from customers using the distribution list migration module is the ability to batch migrations together. When doing multiple individual distribution list migrations each thread invokes Active Directory replication and Azure Active Directory Connect operations separately. We’re thrilled to announce that version 2.3.4, can now perform multiple batched distribution list migrations!

Originally, the main functions of the module worked on a single object, and of course PowerShell is not known for being easily multi-threaded. This presented challenges when creating a module that allows both for single distribution list migrations and multiple distribution list migrations. To overcome these challenges, we added a controller function that is responsible for collecting the primary parameters and SMTP addresses to be processed. The controller then breaks the list of SMTP addresses into groups of no more than five simultaneous migrations (we chose five to manage memory overhead on the host and to prevent overloading or throttling).

The controller uses PowerShell jobs, and each job provisioned is a single instance of the distribution list migration. In an effort to ensure repeated directory replication and Azure replication does not occur, we added new code that monitors a central file directory for status files. Each thread writes status files to the directory when the migration process reaches certain points. When all threads have acknowledged reaching the designated points in the migration flow, the controller script continues with certain functions. For example, during the migration, distribution lists need to move to an organizational unit (OU) that does not sync. This process deletes the distribution list from Office 365. We would not want a small list moved earlier than others, as this can result in the distribution list being unavailable for a longer period of time. Therefore, the threads wait until all are ready to move to the non-syncing OU, at which time the migration process moves all groups. The following flow chart shows an example of thread coordination between individual migration instances.

The controller is also responsible for monitoring job completion. Job completion does not mean the distribution group successfully migrated; it means the job attached to the individual migration is not in a running state. For example, if the migration fails for any reason, the process considers the job as failed. If the migration completes successfully, the process considers the job as completed. When the controller detects that all provisioned threads are no longer running, the process deletes the jobs. If more than five distribution lists are in the array, the controller continues to provision jobs in batches of no more than five until all jobs have been attempted.

When running individual migrations, administrators can choose to keep folder permissions or Send As rights, and the discovery of these permissions can occur during the migration. This can be a long and memory-intensive process. If an admin wants to scan for and retain options like full mailbox access, folder permissions, and Send As rights, you must first run the collection scripts. The collection scripts create offline files that can be imported and scanned as a part of the bulk migration process. The script does not support live collection of this data as a part of each individual migration.

How do I perform a multi-distribution list migration?

The first step to performing a migration of multiple distribution lists is building the list of distribution lists to be migrated. For ease of testing, I created a text file and saved it as bulk.txt. The text file contains the SMTP address of each group to be migrated, which is then imported into a variable.

For example:

TestGroup0@contoso.com
TestGroup1@contoso.com
TestGroup2@contoso.com
TestGroup3@contoso.com
TestGroup4@contoso.com
TestGroup5@contoso.com
TestGroup6@contoso.com
TestGroup7@contoso.com

$groups = get-content c:\data\bulk.txt

With the groups stored in a variable, I can define the rest of the variables for migration, including machine names, permissions, and file storage locations. The following command is used to invoke a sample multi-distribution list migration.

Sample command for performing pre-collection:

Start-MultipleDistributionListMigration -groupSMTPAddresses $groups -globalCatalogServer GC.domain.com -activeDirectoryCredential $onpremcred -logFolderPath c:\temp -aadConnectServer aadConnect.domain.com -aadConnectCredential $onpremcred -exchangeServer exchange.domain.com -exchangeCredential $onpremcred -exchangeOnlineCredential $cloudCred –azureADCredential $cloudCred -useCollectedFullMailboxAccessOnPrem:$TRUE -useCollectedFullMailboxAccessOffice365:$TRUE -useCollectedSendAsOnPrem:$TRUE -useCollectedFolderPermissionsOnPrem:$TRUE -useCollectedFolderPermissionsOffice365:$TRUE -triggerUpgradeToOffice365Group:$TRUE -enableHybridMailflow:$TRUE -dnNoSyncOU “OU=DoNotSync,DC=domain,DC=com”

Sample command for performing migration and enabling hybrid mail flow for the group:

Start-MultipleDistributionListMigration -groupSMTPAddresses $groups -globalCatalogServer GC.domain.com -activeDirectoryCredential $onpremcred -logFolderPath c:\temp -aadConnectServer aadConnect.domain.com -aadConnectCredential $onpremcred -exchangeServer exchange.domain.com -exchangeCredential $onpremcred -exchangeOnlineCredential $cloudCred –azureADCredential $cloudCred  -enableHybridMailflow:$TRUE -dnNoSyncOU “OU=DoNotSync,DC=domain,DC=com”

Sample command for performing migration:

Start-MultipleDistributionListMigration -groupSMTPAddresses $groups -globalCatalogServer GC.domain.com -activeDirectoryCredential $onpremcred -logFolderPath c:\temp -aadConnectServer aadConnect.domain.com -aadConnectCredential $onpremcred -exchangeOnlineCredential $cloudCred –azureADCredential $cloudCred -dnNoSyncOU “OU=DoNotSync,DC=domain,DC=com”

The log directory specified in the above command serves as the location for the storage of all logs and thread tracking files. In the specified directory is the Master directory which contains the log file associated with the controller responsible for provisioning the jobs and monitoring job status. Each thread has a separate thread folder created with the thread number. It is in this folder that the individual threads create a logging directory for each distribution list being migrated. This is where administrators may locate the folders for each migration indicating success or failure of the individual distribution lists. The status directory created in this folder is where each thread logs the thread status as each migration progresses. A file created in this directory indicates that the individual threads migration have reached a pre-determined location in code and are waiting for all other threads to remember the same place. The audit data directory may also exist if pre-collection was performed. This is the fixed directory where the offline storage files are located and will be read during the migration process.

In summary, this change now allows administrators to migrate distribution lists faster and with the same efficiency as individual migrations.