Update large list with PnP-Batch with retries to address throttling challenges

Summary

This sample shows how to efficiently update a large SharePoint list of 60,000 or more items using PnP-Batch, significantly reducing update time. It addresses throttling challenges, emphasizing exception handling and retry mechanisms to ensure smooth updates.

• PnP PowerShell

Copied

$siteUrl = "https://contoso.sharepoint.com/teams/app-test"

Connect-PnPOnline –Url $siteUrl -Interactive

function UpdateType ($TypeColumn, $list) {
  do {
    try {
      $StopLoop = $false
      $batch = New-PnPBatch
      $index = 1; 
      $itemId = 0; 
      $listItems = Get-PnPListItem -List $list  -PageSize 500 | Where {$_.FieldValues.$TypeColumn -ne $null }
      $totalCount =  $listItems.Count

      $listItems| ForEach-Object {
        $itemId = $_.Id
        Set-PnPListItem -List $list -Identity $_.Id -Values @{$TypeColumn = $null;} -UpdateType SystemUpdate -Batch $batch

        if ($index % 100 -eq 0 -or $index -eq $listItems.Count) {
          write-host "Updating batch starting $index out of $totalCount on library $list"
          Invoke-PnPBatch $batch
          $batch = New-PnPBatch
        }
        $index+=1;
      }

      Write-Host "Job completed"
      $Stoploop = $true
    }
    catch {
      if ($Retrycount -gt 3) {
        Write-Host "Could not send Information after 3 retrys.$itemId after number of item  processed $index"
        $Stoploop = $true
      }
      else {
        Write-Host "Could not send Information retrying in 30 seconds...{$itemId} after number of item  processed {$index}"
        Start-Sleep -Seconds 30
        Connect-PnPOnline –Url $siteUrl -interactive
        $Retrycount = $Retrycount + 1
      }
    }
  }
  While ($Stoploop -eq $false)

  write-host $("End time " + (Get-Date) + " Updating column: " +  $TypeColumn + "from list " + $listName )
}

UpdateType "Type" "List1" 

Check out the PnP PowerShell to learn more at: https://aka.ms/pnp/powershell

The way you login into PnP PowerShell has changed please read PnP Management Shell EntraID app is deleted : what should I do ?

Source Credit

Sample first appeared on Optimising Large List Updates with PnP Batch: Handling Throttling and Enhancing Efficiency

Contributors

Author(s)
Reshmee Auckloo

Disclaimer

THESE SAMPLES ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.