For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.
A high level diagram is visible below:
In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.
Then I only used powershell and az cli to download and upload the files on the storage account.
The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.
The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.
trigger: - none pool: mypool steps: - task: PowerShell@2 displayName: download file from blob storage inputs: targetType: 'inline' script: | az login --identity $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" --auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified } $filename = "custom_name" az storage blob download --file "C:\devopsdir\$filename" --name $json[0].name --container-name blobcontainer --account-name storageaccountname --auth-mode login Write-Host "##vso[task.setvariable variable=downloadfilename]$filename" pwsh: true - task: PowerShell@2 displayName: run python commands inputs: targetType: 'inline' script: | python $(Build.SourcesDirectory)/python/something.py - task: PowerShell@2 displayName: upload file to storage account inputs: targetType: 'inline' script: | $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer --account-name storageaccountname --auth-mode login
The result will be then uploaded on the storage account.