Tag: blob storage

For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.

A high level diagram is visible below:

In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.

Then I only used powershell and az cli to download and upload the files on the storage account.

The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.

The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.

trigger:
- none

pool: mypool

steps:

- task: PowerShell@2
  displayName: download file from blob storage
  inputs:
    targetType: 'inline'
    script: |
      az login --identity
      $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" 
--auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified }
      $filename = "custom_name"
      az storage blob download --file  "C:\devopsdir\$filename" 
--name $json[0].name --container-name blobcontainer  
--account-name storageaccountname --auth-mode login
      Write-Host "##vso[task.setvariable variable=downloadfilename]$filename"
    pwsh: true


- task: PowerShell@2
  displayName: run python commands
  inputs:
    targetType: 'inline'
    script: |
      python $(Build.SourcesDirectory)/python/something.py	

- task: PowerShell@2
  displayName: upload file to storage account
  inputs:
    targetType: 'inline'
    script: |
      $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") 
      az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer  --account-name storageaccountname --auth-mode login

The result will be then uploaded on the storage account.