Skip to navigation Skip to content
GeralexGR

Personal blog

  • Home
  • About
  • Contact
  • Home
  • About
  • Contact

Tag: storageaccount

Posted on July 14, 2022July 16, 2022 by geralexgr — Leave a comment

Download and upload files on Blob storage using Azure DevOps

For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.

A high level diagram is visible below:

In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.

Then I only used powershell and az cli to download and upload the files on the storage account.

The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.

The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.

trigger:
- none

pool: mypool

steps:

- task: PowerShell@2
  displayName: download file from blob storage
  inputs:
    targetType: 'inline'
    script: |
      az login --identity
      $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" 
--auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified }
      $filename = "custom_name"
      az storage blob download --file  "C:\devopsdir\$filename" 
--name $json[0].name --container-name blobcontainer  
--account-name storageaccountname --auth-mode login
      Write-Host "##vso[task.setvariable variable=downloadfilename]$filename"
    pwsh: true


- task: PowerShell@2
  displayName: run python commands
  inputs:
    targetType: 'inline'
    script: |
      python $(Build.SourcesDirectory)/python/something.py	

- task: PowerShell@2
  displayName: upload file to storage account
  inputs:
    targetType: 'inline'
    script: |
      $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") 
      az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer  --account-name storageaccountname --auth-mode login

The result will be then uploaded on the storage account.

Categories: Automation, Azure, Cloud, Devops
Tags: account, automation, azure devops download, azure devops upload, azuredevops, blob storage, containers, files, storage account, storageaccount, upload
Youtube
Subscribe on my YouTube channel for more tutorials:
Adblock notice
Support this blog to maintain its operational costs by turning off Adblock or donate a small amount through paypal.me
Blog Stats
  • 73,036 Views

Join 3 other subscribers

Recent Posts
  • Install Azure DevOps agent using powershell
  • kubectl commands cheat sheet for daily kubernetes administration
  • Restore transaction log backup on MS SQL server using powershell
  • Build NodeJS applications using Azure devops and npm
  • Download and upload files on Blob storage using Azure DevOps
Top Posts
  • Ansible loop over nested dictionary subelements - list object has no attribute
  • Trigger Azure Devops build pipelines using REST API
  • Authentication failed. The 'Authorization' header is missing - Invoke-WebRequest Azure ARM
  • Build pipeline on tag push - Azure DevOps build triggers
  • Error MSB3073 The command "npm run build" exited with code 1 - Visual Studio
  • The checksum from the provided manifest file do not match the content of file .ovf
  • Azure DevOps best practices - jobs and stages
  • Update variable group using Azure DevOps rest API - pipeline example
  • Download latest file from blob storage through az cli
  • kubectl commands cheat sheet for daily kubernetes administration
Categories
  • Ansible
  • Automation
  • Azure
  • BotFramework
  • C# & .NET
  • Cloud
  • Devops
  • Docker
  • Infrastructure
  • Kubernetes
  • Linux
  • Scripts
  • Security
  • Terraform
  • UWP apps – Windows 10
  • VMware
  • Web Development
  • Windows
  • Xamarin
Tags
.net agent ansible apache api appservice app service automation az cli Azure azuredevops azure devops build c# centos containers deploy devops docker esxi github group linux Mysql PHP pipeline pipelines postman powershell push redhat release rest rhel security storage account terraform update users variables visual studio vmware windows xamarin xamarin.forms
Archives
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021
  • May 2021
  • April 2021
  • March 2021
  • February 2021
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • December 2019
  • August 2019
  • April 2019
  • March 2019
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • March 2018
  • July 2017
  • May 2017
  • February 2016
  • October 2015
  • August 2015
  • February 2015
  • November 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
© GeralexGR 2022
Built with Storefront.