Skip to navigation Skip to content
GeralexGR

Personal blog

  • Home
  • About
  • Contact
  • Home
  • About
  • Contact

Tag: blob storage

Posted on July 14, 2022July 16, 2022 by geralexgr — Leave a comment

Download and upload files on Blob storage using Azure DevOps

For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.

A high level diagram is visible below:

In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.

Then I only used powershell and az cli to download and upload the files on the storage account.

The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.

The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.

trigger:
- none

pool: mypool

steps:

- task: PowerShell@2
  displayName: download file from blob storage
  inputs:
    targetType: 'inline'
    script: |
      az login --identity
      $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" 
--auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified }
      $filename = "custom_name"
      az storage blob download --file  "C:\devopsdir\$filename" 
--name $json[0].name --container-name blobcontainer  
--account-name storageaccountname --auth-mode login
      Write-Host "##vso[task.setvariable variable=downloadfilename]$filename"
    pwsh: true


- task: PowerShell@2
  displayName: run python commands
  inputs:
    targetType: 'inline'
    script: |
      python $(Build.SourcesDirectory)/python/something.py	

- task: PowerShell@2
  displayName: upload file to storage account
  inputs:
    targetType: 'inline'
    script: |
      $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") 
      az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer  --account-name storageaccountname --auth-mode login

The result will be then uploaded on the storage account.

Categories: Automation, Azure, Cloud, Devops
Tags: account, automation, azure devops download, azure devops upload, azuredevops, blob storage, containers, files, storage account, storageaccount, upload
Mastering Azure Devops CI/CD Pipelines with YAML
Learn how to create advanced automation scenarios using YAML and Azure Devops: https://www.udemy.com/course/mastering-azure-devops-cicd-pipelines-with-yaml
Youtube
Subscribe on my YouTube channel for more tutorials:
Azure DevOps platform Fundamentals – Build CI/CD pipelines
Start your journey with Azure DevOps platform: https://www.udemy.com/course/azure-devops-platform-fundamentals-build-cicd-pipelines/
Adblock notice
Support this blog to maintain its operational costs by turning off Adblock or donate a small amount using the button below
Blog Stats
  • 174,409 Views

Join 6 other subscribers
Recent Posts
  • Get /app/metrics remote error: tls handshake failure – Promotheus
  • Scan azure devops repositories for credentials and passwords
  • Automatic rollback procedure for Azure DevOps
  • Install kubernetes plugins with krew
  • InfluxDB Error: error authorizing query: user not authorized to execute statement ‘SHOW RETENTION POLICIES ON _internal’, requires READ on _internal
Top Posts
  • Error while proxying request: getting credentials: exec: executable kubelogin not found
  • Pass variables values inside terraform modules
  • Trigger Azure Devops build pipelines using REST API
  • Scan azure devops repositories for credentials and passwords
  • Authentication failed. The 'Authorization' header is missing - Invoke-WebRequest Azure ARM
  • Pass parameters from build to release pipelines on Azure devops
  • Parameters and variables - GitHub workflows
  • remote: TF401019: The Git repository with name or identifier does not exist or you do not have permissions for the operation you are attempting.
  • Creating Windows/Linux Web App terraform: (Site Name "" / Resource Group ""): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=
  • dynamically set dependsOn using variables - Azure devops
Categories
  • Ansible
  • Automation
  • Azure
  • BotFramework
  • C# & .NET
  • Cloud
  • Devops
  • Docker
  • Github
  • Infrastructure
  • Kubernetes
  • Linux
  • Monitoring
  • Scripts
  • Security
  • Terraform
  • Testing
  • UWP apps – Windows 10
  • VMware
  • Web Development
  • Windows
  • Xamarin
Tags
.net ansible api appservice app service az cli Azure azuredevops azure devops build c# centos cli command container containers dependsOn deploy devops docker files git github jobs k8s kubernetes linux modules pipeline pipelines powershell python redhat release security storage account terraform variable variables visual studio vmware web app windows xamarin xamarin.forms
Archives
  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021
  • May 2021
  • April 2021
  • March 2021
  • February 2021
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • December 2019
  • August 2019
  • April 2019
  • March 2019
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • March 2018
  • July 2017
  • May 2017
  • February 2016
  • October 2015
  • August 2015
  • February 2015
  • November 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
Check out my Azure DevOps Udemy course
© GeralexGR 2023
Built with Storefront.