Skip to navigation Skip to content
GeralexGR

Personal blog

  • Home
  • About
  • Contact
  • Home
  • About
  • Contact

Tag: storage account

Posted on February 21, 2023February 21, 2023 by geralexgr — Leave a comment

Get blob files from azure storage account using python

You can use python SDK in order to retrieve blob files from a storage account on azure. First you will need to get your connection string for the storage account from the access keys section.

Then you can execute the below python code.

import os, uuid
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
connection_string = "CONNECTION_STRING"
service = BlobServiceClient.from_connection_string(connection_string)
try:
account_info = service.get_account_information()
print('Using Storage SKU: {}'.format(account_info['sku_name']))
container_client = service.get_container_client("files")
try:
for blob in container_client.list_blobs():
print("Found blob: ", blob.name)
except ResourceNotFoundError:
print("Container not found.")
except Exception as ex:
print('Exception:')
print(ex)
view raw azure-storage-get-blobs-python hosted with ❤ by GitHub

As seen from the screenshots, I have a container named files

And inside the container I have a test.txt file.

By running the python script (in my case I added on a container) you will get the files inside the blob container.

Quickstart: Azure Blob Storage client library for Python – Azure Storage | Microsoft Learn

Categories: Azure, Devops, Scripts
Tags: Azure, azure-identity, azure-storage-blob, blob, blob retrieve python, python, python azure blob, python sdk, storage, storage account
Posted on July 14, 2022July 16, 2022 by geralexgr — Leave a comment

Download and upload files on Blob storage using Azure DevOps

For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.

A high level diagram is visible below:

In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.

Then I only used powershell and az cli to download and upload the files on the storage account.

The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.

The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.

trigger:
- none

pool: mypool

steps:

- task: PowerShell@2
  displayName: download file from blob storage
  inputs:
    targetType: 'inline'
    script: |
      az login --identity
      $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" 
--auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified }
      $filename = "custom_name"
      az storage blob download --file  "C:\devopsdir\$filename" 
--name $json[0].name --container-name blobcontainer  
--account-name storageaccountname --auth-mode login
      Write-Host "##vso[task.setvariable variable=downloadfilename]$filename"
    pwsh: true


- task: PowerShell@2
  displayName: run python commands
  inputs:
    targetType: 'inline'
    script: |
      python $(Build.SourcesDirectory)/python/something.py	

- task: PowerShell@2
  displayName: upload file to storage account
  inputs:
    targetType: 'inline'
    script: |
      $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") 
      az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer  --account-name storageaccountname --auth-mode login

The result will be then uploaded on the storage account.

Categories: Automation, Azure, Cloud, Devops
Tags: account, automation, azure devops download, azure devops upload, azuredevops, blob storage, containers, files, storage account, storageaccount, upload
Posted on May 4, 2022May 4, 2022 by geralexgr — Leave a comment

Download latest file from blob storage through az cli

Given that you have a blob storage container with multiple files, you could download the most latest one easily with az cli

In my scenario I have a container named backups which includes multiple MS SQL backups. I wanted to download the latest in order to restore through a pipeline on SQL server.

In order to accomplish that you should first login with az cli.

az login

If this is not possible through automation you would have to create a managed identity for your resource. The code is uploaded on the below gist.

$container_name_input = "backups"
$sg_con_string = "XXX"
$json = az storage blob list –container-name $container_name_input –connection-string $sg_con_string | ConvertFrom-Json | Sort-Object -Descending { $_.properties.CreationTime } | Select-Object name
az storage blob download –file $json[0].name –name $json[0].name –container-name $container_name_input –connection-string $sg_con_string
view raw download-latest-blob-storage-account-az-cli hosted with ❤ by GitHub

You should replace your connection string by getting yours under access keys.

If you remove descending date the default value would be Ascending.

By executing the script you will get a json output like the below:

Categories: Azure, Cloud, Scripts
Tags: account az cli download, az, az cli, blob, blob container download, cli, download, sg account download blob, storage, storage account, storage-account
Posted on March 16, 2022March 16, 2022 by geralexgr — 2 Comments

Write blob file data on Azure storage account – C# .NET

In this article I will demonstrate how you can write a simple .NET console application to write data inside a blob container on an Azure storage account.

Storage accounts support the below four data structures.

For the cases of an application, I had to write every x seconds data on a file in order to get the coordinates of an app. For this purpose I selected a storage account container on which I will store the Latitude and Longitude in a .json format.

Inside my storage account I created a container which I will use named application.

Inside application container I created a file named locations.json. This will be the file that will get updated from the mobile app.

My console app consists of two .cs files. One is the location class which will be my model for the serialization.

   public class Location
    {
        public Double Latitude { get; set; }    
        public Double Longitude { get; set; }
    }

And the main code which consists of two functions. The RetrieveBlobContents function can read the stream content of a requested file from the blob. As I have specific names for my scenario, I hard coded them on the application. My containerName is application and the file which I want to read is called locations.json

The WriteContentOnBlob function will serialize a random Location object and will update the file every 5 seconds. The result is accomplished by writing a file locally and pushing this local file on the blob storage. The content of the file will be first extracted on the current directory/data folder.

Program.cs

using System;
using System.Threading.Tasks;
using System.Configuration;
using System.Collections.Generic;
using System.Net;
using System.Timers;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System;
using System.IO;
using System.Threading.Tasks;
using System.Text;
using storageaccount;
using Newtonsoft.Json;
public class Program
{
public static readonly string BlobConnectionString = ""
public static readonly string containerName = "application";
public static readonly string fileName = "locations.json";
static async Task Main(string[] args)
{
//await RetrieveBlobContents();
while (true)
{
Thread.Sleep(5000);
await WriteContentOnBlob();
}
}
private static async Task RetrieveBlobContents()
{
BlobServiceClient blobServiceClient = new BlobServiceClient(BlobConnectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blobClient = containerClient.GetBlobClient(fileName);
var response = await blobClient.DownloadAsync();
using (var streamReader = new StreamReader(response.Value.Content))
{
while (!streamReader.EndOfStream)
{
var line = await streamReader.ReadLineAsync();
Console.WriteLine(line);
}
}
}
private static async Task WriteContentOnBlob()
{
BlobServiceClient blobServiceClient = new BlobServiceClient(BlobConnectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
BlobClient blobClient = containerClient.GetBlobClient(fileName);
Directory.CreateDirectory("./data");
string localPath = "./data/";
Random rnd = new Random();
Double _longtitude = rnd.NextDouble();
Double _latitude = rnd.NextDouble();
Location blobLocation = new Location();
blobLocation.Latitude = _latitude;
blobLocation.Longitude = _longtitude;
string json = JsonConvert.SerializeObject(blobLocation);
//blobClient.write(json);
string localFilePath = Path.Combine(localPath, fileName);
await File.WriteAllTextAsync(localFilePath, json);
await blobClient.UploadAsync(localFilePath, true);
Console.WriteLine("Latitude: {0} Longtitude: {1} ",_latitude,_longtitude);
}
}
view raw azure-storageaccount-demo hosted with ❤ by GitHub

By running the application the values will be replaced every time on the locations.json file.

Categories: Azure, C# & .NET, Cloud
Tags: .net, account, Azure, blob storage c#, c#, containers, file, storage account, uploadAsync blob, write data, write storage account
Mastering Azure Devops CI/CD Pipelines with YAML
Learn how to create advanced automation scenarios using YAML and Azure Devops: https://www.udemy.com/course/mastering-azure-devops-cicd-pipelines-with-yaml
Youtube
Subscribe on my YouTube channel for more tutorials:
Azure DevOps platform Fundamentals – Build CI/CD pipelines
Start your journey with Azure DevOps platform: https://www.udemy.com/course/azure-devops-platform-fundamentals-build-cicd-pipelines/
Adblock notice
Support this blog to maintain its operational costs by turning off Adblock or donate a small amount using the button below
Blog Stats
  • 174,409 Views

Join 6 other subscribers
Recent Posts
  • Get /app/metrics remote error: tls handshake failure – Promotheus
  • Scan azure devops repositories for credentials and passwords
  • Automatic rollback procedure for Azure DevOps
  • Install kubernetes plugins with krew
  • InfluxDB Error: error authorizing query: user not authorized to execute statement ‘SHOW RETENTION POLICIES ON _internal’, requires READ on _internal
Top Posts
  • Error while proxying request: getting credentials: exec: executable kubelogin not found
  • Pass variables values inside terraform modules
  • Trigger Azure Devops build pipelines using REST API
  • Scan azure devops repositories for credentials and passwords
  • Pass parameters from build to release pipelines on Azure devops
  • Authentication failed. The 'Authorization' header is missing - Invoke-WebRequest Azure ARM
  • Parameters and variables - GitHub workflows
  • Creating Windows/Linux Web App terraform: (Site Name "" / Resource Group ""): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=
  • remote: TF401019: The Git repository with name or identifier does not exist or you do not have permissions for the operation you are attempting.
  • dynamically set dependsOn using variables - Azure devops
Categories
  • Ansible
  • Automation
  • Azure
  • BotFramework
  • C# & .NET
  • Cloud
  • Devops
  • Docker
  • Github
  • Infrastructure
  • Kubernetes
  • Linux
  • Monitoring
  • Scripts
  • Security
  • Terraform
  • Testing
  • UWP apps – Windows 10
  • VMware
  • Web Development
  • Windows
  • Xamarin
Tags
.net ansible api appservice app service az cli Azure azuredevops azure devops build c# centos cli command container containers dependsOn deploy devops docker files git github jobs k8s kubernetes linux modules pipeline pipelines powershell python redhat release security storage account terraform variable variables visual studio vmware web app windows xamarin xamarin.forms
Archives
  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021
  • May 2021
  • April 2021
  • March 2021
  • February 2021
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • December 2019
  • August 2019
  • April 2019
  • March 2019
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • March 2018
  • July 2017
  • May 2017
  • February 2016
  • October 2015
  • August 2015
  • February 2015
  • November 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
Check out my Azure DevOps Udemy course
© GeralexGR 2023
Built with Storefront.