storage account Archives

Posted on July 14, 2022July 16, 2022 by geralexgr — Leave a comment

Download and upload files on Blob storage using Azure DevOps

For some data processing scenarios I had to create an automation that would download some files from a storage account, perform actions on them (python, custom tools) and lastly upload the processed files again in the storage account.

A high level diagram is visible below:

In order to automate this scenario I used a custom devops agent on azure devops and assigned a managed identity on this agent (virtual machine) on the storage account in order to interact with it without using credentials.

Then I only used powershell and az cli to download and upload the files on the storage account.

The three pipeline tasks that are required to perform the upload, processing, download actions can be found below.

The json object is used to download a specific file based on the requirements for example the first entry on chronological order. This is why the sort-object -descending is used.

trigger:
- none

pool: mypool

steps:

- task: PowerShell@2
  displayName: download file from blob storage
  inputs:
    targetType: 'inline'
    script: |
      az login --identity
      $json = az storage blob list --container-name blobcontainer --account-name storageaccountname --prefix "folder1/subcategory" 
--auth-mode login | ConvertFrom-Json | Sort-Object -Descending { $_.properties.lastModified }
      $filename = "custom_name"
      az storage blob download --file  "C:\devopsdir\$filename" 
--name $json[0].name --container-name blobcontainer  
--account-name storageaccountname --auth-mode login
      Write-Host "##vso[task.setvariable variable=downloadfilename]$filename"
    pwsh: true


- task: PowerShell@2
  displayName: run python commands
  inputs:
    targetType: 'inline'
    script: |
      python $(Build.SourcesDirectory)/python/something.py	

- task: PowerShell@2
  displayName: upload file to storage account
  inputs:
    targetType: 'inline'
    script: |
      $name = "$(downloadfilename)" + "_" + (get-date -format "yyyyMMdd") 
      az storage blob upload --file "C:\devopsdir\$name" --name "folder1/subcategory/$name" --container-name blobcontainer  --account-name storageaccountname --auth-mode login

The result will be then uploaded on the storage account.

Posted on May 4, 2022May 4, 2022 by geralexgr — Leave a comment

Download latest file from blob storage through az cli

Given that you have a blob storage container with multiple files, you could download the most latest one easily with az cli

In my scenario I have a container named backups which includes multiple MS SQL backups. I wanted to download the latest in order to restore through a pipeline on SQL server.

In order to accomplish that you should first login with az cli.

az login

If this is not possible through automation you would have to create a managed identity for your resource. The code is uploaded on the below gist.

	$container_name_input = "backups"
	$sg_con_string = "XXX"
	$json = az storage blob list –container-name $container_name_input –connection-string $sg_con_string \| ConvertFrom-Json \| Sort-Object -Descending { $_.properties.CreationTime } \| Select-Object name
	az storage blob download –file $json[0].name –name $json[0].name –container-name $container_name_input –connection-string $sg_con_string

view raw download-latest-blob-storage-account-az-cli hosted with ❤ by GitHub

You should replace your connection string by getting yours under access keys.

If you remove descending date the default value would be Ascending.

By executing the script you will get a json output like the below:

Posted on March 16, 2022March 16, 2022 by geralexgr — 2 Comments

Write blob file data on Azure storage account – C# .NET

In this article I will demonstrate how you can write a simple .NET console application to write data inside a blob container on an Azure storage account.

Storage accounts support the below four data structures.

For the cases of an application, I had to write every x seconds data on a file in order to get the coordinates of an app. For this purpose I selected a storage account container on which I will store the Latitude and Longitude in a .json format.

Inside my storage account I created a container which I will use named application.

Inside application container I created a file named locations.json. This will be the file that will get updated from the mobile app.

My console app consists of two .cs files. One is the location class which will be my model for the serialization.

   public class Location
    {
        public Double Latitude { get; set; }    
        public Double Longitude { get; set; }
    }

And the main code which consists of two functions. The RetrieveBlobContents function can read the stream content of a requested file from the blob. As I have specific names for my scenario, I hard coded them on the application. My containerName is application and the file which I want to read is called locations.json

The WriteContentOnBlob function will serialize a random Location object and will update the file every 5 seconds. The result is accomplished by writing a file locally and pushing this local file on the blob storage. The content of the file will be first extracted on the current directory/data folder.

Program.cs

	using System;
	using System.Threading.Tasks;
	using System.Configuration;
	using System.Collections.Generic;
	using System.Net;
	using System.Timers;
	using Azure.Storage.Blobs;
	using Azure.Storage.Blobs.Models;
	using System;
	using System.IO;
	using System.Threading.Tasks;
	using System.Text;
	using storageaccount;
	using Newtonsoft.Json;

	public class Program
	{
	public static readonly string BlobConnectionString = ""
	public static readonly string containerName = "application";
	public static readonly string fileName = "locations.json";




	static async Task Main(string[] args)
	{
	//await RetrieveBlobContents();

	while (true)
	{
	Thread.Sleep(5000);
	await WriteContentOnBlob();
	}

	}

	private static async Task RetrieveBlobContents()
	{
	BlobServiceClient blobServiceClient = new BlobServiceClient(BlobConnectionString);
	BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);

	BlobClient blobClient = containerClient.GetBlobClient(fileName);

	var response = await blobClient.DownloadAsync();
	using (var streamReader = new StreamReader(response.Value.Content))
	{
	while (!streamReader.EndOfStream)
	{
	var line = await streamReader.ReadLineAsync();
	Console.WriteLine(line);
	}
	}

	}

	private static async Task WriteContentOnBlob()
	{
	BlobServiceClient blobServiceClient = new BlobServiceClient(BlobConnectionString);
	BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);

	BlobClient blobClient = containerClient.GetBlobClient(fileName);

	Directory.CreateDirectory("./data");
	string localPath = "./data/";

	Random rnd = new Random();
	Double _longtitude = rnd.NextDouble();
	Double _latitude = rnd.NextDouble();

	Location blobLocation = new Location();
	blobLocation.Latitude = _latitude;
	blobLocation.Longitude = _longtitude;

	string json = JsonConvert.SerializeObject(blobLocation);
	//blobClient.write(json);

	string localFilePath = Path.Combine(localPath, fileName);
	await File.WriteAllTextAsync(localFilePath, json);
	await blobClient.UploadAsync(localFilePath, true);


	Console.WriteLine("Latitude: {0} Longtitude: {1} ",_latitude,_longtitude);

	}


	}

view raw azure-storageaccount-demo hosted with ❤ by GitHub

By running the application the values will be replaced every time on the locations.json file.