You can use python SDK in order to retrieve blob files from a storage account on azure. First you will need to get your connection string for the storage account from the access keys section.
Then you can execute the below python code.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Azure Batch can be a great tool for instant batch processing as it creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes. Sometimes however a container could be a more appropriate solution for simplicity and scaling than a virtual machine. In this guide I will explain how you could use containers for batch service in order to run jobs and tasks.
First things first, you will need to have a azure container registry or another public or private registry to store your container image. I have already created mine and pushed my batchcontainer image inside which is a .NET micro service that returns a hello world message as an output.
internal class Program
static void Main(string args)
The next step would be to create your batch service account. The part on which you set your container as the workload is when you create a pool. Pools consist of the compute node that will execute your jobs and there you will add a new pool which will host containers from the image that you pushed earlier.
On the node selection you will have to select Marketplace on the Image type and specifically microsoft-azure-batch and ubuntu-server-container of 20-04-lts version. Then you will need to select Custom on the container configuration and add your container registry by pressing the hyperlink.
Then you will need to input the username and password for the container registry as well as the registry URL.
When you have your pool ready you can go and create your job. You can leave the default settings on the job creation but you should specify the pool where the job will run.
Then you can create a task or multiple tasks for your job and provide the commands or inputs for them. In my case I created a task named kati with the command of my name. This will be provided as input in my container which is a .NET microservice that prints a hello world message based on the input.
The important thing to do is to fill the image name from your repository. You can also provide container run options that you want for this node to have like mount of directories etc.
As a result the output would be Hello gerasimos
The output of the run can be found on the stdout.txt file which is located on the task pane. You can also find a stderr.txt file which will log errors/failures that could appear during the execution.
Lastly, you can locate your job execution by navigating in the nodes where you can find a history of your tasks. As you can see I have two successful task executions and non failed.
Azure Batch can be a great tool for instant batch processing as it creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes. The important thing using this service is that there is no additional charge for using Batch. You only pay for the underlying resources consumed, such as the virtual machines, storage, and networking.
In this post I will demonstrate how one can create a new job and task from az cli for batch service. The trick in this implementation will be the json that is provided as input for the task definition as not all available options are provided from az cli.
One important missing configuration will be the container image that can be provided in the task trough Azure portal but not with az cli.
In order to create a task using az cli and bypass this issue, you can use the json-file parameter. This option will trigger the creation using the rest api and provide the parameters for the container image.
When there is a batch service pool available, you will need to create a job.
az batch account login -g RESOURCE_GROUP -n NAME
az batch job create --id JOB_NAME --pool-id POOL_NAME
Then you can create a new task using a json file.
az batch task create --job-id JOB_NAME --json-file
Docker desktop is not easy to run as a background task on a windows server. A common issue that you may find would be that although the service is running, when the user log out from the machine, then docker stops working.
Error during connect: In the default daemon configuration on Windows, the docker client must be run with elevated privileges to connect.: Post open //./pipe/docker_engine: The system cannot find the file specified Process exited with code 1
In order to bypass this behavior you can leave the user session online inside the server by using lock instead of sign out in the windows server machine.
Given that the machine restarts, the docker service will stop working on the background. In order to bypass this problem you can use an external utility from sysinternals in order to auto logon the user.