Category: Devops

Posted on by — Leave a comment

Automatic rollback procedure for Azure DevOps

Azure devops pipelines provide a variety of tools for automated procedures. One mechanism that administrators can build using the YAML structure is an automated rollback mechanism during a deployment.

This means that after a deployment you can revert the previous state using your YAML tasks without having to redeploy. Another case would be a broken deployment which can be identified by monitoring tools and then a validation could approve or not the final release. This is exactly depicted in the below image. After releasing a version we have a validation step that requires manual approval from an administrator. If the validation is approved the release will proceed else the rollback will be triggered.

This mechanism is described below with YAML. Release stage includes release, validation and rollback jobs. Release job performs the actual release. Validation will depend on release job and will continue only if is approved. The rollback job will run only if validation failed which means that an administrator canceled the approval.

trigger: none
pr: none

stages:

- stage: releaseStage
  jobs:

  - deployment: release
    displayName: Release
    environment:
      name: dev
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
            - task: PowerShell@2
              displayName: hostname
              inputs:
                targetType: 'inline'
                script: |
                    deployment script here...
  
  - job: validation
    dependsOn: release
    pool: server
    steps:
    - task: ManualValidation@0
      inputs:
        notifyUsers: 'admin@domain.com'
        instructions: 'continue?'
        onTimeout: reject

  - deployment: rollback
    displayName: rollback
    dependsOn: validation
    condition: failed()
    environment:
      name: dev
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
            - task: PowerShell@2
              displayName: rolling back
              inputs:
                targetType: 'inline'
                script: |
                    rollback script here..
                    Write-Host "rollback"

When the release can be verified from the administrator the rollback will be skipped. This is the case when the validation is approved from the user.

Validation task will ask the user for a review.

On the other hand if validation is rejected the rollback stage will run.

Posted on by — Leave a comment

Install kubernetes plugins with krew

There are a lot of tools built around Kubernetes the industry standard orchestration tool that enterprises use in order to host micro-services. One open source project which allow you to install helpful plugins for k8s is krew and you can find it on GitHub.

By installing krew, you will get access to many helpful plugins that have been created for developers and uploaded to krew marketplace. One example of these plugins is the resource-capacity plugin which gives you at a glance limit, resources usage for your k8s pods.

First you will need to install krew for your operating system. The installation instructions can be found in the link below, and you should follow the steps provided.

https://krew.sigs.k8s.io/docs/user-guide/setup/install/

If you use windows, you will need to add krew on your path in order to have the tool available on your command line. First navigate in your system advanced settings and press environmental variables.

Then you will need to add in your path variable, the installation path of krew which should be %USERPROFILE%\.krew\bin

After installing krew, you can run a command to verify if PATH variables work. You will need first to restart your cmd

As a last step you can install plugins using krew with 

kubectl krew install plugin-name

In my case I installed the resource capacity plugin which give CPU, Memory Requests and Limits for your k8s cluster.

https://github.com/robscott/kube-capacity

FInally you can find your installed plugins inside .krew/store folder.

Posted on by — Leave a comment

InfluxDB Error: error authorizing query: user not authorized to execute statement ‘SHOW RETENTION POLICIES ON _internal’, requires READ on _internal

When using InfluxDB along with grafana you can get some internal statistics for the usage with a build-in internal database. In some cases when you try to create the grafana connection with the influxdb you can notice the error that is mentioned below.

InfluxDB Error: error authorizing query: user not authorized to execute statement ‘SHOW RETENTION POLICIES ON _internal’, requires READ on _internal

This error could be either because of permissions for the user or when the monitoring is not enabled. In order to verify you can login in influxdb

influx -username "user" -password "password"

and then list all available databases.

show databases

If the database _internal is not listed most probably you will not have the internal statistics enabled. You can read more about internal measurements based on your influxdb version in the documentation.

https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/

InfluxData does not recommend using the _internal database in a production cluster. It creates unnecessary overhead, particularly for busy clusters, that can overload an already loaded cluster. Metrics stored in the _internal database primarily measure workload performance and should only be tested in non-production environments.

In order to verify if internal metrics are enabled you should go in influxdb configuration file

cat /etc/influxdb/influxdb.conf

and check the monitor section

##
## Controls the system self-monitoring, statistics and diagnostics.
##
## The internal database for monitoring data is created automatically if
## if it does not already exist. The target retention within this database
## is called 'monitor' and is also created with a retention period of 7 days
## and a replication factor of 1, if it does not exist. In all cases the
## this retention policy is configured as the default for the database.

You can then comment out the 3 lines that are showed in order to enable internal monitoring.

Finally after doing so you can go in grafana and create a new connection with the influxdb using the _internal database and the user.