Month: March 2023

Posted on by — Leave a comment

Get /app/metrics remote error: tls handshake failure – Promotheus

Teamcity can be integrated with Grafana in order to gather statistics about builds and agents activity. In order to gather those teamcity metrics you will need a prometheus installation and then a configuration for the teamcity source.

When you connect teamcity as a target on prometheus you have two options regarding an https secure communication. Either you trust the not verified certificate and place that in the configuration file, or you should enable tls communication with certificates between teamcity and prometheus.

If you do not set one of the two options on prometheus you might end up with the issue:

Get /app/metrics remote error: tls handshake failure

The first option as discussed would be to trust the insecure certificate using https. 

tls_config:
      insecure_skip_verify: true

The second option would be to communicate through certificates. First create a new folder inside your Prometheus installation and store the certificates for the teamcity server. Then you will need to point the files in the configuration.

    tls_config:
       ca_file:  /var/prometheus/ssl/signalocean.com.crt

The detailed options for tls_config in prometheus can be found below.

Configuration | Prometheus

Posted on by — Leave a comment

Scan azure devops repositories for credentials and passwords

DevSecOps practices are important for organizations especially when it comes to code repositories. Your code should avoid hard coded passwords and secrets for many reasons as a leak may occur. In this guide I will examine how you can massively scan Azure DevOps repositories for security leaks as passwords and secrets with gitleaks utility.

https://github.com/gitleaks/gitleaks

Simon has provided a very useful script that you can use in order to download all your repositories from Azure DevOps.

Cloning all repositories from Azure DevOps using Azure CLI – Simon Wahlin

When you execute the script, all the repositories will be downloaded inside your project folder.

Then you will need to install gitleaks and execute for each repository.

$folder_for_cleanup = "C:\Users\geralexgr\Documents\AzureRepos"
Get-ChildItem $folder_for_cleanup | Sort -Property FullName | ForEach-Object {
                gitleaks detect -s $_.FullName -v >> gitleaks-results.txt
                echo "######################################################################################################" >> gitleaks-results.txt
            }

The scan will go through each repository and scan for leaks. The output will be stored in gitleaks-result text file.

Posted on by — Leave a comment

Automatic rollback procedure for Azure DevOps

Azure devops pipelines provide a variety of tools for automated procedures. One mechanism that administrators can build using the YAML structure is an automated rollback mechanism during a deployment.

This means that after a deployment you can revert the previous state using your YAML tasks without having to redeploy. Another case would be a broken deployment which can be identified by monitoring tools and then a validation could approve or not the final release. This is exactly depicted in the below image. After releasing a version we have a validation step that requires manual approval from an administrator. If the validation is approved the release will proceed else the rollback will be triggered.

This mechanism is described below with YAML. Release stage includes release, validation and rollback jobs. Release job performs the actual release. Validation will depend on release job and will continue only if is approved. The rollback job will run only if validation failed which means that an administrator canceled the approval.

trigger: none
pr: none

stages:

- stage: releaseStage
  jobs:

  - deployment: release
    displayName: Release
    environment:
      name: dev
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
            - task: PowerShell@2
              displayName: hostname
              inputs:
                targetType: 'inline'
                script: |
                    deployment script here...
  
  - job: validation
    dependsOn: release
    pool: server
    steps:
    - task: ManualValidation@0
      inputs:
        notifyUsers: 'admin@domain.com'
        instructions: 'continue?'
        onTimeout: reject

  - deployment: rollback
    displayName: rollback
    dependsOn: validation
    condition: failed()
    environment:
      name: dev
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:
            - task: PowerShell@2
              displayName: rolling back
              inputs:
                targetType: 'inline'
                script: |
                    rollback script here..
                    Write-Host "rollback"

When the release can be verified from the administrator the rollback will be skipped. This is the case when the validation is approved from the user.

Validation task will ask the user for a review.

On the other hand if validation is rejected the rollback stage will run.

Posted on by — Leave a comment

Install kubernetes plugins with krew

There are a lot of tools built around Kubernetes the industry standard orchestration tool that enterprises use in order to host micro-services. One open source project which allow you to install helpful plugins for k8s is krew and you can find it on GitHub.

By installing krew, you will get access to many helpful plugins that have been created for developers and uploaded to krew marketplace. One example of these plugins is the resource-capacity plugin which gives you at a glance limit, resources usage for your k8s pods.

First you will need to install krew for your operating system. The installation instructions can be found in the link below, and you should follow the steps provided.

https://krew.sigs.k8s.io/docs/user-guide/setup/install/

If you use windows, you will need to add krew on your path in order to have the tool available on your command line. First navigate in your system advanced settings and press environmental variables.

Then you will need to add in your path variable, the installation path of krew which should be %USERPROFILE%\.krew\bin

After installing krew, you can run a command to verify if PATH variables work. You will need first to restart your cmd

As a last step you can install plugins using krew with 

kubectl krew install plugin-name

In my case I installed the resource capacity plugin which give CPU, Memory Requests and Limits for your k8s cluster.

https://github.com/robscott/kube-capacity

FInally you can find your installed plugins inside .krew/store folder.