Category: Monitoring

Posted on by — Leave a comment

Get standard error output from command on powershell

There could be times that you will need to get the standard output from a script or command in powershell and handle this information. An example that I use is the output of docker info which I use to monitor the service state along with other things. The output of the docker info will inform you about an error during connection if the engine is not working. In the latest version of docker they changed the way message is displayed and this information will be printed on standard error instead of standard output.

docker info

As a result when you try to find the connection state through a powershell command you will fail identifying the correct state.

 (docker info).Contains('ERROR: error during connect')

This behavior is noticed because the error is printed on standard error.

In order to bypass and get the standard error on the output, you will need to use 2>&1 along with the command.

docker info 2>&1

After that you can handle the output message appropriately.

Posted on by — Leave a comment

Monitoring Windows service on Azure with Event Viewer and Log Analytics

A Log Analytics workspace is a unique environment for log data from Azure Monitor and other Azure services, such as Microsoft Sentinel and Microsoft Defender for Cloud. Expect from that it can be used also for monitoring combined with Azure alerts given that you create the appropriate query.

The below query can be used to monitor a windows server service by querying log analytics. In more detail it searches for eventID=7036 which indicates the service stopped status.

Query code in Kusto language will return the service name, state and time of the event.

Event
| where TimeGenerated >ago(1h)
| where EventLog  == "System" and EventID ==7036 and Source == "Service Control Manager" 
| parse kind=relaxed EventData with * '<Data Name="param1">' Windows_Service_Name '</Data><Data Name="param2">' Windows_Service_State '</Data>'*
| sort by TimeGenerated desc
| where Windows_Service_Name startswith "Docker Desktop" and Windows_Service_State contains "stopped"
| project Computer, Windows_Service_Name, Windows_Service_State, TimeGenerated

You can use the above query to create a azure alert when a service is found as stopped. As I want to monitor the Docker Desktop service, I will need to use that in the where clause of the query where Windows_Service_Name. The alert logic should indicate when a result is returned as a row in a given timeframe then an alert should be generated. This happens because a row is returned only when the event is captured on the event viewer. This means that the service stopped during the TimeGenerated interval of the query. The frequency of evaluation will be the time on which we want to repeat the log analytics query. For example if we need to search every 5 minutes for a stopped service then we should add 5 minutes there.

Finally the alert will be triggered and inform you about windows stopped services.

Posted on by — Leave a comment

Query servers disk size – Azure Log Analytics

Log Analytics can be a powerful monitoring tool for your infrastructure as you can query various metrics that are important for your servers availability like disk space.

The below query can be used to get free space for your Linux VM disks. 

 InsightsMetrics
| where Origin == "vm.azm.ms"
    and Namespace == "LogicalDisk"  
    and Name == "FreeSpaceMB"
| extend Disk=tostring(todynamic(Tags)["vm.azm.ms/mountId"]),
    Disk_Size_GB=(todynamic(Tags)["vm.azm.ms/diskSizeMB"]) / (1024)
| summarize Disk_Free_Space_GB = avg(Val) / (1024)
    by Computer,
    Disk, Disk_Size_GB, _ResourceId
| where Disk in ('sda1', 'sdb1', 'sdc1', 'sdd1','/')
| project Computer, Disk, Disk_Size_GB, Disk_Free_Space_GB

You can also select the scope of the query in order to get results for specific resources/resource groups.

The result will bring all disks size and free space that their name is included in the list 

('sda1', 'sdb1', 'sdc1', 'sdd1','/')

The same query with a small change can be applied also for windows vms in order to get the available space for your C: drive.

 InsightsMetrics
| where Origin == "vm.azm.ms"
    and Namespace == "LogicalDisk"  
    and Name == "FreeSpaceMB"
| extend Disk=tostring(todynamic(Tags)["vm.azm.ms/mountId"]),
    Disk_Size_GB=(todynamic(Tags)["vm.azm.ms/diskSizeMB"]) / (1024)
| summarize Disk_Free_Space_GB = avg(Val) / (1024)
    by Computer,
    Disk, Disk_Size_GB, _ResourceId
| where Disk in ('C:')
| project Computer, Disk, Disk_Size_GB, Disk_Free_Space_GB

Finally you can pin this result inside an Azure dashboard by clicking pin and select the specific dashboard.

Posted on by — Leave a comment

Get /app/metrics remote error: tls handshake failure – Promotheus

Teamcity can be integrated with Grafana in order to gather statistics about builds and agents activity. In order to gather those teamcity metrics you will need a prometheus installation and then a configuration for the teamcity source.

When you connect teamcity as a target on prometheus you have two options regarding an https secure communication. Either you trust the not verified certificate and place that in the configuration file, or you should enable tls communication with certificates between teamcity and prometheus.

If you do not set one of the two options on prometheus you might end up with the issue:

Get /app/metrics remote error: tls handshake failure

The first option as discussed would be to trust the insecure certificate using https. 

tls_config:
      insecure_skip_verify: true

The second option would be to communicate through certificates. First create a new folder inside your Prometheus installation and store the certificates for the teamcity server. Then you will need to point the files in the configuration.

    tls_config:
       ca_file:  /var/prometheus/ssl/signalocean.com.crt

The detailed options for tls_config in prometheus can be found below.

Configuration | Prometheus