Posted on Leave a comment

Monitoring Windows service on Azure with Event Viewer and Log Analytics

A Log Analytics workspace is a unique environment for log data from Azure Monitor and other Azure services, such as Microsoft Sentinel and Microsoft Defender for Cloud. Expect from that it can be used also for monitoring combined with Azure alerts given that you create the appropriate query.

The below query can be used to monitor a windows server service by querying log analytics. In more detail it searches for eventID=7036 which indicates the service stopped status.

Query code in Kusto language will return the service name, state and time of the event.

Event
| where TimeGenerated >ago(1h)
| where EventLog  == "System" and EventID ==7036 and Source == "Service Control Manager" 
| parse kind=relaxed EventData with * '<Data Name="param1">' Windows_Service_Name '</Data><Data Name="param2">' Windows_Service_State '</Data>'*
| sort by TimeGenerated desc
| where Windows_Service_Name startswith "Docker Desktop" and Windows_Service_State contains "stopped"
| project Computer, Windows_Service_Name, Windows_Service_State, TimeGenerated

You can use the above query to create a azure alert when a service is found as stopped. As I want to monitor the Docker Desktop service, I will need to use that in the where clause of the query where Windows_Service_Name. The alert logic should indicate when a result is returned as a row in a given timeframe then an alert should be generated. This happens because a row is returned only when the event is captured on the event viewer. This means that the service stopped during the TimeGenerated interval of the query. The frequency of evaluation will be the time on which we want to repeat the log analytics query. For example if we need to search every 5 minutes for a stopped service then we should add 5 minutes there.

Finally the alert will be triggered and inform you about windows stopped services.

Posted on Leave a comment

Get /app/metrics remote error: tls handshake failure – Promotheus

Teamcity can be integrated with Grafana in order to gather statistics about builds and agents activity. In order to gather those teamcity metrics you will need a prometheus installation and then a configuration for the teamcity source.

When you connect teamcity as a target on prometheus you have two options regarding an https secure communication. Either you trust the not verified certificate and place that in the configuration file, or you should enable tls communication with certificates between teamcity and prometheus.

If you do not set one of the two options on prometheus you might end up with the issue:

Get /app/metrics remote error: tls handshake failure

The first option as discussed would be to trust the insecure certificate using https.

tls_config:
      insecure_skip_verify: true

The second option would be to communicate through certificates. First create a new folder inside your Prometheus installation and store the certificates for the teamcity server. Then you will need to point the files in the configuration.

    tls_config:
       ca_file:  /var/prometheus/ssl/signalocean.com.crt

The detailed options for tls_config in prometheus can be found below.

Configuration | Prometheus