Splunk Core

Links and Access

1. NonProd Splunk - Secure Group: cloud_splunk_east_rx_optima_user
2. Prod Splunk - Secure Group: cloud_splunk_east_rx_optima_user

Overview

1. Splunk Setup

2. Custom Logging Dashboard

Splunk Setup

Steps

1. Helm Install.

2. Application Insights Setup.

Helm Install

1. All the monitoring is packed in a single Helm Chart.

2. To modify the deployment, make changes to the values files here.

3. For exporting the logs from the AKS we utilize a Kafka Exporter, which comes by default by OTEL Collector. It requres the

4. App Insights Instrumentation Key
5. Event Hub Broker Endpoint
6. Event Hub Password

Application Insights

From Application Insights we will capture the values needed to export the data.

1. Go to Application Insights

Custom Logging Dashboard

1. Splunk Logs Dashboard - NonProd
2. Splunk Logs Dashboard - Prod

Application Logs

1. Search for all the logs.

   index=cloud_rx_optima
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.body.stringValue output=logs
   | table logs
   | where isnotnull(logs)
   

2. List all the service.

   index=cloud_rx_optima 
   | spath path=data.resourceLogs{}.resource.attributes{} output=attributes
   | rex field=attributes "\{\"key\"\:\"service.name\"\,\"value\"\:\{\"stringValue\"\:\"(?<serviceValue>.+)\"\}\}"
   | where isnotnull(serviceValue)
   | stats count by serviceValue
   | fields - count
   

3. List all the environemnts.

   index=cloud_rx_optima 
   | spath path=data.resourceLogs{}.resource.attributes{} output=attributes
   | rex field=attributes "\{\"key\"\:\"deployment.environment\"\,\"value\"\:\{\"stringValue\"\:\"(?<environment>.+)\"\}\}"
   | where isnotnull(environment)
   | stats count by environment
   | fields - count
   

4. Search logs for particular enviornmnet

   index=cloud_rx_optima 
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.body.stringValue output=log 
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.severityText output=severity 
   | spath path=data.resourceLogs{}.resource.attributes{}.key output=key
   | spath path=data.resourceLogs{}.resource.attributes{}.value.stringValue output=value
   | spath path=data.resourceLogs{}.resource.attributes{} output=attributes
   | search key="deployment.environment" value="OptimaStage"
   | rex field=attributes "\{\"key\"\:\"service.name\"\,\"value\"\:\{\"stringValue\"\:\"(?<serviceName>.+)\"\}\}"
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.key output=exlude_key
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.value.stringValue output=exlude_value
   | search NOT (exlude_key="logtype" AND exlude_value="stderr")
   | where isnotnull(log) AND isnotnull(serviceName)
   | mvexpand log
   | mvexpand severity
   | table serviceName, log, severity
   

5. Search for logs with respect to Service Name, Environment Name, Namespace Name, Pod Name, Container Name.

   index=cloud_rx_optima 
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.body.stringValue output=log 
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.severityText output=severity 
   | spath path=data.resourceLogs{}.resource.attributes{}.key output=key
   | spath path=data.resourceLogs{}.resource.attributes{}.value.stringValue output=value
   | spath path=data.resourceLogs{}.resource.attributes{} output=attributes
   | search key="service.name" value="optima-uwcm-operate-service"
   | search key="deployment.environment" value="OptimaStage"
   | search key="k8s.namespace.name" value="optima-uwcm"
   | search key="k8s.pod.name" value="camunda-platform-operate-7757dd88dd-zmltd"
   | search key="k8s.container.name" value="operate"
   | rex field=attributes "\{\"key\"\:\"service.name\"\,\"value\"\:\{\"stringValue\"\:\"(?<serviceName>.+)\"\}\}"
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.key output=exlude_key
   | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.value.stringValue output=exlude_value
   | search NOT (exlude_key="logtype" AND exlude_value="stderr")
   | where isnotnull(log) AND isnotnull(serviceName)
   | mvexpand log
   | mvexpand severity
   | table serviceName, log, severity
   

Note

1. Remove or add the Search statements for removing or adding anymore query.
2. To suppress the Otel Java agent errors, below lines are added. They can be removed after the Otel Javaagent have been removed. | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.key output=exlude_key | spath path=data.resourceLogs{}.scopeLogs{}.logRecords{}.attributes{}.value.stringValue output=exlude_value | search NOT (exlude_key="logtype" AND exlude_value="stderr")