Splunk Alert and Metric Integration

This article explains the Provider Connectivity Assurance alert export to Splunk Enterprise feature, enabling a seamless integration, and offering a consolidated view of service health by combining alerts from multiple sources that collect data from logs, metrics, telemetry, events, firewalls, and third-party applications.

Unveiling Cross-Domain Performance Observability

Provider Connectivity Assurance integration with Splunk Enterprise, uses HTTP Event Collector (HEC), which is better suited than REST APIs for alert event notifications between the Provider Connectivity Assurance and Splunk. The HEC advantage is its ability to handle high-volume data streams efficiently. Where REST APIs, require individual requests, HEC can process multiple events in a single HTTP request, reducing overhead and latency. HEC supports token-based authentication for security and allows direct event ingestion into Splunk Enterprise without requiring additional parsing or processing, making it ideal for real-time monitoring and analytics use cases.

HEC vs. REST: Optimal Data Integration

Which Splunk Solution is Right for You?

Splunk Cloud and Enterprise are both powerful data analytics platforms.

• Splunk Enterprise: Self-managed, deployed on-premises or in the cloud, offering more control but requiring internal maintenance, scaling, and infrastructure management.
• Splunk Cloud: Fully managed, cloud-based, and ideal for quick scaling and avoiding on-prem management complexity.

Both have similar functionality, Splunk Enterprise provides more customization and control. Splunk Cloud simplifies operations with a managed service. Provider Connectivity Assurance integration for Splunk is supported for Splunk Enterprise currently, starting with release 25.02.

Provider Connectivity Assurance brings to Splunk granular micro-second performance monitoring with Splunk Enterprise, leading the Provider Connectivity Assurance to accelerate decisions, troubleshooting and reduce alert fatigue. By aggregating and correlating data across network domains to generate fewer and more relevant high-value alerts, it ensures focus is on critical services impacting issues while also enhancing operational intelligence.

Making it Real with Real Networks

Root Cause Analysis and Closed-Loop Use Case

Businesses can break down silos, gain actionable insights, and accelerate issue diagnosis by correlating key events with performance metrics for a more comprehensive understanding of their entire infrastructure.

This combined solution ensures, security, and compliance, and offers unmatched capabilities like millisecond-level monitoring and in-depth cross-domain insights that many other solutions do not offer.

How to Enable Splunk Integration

1. Contact CSM and request that the Splunk Feature Flag is enabled.
2. Configure Alert Export, using the REST API to create an Alert Export Configuration Entry in Provider Connectivity Assurance with necessary parameters.
   Note: API Documentation is available here.

Example API Payload:

```bash
// URL Endpoint: https://<tenant>/api/v3/alertexport
// HTTP Method Post
{
  "data": {
    "type": "alertExports",
    "attributes": {
      "alertexportname": "<name for alert export>",
      "alertexporturl": "<url with protocol, host and port number>",
      "alertexportenabled": true,
      "alertexporttoken": "<Splunk HEC Alert Export Token>",
      "alertpolicylist": [
        "list of alert policies that you would like to export alerts for using this connection"
      ],
      "alertexporttype": "splunk-hec"
    }
  }
}

```

3. Alerts generated for the specified alert policies will now be exported to your Splunk's HEC instance. For further information on how to Setup and use HTTP Event Collector in Splunk Web. See Splunk Enterprise Getting Data In to learn more.

Technical Considerations and Requirements

Firewall Connectivity

When connecting the Provider Connectivity Assurance platform to Splunk via HEC behind a firewall, you will need to ensure the firewall rules allow traffic on the correct port (TCP 8088 on Splunk Cloud Platform free trials or TCP 443 by default on Splunk Cloud Platform instances).

Important requirements:

• Inbound rules need to permit traffic from trusted sources.
• Ensure token-based authentication is configured.
• Keep the HEC token secure and avoid hardcoding it in configuration files.
• Enable logging and monitoring of connections to ensure traffic to the HEC is not being blocked or dropped by the firewall.

Important Consideration: Cases where firewalls are configured for IP Whitelisting
If a firewall is using IP whitelisting, make sure that Splunk HEC server’s IP address are permitted.

Handling Self-Signed Certificates

If your Splunk instance uses a self-signed certificate, configure the Splunk HTTP Event Collector to use HTTP, as the Provider Connectivity Assurance cannot validate self-signed certificates.

Alert Export to Splunk Cloud

The current Provider Connectivity Assurance architecture does not allow direct outbound connectivity from worker nodes to Splunk Cloud. Address this limitation with a careful design plan if necessary.

Performance and Scaling

The standard Alert Export deployment efficiently handles up to 2,000 alerts per minute.