Step 2: Provider Connectivity Assurance for Mobility Performance Monitoring

Once the Fault & Mobility Performance Monitoring Collector has been installed (as covered in the previous step), the final step is to configure Provider Connectivity Assurance to ingest and process mobility performance monitoring data.

The high level process is as follows:

The step-by-step procedure is outlined below, and this diagram provides a visual of the components to be configured:

Getting ready

There are a few things you will need to set up before you continue with this configuration procedure.

Software tools needed

A text editor (notepad++, microsoft visual studio code, textedit, gedit, vim, nano, etc) to edit and save configurations. Do not use a word processor (word, wordpad, pages, etc).
A REST API client (Postman, Bruno) to query API endpoints and GET, PUT, POST, PATCH configurations as per the procedure.

API documentation

The Skylight Analytics API (now Provider Connectivity Assurance API) and Sensor Orchestrator API collections which contain APIs required in order to configure our Telemetry Collector. Each one can be downloaded from https://api.accedian.io by clicking on “Platform and Session Metric Query APIs” and “Sensor Agent Management APIs” (highlighted below) and then choose the download button at the top of each page. Once you have both files downloaded to your PC, import them as Collections in your REST API client. Screenshots:

Once you select either of these two API collections, the download button appears at the top of the page:

Personal Access Token

You will require a personal access token (PAT) to authenticate the Telemetry Collector with PCA. This will also be required in order to authenticate as a user and leverage APIs.

To create this access token, you need to access and log in to the Zitadel Administration user interface, which is included as the identity and access management service within Provider Connectivity Assurance. If DNS is available, the Zitadel Administration user interface is accessed at https://auth.<deployment URL>. If DNS is not available, use the IP address of the Provider Connectivity Assurance deployment tenant and the specific port 3443, i.e. https://<IP address>:3443

To create this access token, follow these steps in the Zitadel Administration user interface:

Change the organization to the Provider Connectivity Assurance tenant (The name “PCA” may have been used for the tenant during install, or the installer may have chosen some other name, check the drop down list to find your specific tenant name).
Under the Users Tab, use the button to create a new “Service User”, and choose + New button. (See screenshot below) There is drop down menu option about token type, choose ‘Bearer’ option.
Next we need to assign the “tenant-admin” role to this user. To do this, choose “Authorizations” menu option on the left side navigation, and choose the +New button. See screenshot below:
In the “Search for a Project” pop up, choose “Analytics”:
Select the “tenant-admin” role:

Generate a new Personal Access Token with an appropriate expiration date. Choose “Personal Access Tokens” and use + New button (Screenshot below) The token will need to be regenerated and replaced before its expiry, so choose a timeline that works for your organization.

Save this Personal Access Token to a file on your computer as you will not be able to retrieve it again later from the Zitadel administration user interface. It will be required in the Telemetry Collector configuration procedure, and also for sending API calls with the API client of your choice (Postman or Bruno).

The generated value of the token will look something like this:

pNy2-3NuLapIXxPHWokyUEc4H2i5t67NRyStCHOK466MvOVseVykfgeI4AOfzWKLT4kkvr3KWDg

REST API Client Setup

To configure the API client software you will need to configure your collection to use the Personal Access Token generated in a prior step. In your API client, navigate to the Collection Authorization configuration and choose Auth Type -> Bearer Token and paste the Personal Access Token in the Token field. Screenshot:

Now that these pre-requisites are complete, you are ready to start the configuration procedure.

1 Configure and Deploy Sensor Collector

Use the Provider Connectivity User Interface to create a New Sensor Collector by choosing Sensors-> Collectors->Sensor Collector.
Choose the + button on the right side to create a new sensor collector. Select Gateway in the Type field, and choose "Configure Sensor Collector" button to proceed, as shown in this screenshot:
Give the Sensor Collector a meaningful name. This needs to be a unique name as compared to all other named Sensor Collectors on the tenant, and it must not contain spaces.
Naming Tip - The best practice is to create a name that includes:
- an indication of the incoming data type
- name of the server upon which it will be deployed
- name of the Provider Connectivity tenant to which it is connected
e.g. PM_mobility_<server_name>_<tenant_name> would be a good name format, because these identifiers make it easy to find the Sensor Collector in the inventory later on, when it is listed amongst many Sensor Collectors.
In the Metric configuration field, select telemetry-collector as per the screenshot below.
Populate the zone value, using a unique value. The recommended best practice is to use the same value that was used as the Sensor Collector name. e.g. in this screenshot, the zone and the name are both identically set to “my-new-sensor-collector”:
Click the checkmark in the top right corner to save.
Select your Sensor Collector from the list, and click the ellipsis (three dots) in the upper right corner and choose "Download (docker)". This triggers the download to your local machine. Be patient, the package is built dynamically and it takes a few minutes to complete.

image 6.png

The banner at the top shows "preparing file for download... (this may take a minute)" as a reminder that patience is required:

image 7.png

Transfer the .tar file to the target machine where the Sensor Collector will run.
On the target machine, after you have transferred the .tar file, execute these commands:

mkdir sensorCollectorDir
mv sensorCollectorArchive.tar.gz sensorCollectorDir
cd sensorCollectorDir/
gunzip sensorCollectorArchive.tar.gz 
tar xvf sensorCollectorArchive.tar 
./run.sh

Once you have executed the run.sh shell script above, you should check that Sensor Collector is running correctly.
Instructions for checking if Sensor Collector is running correctly are here.

If you are having issues with running the Sensor Collector, information on troubleshooting Sensor Collector can be found here

When Sensor Collector is running correctly, you can proceed to the next section on Telemetry Collector installation.

2 Configure, Customize and Deploy Telemetry Collector

2.1 Create a Telemetry Collector Configuration

Every Telemetry Collector must have its' run time configuration defined in Provider Connectivity Assurance. This will be pushed down to the Telemetry Collector when it is initially deployed and "phones home" to obtain the configuration.

To prepare the Telemetry Collector instance for Mobility PM, the high level process is as follows:

Use the Provider Connectivity Assurance User Interface to create a Telemetry Collector configuration, choosing a default configuration which will be customized for our Mobility use case
Edit the default out-of-the-box configuration to customize it for Mobility PM and provide the connectivity (IP address) details for the Sensor Collector that is running in our environment
Finally, we will upload our customized configuration to the Provider Connectivity Assurance tenant, so that it is available to the Telemetry Collector when it “phones home”

The detailed steps are as follows:

Use the Provider Connectivity Assurance User Interface to create a New Telemetry Collector by choosing Sensors-> Collectors->Telemetry.
Click + on the upper right.
The New telemetry collector side bar appears, as show below:

Enter the details:
Name: Recommendation is to choose a name that will make it easy to correspond to the Sensor Collector.
Naming Tip - Use the name that was chosen for the Sensor Collector, prefixed with TC for “telemetry_collector”. e.g. TC_PM_mobility_<server_name>_<tenant_name>
Transform configuration: Choose the "Cisco-Telemetry-IOS-XR" option (which will be edited in the next step to work for Mobility PM)
Data collector server: Enter the IP address of the host where Sensor Collector is deployed, or alternatively, the docker bridge IP if you have installed the Sensor Collector and Telemetry Collector on the same host. Tip: Do not use localhost or 127.0.0.1.
Click ✓ to save the changes.

Now there is a saved Telemetry Collector configuration in Provider Connectivity Assurance that is provisioned to connect to the correct Sensor Collector, and the next step is to edit that configuration to work for Mobility PM instead of IOS XR telemetry.

2.2 Edit the transform configuration to ingest mobility PM data

In this step, we are editing and customizing the out-of-the-box IOS-XR Telemetry Collector configuration that we created in the previous step, and modifying it so that it will ingest mobility core PM.

The steps to follow are:

1. Download the telegraf.conf file from the Cisco box file share - note you must be logged in to Box to download the files successfully:

Cisco box

2. Using a text editor like nano or vi (not a word processor like Microsoft Word), edit the Telegraf.conf file. Find the inputs.kafka_consumer configuration and replace the brokers placeholder with the IP address of the machine where the Fault & Mobility PM Collector is running.

3. Save the Telegraf.conf file and execute this command in a terminal window to run Base64:

The output of base64 command is displayed in the screenshot below. The output will need to be copy-and-paste into a configuration file in the next steps, so keep the terminal open with this output.

Retrieve the existing configuration from Provider Connectivity Assurance using the "GET" API. This API requires the Agent ID which is available in the User Interface under Sensors-> Collectors->Telemetry, as highlighted here. Keep note of this Agent ID, it is required in a few upcoming steps:

The GET API call that requires the Agent ID is shown here:

If needed, here is the API reference documentation on retrieving the configuration by Agent ID .

In the response of the GET API call, find the following “dataCollection” location within the JSON structure and replace the base64 encoded telegraf config with the one created 2 steps ago.

6. Using Postman or the API client of your choice, use the API PUT command to send the new updated configuration back to Provider Connectivity Assurance, using the same Agent ID used in step 5. Here is the Reference API document for PUT command

2.3 Create your Telemetry Collector Secrets

The Telemetry Collector needs a secrets.yaml file containing its agent identifier and the Personal Access Token (PAT) for authentication.

Here is a sample file you can customize with your telemetry collector information

agentConfig:
    identification:
        agentId: {{agentIdFromPCA}}
        authenticationToken: {{patFromZitadel}}

Note: The secrets.yaml file is sensitive to spacing and alignment. Make sure to keep the same structure and copy the PAT as a single line. There should be one carriage-return (new line) after the PAT.

2.4 Deploy the Telemetry Collector for Mobility PM ingestion

Create a working directory for your Telemetry Collector on the host where you intend to run the instance.
In that directory, create a secrets.yaml file containing its agent identifier and the Personal Access Token (PAT) for authentication.
Here is a sample file you can customize with your Agent ID and PAT from previous steps.
```
agentConfig:
    identification:
        agentId: {{agentIdFromPCA}}
        authenticationToken: {{patFromZitadel}}
```
Note: The secrets.yaml file is sensitive to spacing and alignment. Make sure to keep the same structure and copy the PAT as a single line. There should be one carriage-return (new line) after the PAT.

In that same directory, create a docker-compose.yml file that resembles the following:

services:
  telemetry-collector:
    container_name: "telemetry-collector"
    image: "{{ telemetry-collector-image-name:version }}"  
    environment:
      AGENT_MANAGEMENT_PROXY: "{{ sensorcollector-ip }}"
      AGENT_MANAGEMENT_PROXY_PORT: 55777
    volumes:
      - ./secrets.yaml:/run/secrets/secrets.yaml

telemetry-collector-image-name:version corresponds to the latest available software version of the telemetry collector for your environment.
AGENT_MANAGEMENT_PROXY corresponds to your local Sensor Collector. The {{ sensorcollector-ip }} placeholder should be replaced with the host IP (or docker bridge IP) of your deployed Sensor Collector.
AGENT_MANAGEMENT_PROXY_PORT corresponds to the Agent Proxy port you chose when creating your Sensor Collector configuration. 55777 is the default value. However, if you choose something else, then this must match.

At this stage, you can deploy the Telemetry Collector by running the docker compose up command:

docker compose up -d

The Telemetry Collector will connect to Provider Connectivity Assurance and retrieve the configuration that we previously created.

Check the logs to ensure connectivity was successful

Obtain the container ID using these commands:

sudo docker ps

CONTAINER ID | IMAGE | COMMAND | CREATED | STATUS | PORTS |NAMES
923e4a3623b4 gcr.io/sky-agents/agent-telemetry-amd64:versiontag "/docker-entrypoint.…"   2 hours ago Up About an hour`

Now you can tail the logs for that container:

sudo docker logs -f <container id>

Here are the specific docker logs to search for that confirm that the Telemetry Collector has successfully connected and is operating correctly:

Additional troubleshooting tips

If for any reason you are not seeing the data show up in PCA, use this command on the host that is running the telemetry collector to confirm that it can reach the kafka bus, and that the expected PM Mobility data is flowing. This is like a "tail" command to monitor the kafka bus. This command doesn’t require anything on the host beyond docker and an internet connection. It will produce log messages that can be used to confirm that the Sensor Collector and Telemetry Collector are working properly:

docker run -it confluentinc/cp-kafka kafka-console-consumer --bootstrap-server {{server_ip}}:9092 --topic pca_kpi_topic

3 Install Ingestion Dictionaries for Mobility PM Data

This describes the high level process to ingest mobility PM data into Provider Connectivity Assurance:

Let the mobility PM data flow from the Fault and Mobility PM collector to Provider Connectivity Assurance (via the Telemetry Collector and the Sensor Collector) but do not enable the metrics by toggling them on in the user interface yet – there are changes to make before data is enabled/toggled on.
This will automatically create new object type Ingestion Dictionaries under the Settings -> Ingestion -> Openmetrics area within Provider Connectivity Assurance user interface.
In this procedure, we will be customizing those automatically-generated ingestion dictionaries in order to standardize on metric naming and assign units to the incoming metrics.
Once the dictionaries are customized, we will use the Provider Connectivity User Interface to “toggle on” and enable the mobility PM metrics.

3.1 Customize the automatically-created dictionaries

Once data is flowing and the default Openmetrics ingestion dictionaries show up in Provider Connectivity Assurance, use your API client software to retrieve them one-by-one for each mobility PM object type, and apply the customizations provided.

The steps are as follows:

Get the IDs of all the object type ingestion dictionaries on the PCA tenant
To get the IDs, use the broader GET “all” dictionaries API call, which returns all the IDs for all ingestion dictionaries in the tenant:

GET {{baseUrl}}/api/v3/ingestion-dictionaries

This document describes the ‘GET ALL’ API functionality.

From that superset of ALL ingestion dictionaries on the tenant, extract only the mobilitycore ingestion dictionaries, because they are the only ones to be customized for mobility PM
In the response in your API client from the previous step, search for object types that have the prefix “openmetricscisco-mobilitycore”, and note the ID for each of those dictionaries. Save this information as a list of IDs to be used in the subsequent customization steps. An example Ingestion Dictionary ID looks something like “openmetricscisco-mobilitycore-pm-workload-utilization”
Retrieve one ingestion dictionary at a time, using the dictionary ID

Use the GET API with one dictionary ID, to retrieve one dictionary at a time for editing

GET {{baseUrl}}/api/v3/ingestion-dictionaries/:IngestionDictionaryId

4. Edit the ingestion dictionary for one openmetricscisco-mobilitycore object type

In the response payload in your API client, edit each individual dictionary and replace the dictionaryType and metrics as follows:

Change the dictionaryType value from the default “custom” value to “global”
Open the corresponding customized ingestion dictionaries JSON file in the Cisco box file share - note you must be logged in to Box to download the files successfully:
Cisco box
While keeping the rest of the default dictionary JSON response intact, replace only the “data” -> “attributes” -> “metrics” array with the one you find in the custom dictionary file in Cisco box.
Be especially careful to preserve the “_id”, “_rev”, “dictionaryName” and “tenantId” values intact.
5. PATCH the new customized dictionary
Once your new custom dictionary is assembled, send it back to your Provider Connectivity Assurance tenant using the PATCH API
```
PATCH {{baseUrl}}/api/v3/ingestion-dictionaries/:IngestionDictionaryId
```
Repeat steps 3, 4 and 5 for every openmetricscisco-mobilitycore dictionary ID.

4 Enable Data Flow

Once your custom dictionaries are all patched in the Provider Connectivity Assurance tenant, data flow can be enabled in the PCA UI.

Reload the UI in your browser window (because dictionaries are loaded in memory upon login, they will need to be refreshed) then navigate to Settings -> Ingestion -> Openmetrics.
Look for cisco-mobilitycore object types, validate that all your metrics are showing up with user friendly camelCase names. If anything is not camelCase, double check that you patched that dictionary, and report any issues over intercom.
For each object type, toggle on the metrics that are needed by your customer and use-case.
Once you are done for an object type, make sure to save your selections using the top right checkmark button

4.1 Create metadata categories

Navigate to Settings -> Metadata -> Categories
Click the “ + Add category ” button and add the following categories
• agentId
• index
• node_id
• schema
• source_ip

4.2 Assign dynamic metadata categories

Navigate to Settings -> Metadata -> Dynamic Mappings
For each of the categories created in the previous step, run a search and map each Field to a Mapping for each cisco-mobilitycore-pm-* Datasource.
Click the blue checkmark icon to save the configuration

5 Set-Up Dashboards

Refer to these instructions for detailed documentation on authoring Dashboards, and here is a summary of key steps for creating mobility PM dashboards to get you started:

To edit a dashboard, enter Author mode:

And add a card:

Cards can have one or more Widgets (e.g. Aggregate, Timeseries, Table):

Widgets visualize data. Mobility PM data are categorized primarily by “node_id” and “index”.
“node_id” is an entity identifier and can be used for the most detailed breakdowns of data.
“index” is used for macro level categorization and has values that represent various concepts, such as APN or SMF

If specific metadata categories aren’t present for a given entity the name as an indicator of additional categorization, e.g. the GTP-U suffix present in this entity name

Filtering to specific entities can be done using the Session identifier.

Categories can be used to create effective dashboards.

For instance, creating a Network Summary dashboard with Top N “node_id” Throughput trends or creating a more specific breakdown filtered to a given set of “index” values such as APN.

Multiple metrics can be combined in the same widget for comparison, including permutations of metrics and categories.

For convenience, Widgets, Cards and Dashboards can be cloned and Dashboards may be exported and imported between systems, provided that the same metrics and metadata are present on the importing system.

For more detailed instructions on how to configure Widgets, refer to these instructions.

Step 2: Provider Connectivity Assurance for Mobility Performance Monitoring - Configuration Procedure