000039312 - Feed and Parser related metrics are not available for the New Health and Wellness in RSA NetWitness Platform 11.5

Document created by RSA Customer Support Employee on Sep 20, 2020Last modified by RSA Customer Support Employee on Sep 20, 2020
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000039312
Applies ToRSA Product Set: NetWitness Platform
RSA Product/Service Type: Health & Wellness/Metrics Server
RSA Version/Condition: 11.5
Platform: CentOS
O/S Version: 7
IssueThe Feed and Parser metrics from the Network/Log Decoders are not available in the New Health & Wellness area. The metrics for Feeds and Parsers are disabled by default to keep the Health & Wellness system from hitting its default index field limit.
ResolutionThis document is divided into several sections. Perform the following actions to enable the Feed and Parser metrics within the Network and Log Decoders using the New Health & Wellness system.

Note: The following steps must be performed for each Network Decoder or Log Decoder where the Parser and Feed metrics are required.

 

Retrieving the Core Service UUID


This article requires the retrieval of a Network or Log Decoder's Service ID within the NetWitness system. The following steps will show how to retrieve the Service ID for use in this document.

  1. Log into the NetWitness UI and go to User-added image (Admin Area) > Service and select a Network or Log Decoder.
  2. Go to the Action icon for the service and click View > Explore.
  3. Expand the sys node, and select the stats node.
  4. In the left-hand panel look for UUID (uuid) and copy the id for later use. Ex UUID: 0089b3dd-715d-43ed-932b-e9e1777e3625
 

Retrieving the Include/Exclude Metrics Criteria for a Network/Log Decoder



  1. SSH into the Admin Server (generally the server containing the UI).
  2. Access the nw-shell program, by typing nw-shell at the prompt.


# nw-shell


  1. Connect to the metrics-server using connect --service metrics-server.


offline >> connect --service metrics-server


  1. Type login to log into the metrics-server and use the NetWitness administrator account credentials.


metrics-server:Folder:/rsa >> login
user: admin
password: ***********
admin@metrics-server:Folder:/rsa >>


  1. Retrieve the current configuration for the Network Decoder or Log Decoder being adjusted and output it to a custom file on /root.


admin@metrics-server:Folder:/rsa >> cd /rsa/metrics/elastic/get-config
admin@metrics-server:Folder:/rsa/metrics/elastic/get-config >> invoke <Service-ID> outputFile /root/serviceConfig.json


Note: To retrieve the Service-ID used by this step, see Retrieving the Core Service UUID section in this article.

 

Updating the Include/Exclude Metrics Criteria for a Network/Log Decoder


Before proceeding with this section the Metrics-Server configuration for a Network/Log Decoder needs to be saved to a JSON file. See Retrieving the Include/Exclude Metrics Criteria for a Network/Log Decoder in this article before continuing. This section of the article will focus on adjustments to the Inclusion and Exclusion sections from the JSON file that was created earlier.

See the screenshot for an example of the output file that will be used.
User-added image
  1. Create a backup of the output file. Since the original file will be edited manually, it is important to have a clean backup should the original configuration need to be reapplied.
  2. Edit the original output file in /root with vi.

    # vi /root/serviceConfig.json

  3. Once editing the JSON file, to allow the New Health & Wellness to retrieve feed and parser statistics, remove the "/decoder/parsers/feeds/**" entry under the "exclusion" section. The ending coma for the line should also be removed.

    Original Exclusion Section
    "exclusion" : [
        "/decoder/config/rules/**",
        "/decoder/config/recovery/**",
        "/decoder/parsers/feeds/**",    <--- This is the line in question
        "/sys/config/scheduler/**",
        "/sdk/stats/queries/**",
        "/decoder/devices/**"
      ],
     
    Adjusted Exclusion Section
    "exclusion" : [
        "/decoder/config/rules/**",
        "/decoder/config/recovery/**",
        "/sys/config/scheduler/**",
        "/sdk/stats/queries/**",
        "/decoder/devices/**"
      ],


    Note: When working with Core Services Inclusion/Exclusion rules, such as Network/Log Decoders, it is important to remember that the exclusion rules take precedence over the inclusion rules. In the JSON file output under the Inclusion section, it can be seen that /decoder/* is included, yet /decoder/parsers/feeds* is excluded which causes the parser and feeds to not provide statistics to the New Health and Wellness.

  4. Save serviceConfig.json file in vi with :wq!
  5. Access the nw-shell program, by typing nw-shell at the prompt.

    # nw-shell

  6. Connect to the metrics-server using connect --service metrics-server.

    offline >> connect --service metrics-server

  7. Type login to log into the metrics-server and use the NetWitness administrator account credentials.

    metrics-server:Folder:/rsa >> login
    user: admin
    password: ***********
    admin@metrics-server:Folder:/rsa >>

  8. Set the configuration for the Network Decoder or Log Decoder to the new settings using the custom file in /root.

    admin@metrics-server:Folder:/rsa >> cd /rsa/metrics/elastic/set-config
    admin@metrics-server:Folder:/rsa/metrics/elastic/get-config >> invoke file /root/serviceConfig.json

  9. The specific UUID referenced Network/Log Decoder is now updated to provide parser and feed statistics to the New Health & Wellness.
 

Adjusting Index Field Limits in the Health & Wellness UI


This section is designed to help adjust Indexing Field limits, via the nw-index-template, that can be reached after enabling additional statics on the Network/Log Decoders. The default index field limit in the New Health & Wellness is set to 20,000 fields. If there are errors in the /var/log/messages or elasticsearch.log, then the index field will need to be adjusted.

See the following log error example:

Telemetry server responded with error code '400':
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [20000] in index [nw-decoder]
has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [20000] in index [nw-decoder]
has been exceeded"},"status":400}

To update the nw-index-template, follow these steps:

  1. Log into the NetWitness UI.
  2. Click the User-added image (Admin Area) icon, then click Health and Wellness.
  3. Click the New Health & Wellness tab and click on the Pivot to Dashboard button. This will open the new Deployment Health Overview
  4. Click on the User-added image Dev Tools icon, which opens the browser-based Dev Tools Console.
  5. Clear all content in the left panel and type in GET _template/nw-index-template and click the User-added image small right arrow (Run) to retrieve the existing nw-index-template.
  6. Check the existing index limit from the right-hand panel. See the screenshot for and example where it is set to 20000. Make an external copy of the complete template for later use.
User-added image
  1. Click on User-added image Management, then click on Index-Patterns.
  2. Click the nw* under the Index Pattern list.
  3. At the top of the page will be Fields (#####) where the ##### represents the number of fields currently being indexed.
  4. If the number from the Fields (####) is larger than the index limit retrieved in step 6, an index template adjustment will be required.
  5. Go to the User-added image Dev Tools area.
  6. Update the index limit in the nw-index-template that was copied in step 6. For example, changing the limit from 20000 to 28000.
  7. Copy the template into the right-hand panel in the Dev Tools Console area.
  8. In the left-hand panel type the following, PUT _template/nw-index-template and click the User-added image small right arrow (Run) to update the nw-index-template on the system.

Warning: Ensure that the whole template is copied into the right panel and the limit is updated before running the PUT command. If the whole template is not used, it will break Health & Wellness.


Note: For more information concerning mapping limits within the new Health and Wellness, please see Elastic Mapping (Non-RSA Site)


  1. Go to User-added image Management, then Index Patterns.
  2. Click on the nw-decoder* pattern. Once it opens, click the User-added image Refresh icon and then the Refresh button.
  3. Go back to Index Patterns, click nw-logdecoder*. Once open, click the User-added image Refresh icon and then the Refresh button.
At this point, the required field index limits have been increased for the Network and Log Decoders based on the new nw-index-template. If there are any questions or issues while using this article, please open an RSA NetWitness Support Case.

Attachments

    Outcomes