Skip to content
  • There are no suggestions because the search field is empty.

Large CSV feed files not working with custom feeds in Netwitness

Issue

When using large csv files for custom feeds, the time taken for them to compile can be too great and result in a "Failed" message under the Feeds screen. This article explains a workaround.

The issue has been seen with csv feed files over 20MB. These may take over 20 minutes to compile on some systems.

Failed

The steps below explain how to reproduce the issue.
  1. Create your custom feed as normal.
  2. Notice that for large CSV files the feed will fail to apply to the decoder.

Resolution

A custom shell script can be created that will do the following:

  • Find existing feeds (on the Admin Server)
  • Recompile those feeds
  • Transfer (via salt-cp) to the Defined Decoder services
  • Reload the Feeds on the pre-defined Decoder services

Prerequisites:

  • Obtain the salt UUID of the decoder Hosts. This can be done by getting the UUID associated with the IP address of the host from /etc/hosts. In my below example, I want the UUIDs for 192.168.5.162 (my Network Decoder) and 192.168.5.166 (my Log Decoder):
    [root@NEW-NW11-NW-NODE-ZERO ~]# grep "192.168.5.162\|192.168.5.166" /etc/hosts
    192.168.5.166 7923c28f-225a-46ba-972c-ce3925b0b061 7923c28f-225a-46ba-972c-ce3925b0b061.netwitness
    192.168.5.162 ddc26106-dfce-4ba0-a057-999c7fa48c76 ddc26106-dfce-4ba0-a057-999c7fa48c76.netwitness
    • The BOLDED portion is my UUID for each:
      • 192.168.5.166    7923c28f-225a-46ba-972c-ce3925b0b061 7923c28f-225a-46ba-972c-ce3925b0b061.netwitness
      • 192.168.5.162    ddc26106-dfce-4ba0-a057-999c7fa48c76 ddc26106-dfce-4ba0-a057-999c7fa48c76.netwitness

ManualDeployFeeds.sh Script - Copy and paste this script to /root/ManualDeployFeeds.sh

find /var/lib/netwitness/uax/scheduler/ |grep xml >/tmp/feeds
for feed in $(cat /tmp/feeds)
do
FEEDDIR=$(dirname $feed)
FEEDNAME=$(basename $feed)
echo $FEEDDIR
echo $FEEDNAME
cd $FEEDDIR
NwConsole -c "feed create $FEEDNAME" -c "exit"
salt-cp '7923c28f-225a-46ba-972c-ce3925b0b061' --chunked *.feed* /etc/netwitness/ng/feeds
salt-cp 'ddc26106-dfce-4ba0-a057-999c7fa48c76' --chunked *.feed* /etc/netwitness/ng/feeds

NwConsole -k -c "tlogin server=192.168.5.166 port=56002 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "/decoder/parsers feed op=notify" -c "exit"
NwConsole -k -c "tlogin server=192.168.5.162 port=56004 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "/decoder/parsers feed op=notify" -c "exit"

done

Alter the following BOLDED values with those from your host(s):

  • salt-cp '7923c28f-225a-46ba-972c-ce3925b0b061'  - Use the UUID obtained from above
  • salt-cp 'ddc26106-dfce-4ba0-a057-999c7fa48c76'  - Use the UUID obtained from above
  • tlogin server=192.168.5.166  - Use the IP associated with the UUID obtained from above
  • tlogin server=192.168.5.162  - Use the IP associated with the UUID obtained from above
  • Note the Ports in the above examples. They are different for Network Decoders (56004) and Log Decoders (56002) , so if you add additional entries you need to ensure the NwConsole command ports are adjusted for the right type of service. 
  • Add additional entries for the salt-cp and tlogin command for additional hosts

Save the file, and make it executable:

chmod +x /root/ManualDeployFeeds.sh

Execute the script:  

./ManualDeployFeeds.sh

To confirm feeds were pushed, check for the most recent file names and timestamps on the targeted hosts with: 

[root@NW11-NETWORK-HYBRID feeds]# ls -lrth /etc/netwitness/ng/feeds
total 53M
-rw-r--r--. 1 root root 8 Mar 12 20:16 fakefeed.feed

Notes

More information on creating a custom feed can be found in NetWitness documentation.
https://community.netwitness.com/s/article/ManageCustomFeeds 


Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Log Decoder, Packet Decoder
NetWitness Version/Condition: 12.x
Platform: CentOS / AlmaLinux


Summary

When using large csv files for custom feeds, the time taken for them to compile can be too great and result in a Failed message under the Feeds screen. This article explains a workaround.


Approval Reviewer Queue

Technical approval queue