Upgrading to RSA NetWitness 11.x fails due to a parse error
Issue
Upgrading from 11.3.1.1 to 11.4.1.2 via the NetWitness Admin UI fails and the host status changes to 'Update Error'.The following error is found in /var/lib/netwitness/config-management/cache/stack-trace.
Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/2.5.0/net/http/response.rb:122:in 'error!'
/opt/chef/embedded/lib/ruby/2.5.0/net/http/response.rb:122:in 'error!'
/var/log/messages shows more meaning parsing errors as below.
Aug 14 00:07:24 netwsa-as-01p salt-minion: [2020-08-14T00:07:24+00:00] ERROR: #<FFI_Yajl::ParseError: parse error: trailing garbage
Aug 14 00:07:24 netwsa-as-01p salt-minion: Tls" : true } } }, "override_attributes" : { }
Aug 14 00:07:24 netwsa-as-01p salt-minion: (right here) ------^
Aug 14 00:07:24 netwsa-as-01p salt-minion: >
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:87:in `do_yajl_parse'
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:87:in `parse'
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:46:in `parse'
Aug 14 00:07:24 netwsa-as-01p salt-minion: Tls" : true } } }, "override_attributes" : { }
Aug 14 00:07:24 netwsa-as-01p salt-minion: (right here) ------^
Aug 14 00:07:24 netwsa-as-01p salt-minion: >
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:87:in `do_yajl_parse'
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:87:in `parse'
Aug 14 00:07:24 netwsa-as-01p salt-minion: /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/ffi-yajl-2.3.1/lib/ffi_yajl/parser.rb:46:in `parse'
================================================================================
Aug 14 00:07:24 netwsa-as-01p salt-minion: Error expanding the run_list:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ================================================================================
Aug 14 00:07:24 netwsa-as-01p salt-minion: Unknown Server Error:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ---------------------
Aug 14 00:07:24 netwsa-as-01p salt-minion: The server had a fatal error attempting to load a role.
Aug 14 00:07:24 netwsa-as-01p salt-minion: Server Response:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ----------------
Aug 14 00:07:24 netwsa-as-01p salt-minion: Exception raised! #<FFI_Yajl::ParseError: parse error: trailing garbage
Aug 14 00:07:24 netwsa-as-01p salt-minion: Tls" : true } } }, "override_attributes" : { }
Aug 14 00:07:24 netwsa-as-01p salt-minion: (right here) ------^
Aug 14 00:07:24 netwsa-as-01p salt-minion: >
Aug 14 00:07:24 netwsa-as-01p salt-minion: Error expanding the run_list:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ================================================================================
Aug 14 00:07:24 netwsa-as-01p salt-minion: Unknown Server Error:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ---------------------
Aug 14 00:07:24 netwsa-as-01p salt-minion: The server had a fatal error attempting to load a role.
Aug 14 00:07:24 netwsa-as-01p salt-minion: Server Response:
Aug 14 00:07:24 netwsa-as-01p salt-minion: ----------------
Aug 14 00:07:24 netwsa-as-01p salt-minion: Exception raised! #<FFI_Yajl::ParseError: parse error: trailing garbage
Aug 14 00:07:24 netwsa-as-01p salt-minion: Tls" : true } } }, "override_attributes" : { }
Aug 14 00:07:24 netwsa-as-01p salt-minion: (right here) ------^
Aug 14 00:07:24 netwsa-as-01p salt-minion: >
Cause
The parser error is due to incorrect syntax in /etc/netwitness/config-management/environments/netwitness.json. This may occur due to a human error while following a KB like 36446.For example, an extra curly brace and a comma below "global" cause a parsing error while closing the object towards to the end, just before "override_attributes".
{
"name" : "netwitness",
"description" : "Default environment",
"chef_type" : "environment",
"json_class" : "Chef::Environment",
"default_attributes" : {
"nw-repositories" : {
"repo_base" : {
"url" : "file:///var/netwitness/common/repo"
}
},
"global" : {
},
"mongo" : {
"port" : "27017",
"host" : {
"hostname" : "10.1.1.1"
},
"useTls" : true
},
"nodezeroipaddress" : "10.1.1.1",
"nw-services" : [ ],
"rabbitmq" : {
"port" : "5671",
"host" : {
"hostname" : "10.1.1.1"
},
"use Tls" : true
}
}
},
"override_attributes" : { }
}
Some non-printable garbage characters may cause such problem as well.
"name" : "netwitness",
"description" : "Default environment",
"chef_type" : "environment",
"json_class" : "Chef::Environment",
"default_attributes" : {
"nw-repositories" : {
"repo_base" : {
"url" : "file:///var/netwitness/common/repo"
}
},
"global" : {
},
"mongo" : {
"port" : "27017",
"host" : {
"hostname" : "10.1.1.1"
},
"useTls" : true
},
"nodezeroipaddress" : "10.1.1.1",
"nw-services" : [ ],
"rabbitmq" : {
"port" : "5671",
"host" : {
"hostname" : "10.1.1.1"
},
"use Tls" : true
}
}
},
"override_attributes" : { }
}
Resolution
Take a backup of /etc/netwitness/config-management/environments/netwitness.json and modify it to remove the syntax errors.Product Details
RSA Product Set: NetWitness PlatformRSA Product/Service Type: NetWitness Platform
RSA Version/Condition: 11.x
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue