Skip to content
  • There are no suggestions because the search field is empty.

sshd_config and iptables are suddenly changed back to default in admin server

Issue

In some cases, you may observe that sshd_config and iptables files are automatically changed in SA server.
During Netwitness upgrade or patch, this is an expected behavior by chef run list. 
But in this case, these files were suddenly changed back to default on 9/13 without upgrade or patch activity.
User-added
User-added

You can see that the refresh host orchestration task is triggered from Jetty in sa.log: 
2021-09-13 17:04:19,998 [listenerContainer-1] INFO  com.rsa.smc.sa.admin.management.yum.StagedRepoEngine - UpdateHostTask(super=AsynchronousTask(super=Task(id=613f057badfb20061b9add11, host=Host(id=cc4ec6f1-9162-4097-a4e8-ac640abc8595, hostname=null, ipv4=null, ipv4Public=null, displayName=null, version=null, thirdParty=false), queued=2021-09-13T08:02:03.369Z, start=2021-09-13T08:02:03.377Z, end=null, status=RUNNING, childTasks=[], parentTaskId=null), successful=false), stdout=null, stderr=null, version=11.4.1.1, stageDir=null, updateTaskType=RefreshHost, triggeredHostsWithError=null, nodeInfraServerId=null, noTriggers=false)
This is what triggered chef to run with the full SA run_list as seen in the chef-solo.log:
[2021-09-13T17:02:37+09:00] INFO: Setting the run_list to ["recipe[nw-repositories]", "recipe[nw-dns]", "recipe[nw-firewall]", "recipe[nw-ntp]", "recipe[nw-dns-client]", "recipe[nw-pki]", "recipe[salt-minion]", "recipe[nw-hwrpm]", "recipe[nw-java]", "recipe[rsa-audit]", "recipe[rsa-sms-runtime]", "recipe[nw-rabbitmq]", "recipe[rsa-collectd]", "recipe[nw-rng]", "recipe[ssh]", "recipe[auditd-wrapper]", "recipe[nw-base]", "recipe[salt-master]", "recipe[nw-re-server]", "recipe[fne-server]", "recipe[nw-mongo]", "recipe[nw-nginx]", "recipe[nw-appliance]", "recipe[rsa-audit-server]", "recipe[rsa-sms-server]", "recipe[salt-api]", "recipe[rsa-security-server]", "recipe[metricbeat]", "recipe[rsa-config-server]", "recipe[rsa-license-server]", "recipe[rsa-orchestration-server]", "recipe[rsa-admin-server]", "recipe[rsa-source-server]", "recipe[rsa-integration-server]", "recipe[rsa-investigate]", "recipe[rsa-node-infra-server]", "recipe[rsa-content-server]", "recipe[rsa-response]", "recipe[nw-jetty]", "recipe[nw-broker]"] from CLI options
 
As a result, it reverted the changes made to sshd_config as well as iptables.

The reverted changes are the normal behavior since chef detects any configuration changes then uses a predefined template (/var/netwitness/config-management/cookbooks/platform/ssh/templates/default/sshd_config.erb) to regenerate the default config file. 

Currently there is no way to bypass such changes to the configuration other than changing the needed fields in the template itself but this is neither supported nor recommended. If customer is experiencing this issue again, they should contact support to seek for further guidance before proceed with any changes

Resolution

*Root Cause

If you add NTP servers in the NW UI, the refresh host will run as a result.

A chef-runis triggered on each provisioned host with the run_list including [nw-dns] on the SA and [nw-dns-client] on each provisioned host.

After the first chef-run is completed on the SA, another chef run is triggered which includes the SA’s full run_list without the [nw-pre-install] and [nw-post-install]. 

This run_list is the same as the one that gets used if you run orchestration-cli-client --update-admin-node.

You can observe following logs in orchestration-client.log at the time of the chef-run as an evidence.

2021-09-13 17:02:36,642 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=cc4ec6f1-9162-4097-a4e8-ac640abc8595, ADDR=3.1.2.30, NAME=nwserver, VERSION=11.4.1.1
2021-09-13 17:02:36,645 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=319af78b-bea6-4113-88e1-70cd83245edb, ADDR=3.1.2.32, NAME=nwhybrid, VERSION=11.4.1.1
2021-09-13 17:02:36,645 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=956c58be46ac49cda1b33d288aee9d2e, ADDR=3.1.2.5, NAME=null, VERSION=null
2021-09-13 17:02:36,661 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Tasks completed successfully...
2021-09-13 17:02:36,661 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Request completed successfully.
2021-09-13 17:05:18,984 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=cc4ec6f1-9162-4097-a4e8-ac640abc8595, ADDR=3.1.2.30, NAME=nwserver, VERSION=11.4.1.1
2021-09-13 17:05:18,987 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=319af78b-bea6-4113-88e1-70cd83245edb, ADDR=3.1.2.32, NAME=nwhybrid, VERSION=11.4.1.1
2021-09-13 17:05:18,987 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=04bccb1d1e8547c3ae639210d1547d2a, ADDR=3.1.2.5, NAME=null, VERSION=null
2021-09-13 17:05:18,987 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Host: ID=d14de949dc5e44a6b46fe93121aeb785, ADDR=100.11.52.223, NAME=null, VERSION=null
2021-09-13 17:05:19,002 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Tasks completed successfully...
2021-09-13 17:05:19,002 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Request completed successfully.


Note) the additional host (100.11.52.223) that appeared in orchestration as a result of the change from the UI. 


Product Details

Product Set: NetWitness Platform
RSA Product/Service Type: All Nodes
RSA Version/Condition: 11.x, 12.x
Platform: CentOS, AlmaLinux
 


Approval Reviewer Queue

Technical approval queue