How to collect core files on CentOS 7 when nwe-agent logs shows segfault

Issue

A segmentation fault (aka segfault) is a common condition that causes programs to crash; they are often associated with a file named core. Segfaults are caused by a program trying to read or write an illegal memory location
We can see "segfault" under /var/log/messages | grep nwe-agent.
May 8 17:58:20 mrtg_uw_01 kernel: nwe-agent[13416]: segfault at 1a0 ip 0000000000434d51 sp 00007f2734188a70 error 4 in nwe-agent[400000+2f7000]

When agent is installed - Service shows running:

[root@mrtg_uw_01 tmp]# service nwe-agent status
â— nwe-agent.service - LSB: Starts nwe agent
Loaded: loaded (/etc/rc.d/init.d/nwe-agent; bad; vendor preset: disabled)
Active: active (running) since ìˆ˜ 2019-05-08 17:52:10 KST; 5min ago
Docs: man:systemd-sysv-generator(8)
Process: 4433 ExecStop=/etc/rc.d/init.d/nwe-agent stop (code=exited, status=0/SUCCESS)
Process: 4521 ExecStart=/etc/rc.d/init.d/nwe-agent start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nwe-agent.service
â””â”€4529 /opt/rsa/nwe-agent/bin/nwe-agent

5ì›” 08 17:52:10 mrtg_uw_01 systemd[1]: Starting LSB: Starts nwe agent...
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: Starting ecat agent-linux...
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4521]: Starting nwe agent: [ OK ]
5ì›” 08 17:52:10 mrtg_uw_01 systemd[1]: Started LSB: Starts nwe agent.
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ECAT running from: /
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ***************AGENT CONFIGURATION ******************** :Agent id : 02cefbf1-4730-0cd6-3927-43a436a7e779, Primary server : https://10.11.13.69:443, Primary udp : 444,Assig...85-aa28a0dfcc83 ,
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: **************ECAT CLIENT CERTIFICATE**************** : Version : 3 ,Subject : /CN=EcatClientExported,Issuer : /CN=EcatCA,Serial : 2D50F0ABF59AAFB04761C38D984B9E45,Thumbpr...23:59:59 2039 GMT
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: Loaded certificate and key successfully from: /opt/rsa/nwe-agent/config/client-certificate.p12
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ============================= Agent version 4408 ============================
Hint: Some lines were ellipsized, use -l to show in full.

Service crashes
[root@mrtg_uw_01 tmp]# service nwe-agent status
â— nwe-agent.service - LSB: Starts nwe agent
Loaded: loaded (/etc/rc.d/init.d/nwe-agent; bad; vendor preset: disabled)
Active: active (exited) since ìˆ˜ 2019-05-08 17:52:10 KST; 7min ago
Docs: man:systemd-sysv-generator(8)
Process: 4433 ExecStop=/etc/rc.d/init.d/nwe-agent stop (code=exited, status=0/SUCCESS)
Process: 4521 ExecStart=/etc/rc.d/init.d/nwe-agent start (code=exited, status=0/SUCCESS)

5ì›” 08 17:52:10 mrtg_uw_01 systemd[1]: Started LSB: Starts nwe agent.
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ECAT running from: /
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ***************AGENT CONFIGURATION ******************** :Agent id : 02cefbf1-4730-0cd6-3927-43a436a7e779, Primary server : https://10.11.13.69:443, Primary udp : 444,Assig...85-aa28a0dfcc83 ,
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: **************ECAT CLIENT CERTIFICATE**************** : Version : 3 ,Subject : /CN=EcatClientExported,Issuer : /CN=EcatCA,Serial : 2D50F0ABF59AAFB04761C38D984B9E45,Thumbpr...23:59:59 2039 GMT
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: Loaded certificate and key successfully from: /opt/rsa/nwe-agent/config/client-certificate.p12
5ì›” 08 17:52:10 mrtg_uw_01 nwe-agent[4529]: ============================= Agent version 4408 ============================
5ì›” 08 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scan request recieved
5ì›” 08 17:57:50 mrtg_uw_01 nwe-agent[4529]: Time zone of the machine is : Greenwich
Mean Time
5ì›” 08 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning loaded images list
5ì›” 08 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning running processes
Hint: Some lines were ellipsized, use -l to show in full.

/var/log/messages | grep nwe-agent

May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: Starting ecat agent-linux...
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: Starting ecat agent-linux...
May 8 17:52:10 mrtg_uw_01 nwe-agent: Starting nwe agent: [ OK ]
May 8 17:52:10 mrtg_uw_01 nwe-agent: Starting nwe agent: [ OK ]
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ECAT running from: /
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ECAT running from: /
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ***************AGENT CONFIGURATION ******************** :Agent id : 02cefbf1-4730-0cd6-3927-43a436a7e779, Primary server : https://10.11.13.69:443, Primary udp : 444,Assigned server : https://10.11.13.69:443,Assigned server udp : 444,Assigned server id : 97aab137-51de-45c1-8185-aa28a0dfcc83 ,Server certificate thumbprint : C1E0B485A021FDE70E16EF5012720E9501707193,Beacon interval : 15.000000,Preferred server : ,Relay server : https://:443,Relay server udp: 444,Relay ESH name: ,Self destroy: 0,Server notified : 1,Server discovery done : 1,Agent updated : 0
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ***************AGENT CONFIGURATION ******************** :Agent id : 02cefbf1-4730-0cd6-3927-43a436a7e779, Primary server : https://10.11.13.69:443, Primary udp : 444,Assigned server : https://10.11.13.69:443,Assigned server udp : 444,Assigned server id : 97aab137-51de-45c1-8185-aa28a0dfcc83 ,Server certificate thumbprint : C1E0B485A021FDE70E16EF5012720E9501707193,Beacon interval : 15.000000,Preferred server : ,Relay server : https://:443,Relay server udp: 444,Relay ESH name: ,Self destroy: 0,Server notified : 1,Server discovery done : 1,Agent updated : 0
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: **************ECAT CLIENT CERTIFICATE**************** : Version : 3 ,Subject : /CN=EcatClientExported,Issuer : /CN=EcatCA,Serial : 2D50F0ABF59AAFB04761C38D984B9E45,Thumbprint : 6175DB98B7AF7B0BD1DF3C85C3A960FAA75D4BA9,Not Before : Dec 22 07:36:07 2015 GMT,Not After : Dec 31 23:59:59 2039 GMT
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: **************ECAT CLIENT CERTIFICATE**************** : Version : 3 ,Subject : /CN=EcatClientExported,Issuer : /CN=EcatCA,Serial : 2D50F0ABF59AAFB04761C38D984B9E45,Thumbprint : 6175DB98B7AF7B0BD1DF3C85C3A960FAA75D4BA9,Not Before : Dec 22 07:36:07 2015 GMT,Not After : Dec 31 23:59:59 2039 GMT
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: Loaded certificate and key successfully from: /opt/rsa/nwe-agent/config/client-certificate.p12
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: Loaded certificate and key successfully from: /opt/rsa/nwe-agent/config/client-certificate.p12
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ============================= Agent version 4408 ============================
May 8 17:52:10 mrtg_uw_01 nwe-agent[4529]: ============================= Agent version 4408 ============================
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scan request recieved
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scan request recieved
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Time zone of the machine is : Greenwich Mean Time
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Time zone of the machine is : Greenwich Mean Time
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning loaded images list
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning loaded images list
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning running processes
May 8 17:57:50 mrtg_uw_01 nwe-agent[4529]: Scanning running processes
May 8 17:58:20 mrtg_uw_01 kernel: nwe-agent[13416]: segfault at 1a0 ip 0000000000434d51 sp 00007f2734188a70 error 4 in nwe-agent[400000+2f7000]
May 8 17:58:20 mrtg_uw_01 kernel: nwe-agent[13416]: segfault at 1a0 ip 0000000000434d51 sp 00007f2734188a70 error 4 in nwe-agent[400000+2f7000]

Tasks

How to collect core files on CentOS 7 machine when nwe-agent services crash.

Add this line to /etc/init.d/nwe-agent, in the start() section, right after the umask line:

DAEMON_COREFILE_LIMIT="unlimited"

Restarted the CentOS 7 system. Now core files were created in the root folder:

Sample:
[root@akcentos7 ~]# ls -la /core.*
-rw-------. 1 root root 35106816 May 13 16:53 /core.1143

Resolution

Review the collected core files to better understand why nwe-agent service crashes on the CentOS 7 machine.

Product Details

RSA Product Set: RSA NetWitness Endpoint
RSA Product/Service Type: CentOS 7 Linux Machine

Approval Reviewer Queue

KCS Approval queue