Skip to content
  • There are no suggestions because the search field is empty.

RSA ESA Correlation Server Upgrade Failed due to invalid featureCompatibilityVersion

Issue

The ESA Correlation Server Upgrade Failed due to invalid featureCompatibilityVersion and cant start the MongoDB service.
 
2021-07-23T08:00:51.796+0000 I SHARDING [initandlisten] Marking collection local.system.replset as collection version: <unsharded>
2021-07-23T08:00:51.799+0000 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document
  (ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion
  document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6" }. See https://docs.mongodb.com/manual/release-notes/4.2-compatibility/#feature-compatibility.).
  If the current featureCompatibilityVersion is below 4.0, see the documentation on upgrading at https://docs.mongodb.com/manual/release-notes/4.2/#upgrade-procedures.
2021-07-23T08:00:51.799+0000 I REPL [initandlisten] Stepping down the ReplicationCoordinator for shutdown, waitTime: 10000ms
2021-07-23T08:00:51.799+0000 I SHARDING [initandlisten] Shutting down the WaitForMajorityService
2021-07-23T08:00:51.799+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2021-07-23T08:00:51.799+0000 I NETWORK [initandlisten] Shutting down the global connection pool
2021-07-23T08:00:51.799+0000 I STORAGE [initandlisten] Shutting down the FlowControlTicketholder


Cause

The MongoDB startup error seen on the impacted ESA host occurs when the featureCompatibilityVersion flag is still set to 3.6 instead of 4.0 while upgrading the MongoDB package from v4.0.19 to v4.2.12 version.
2021-07-23T08:00:51.799+0000 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM:
Found an invalid featureCompatibilityVersion document ( ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion
document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6" }. See https://docs.mongodb.com/manual/release-notes/4.2-compatibility/#feature-compatibility.).
If the current featureCompatibilityVersion is below 4.0, see the documentation on upgrading at https://docs.mongodb.com/manual/release-notes/4.2/#upgrade-procedures.


Resolution

Please execute the below steps on the ESA Correlation Server
  1. Downgrade the MongoDB RPMs from existing v4.2.12 (11.6.x compatible) to v4.0.19 (11.5.x compatible) version.
    (a) Copy the v4.0.19 MongoDB RPMs from NW 11.5.0.0 repo to /tmp/mongo-rpms folder

    [root@esaprimary ~] mkdir -p /tmp/mongo-rpms
    [root@esaprimary ~] cd /tmp/mongo-rpms
    [root@esaprimary mongo-rpms]# ls -l
    -rw-r--r--. 1 root root 6056 Jul 27 11:48 mongodb-org-4.0.19-1.el7.x86_64.rpm
    -rw-r--r--. 1 root root 12939348 Jul 27 11:48 mongodb-org-mongos-4.0.19-1.el7.x86_64.rpm
    -rw-r--r--. 1 root root 22518116 Jul 27 11:48 mongodb-org-server-4.0.19-1.el7.x86_64.rpm
    -rw-r--r--. 1 root root 14179171 Jul 27 11:48 mongodb-org-shell-4.0.19-1.el7.x86_64.rpm
    -rw-r--r--. 1 root root 48460884 Jul 27 11:48 mongodb-org-tools-4.0.19-1.el7.x86_64.rpm

    (b) yum downgrade mongodb-org-*.rpm
  2. Update the /etc/mongod.conf to match the options/parameters supported by v4.0.19 MongoDB RPMs.
         Change below parameters (compatible with v4.2.12 MongoDB RPMs):
    tls:
    mode: preferTLS
    certificateKeyFile: /etc/pki/nw/mongo/mongod-combined.pem

          to below parameters (compatible with v4.0.19 MongoDB RPMs):
    ssl:
    mode: preferSSL
    PEMKeyFile: /etc/pki/nw/mongo/mongod-combined.pem
  3. Restart MongoDB service & check if it starts successfully now.
    systemctl restart mongod
  4. Login to MongoDB & check if the featureCompatibilityVersion flag is set to 3.6 using below command:

       db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )  

    [root@esaprimary ~]# mongo admin -u deploy_admin -p <deploy-admin-password>
    MongoDB shell version v4.0.19
    connecting to: mongodb://127.0.0.1:27017/admin?gssapiServiceName=mongodb
    Implicit session: session { "id" : UUID("6f0efa60-5df9-4e69-a1ca-b147dcb86b11") }
    MongoDB server version: 4.0.19

    > db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
    { "featureCompatibilityVersion" : { "version" : "3.6" }, "ok" : 1 }
  5. Modify /etc/mongod.conf to change 'authorization'
         Change below parameters
    security:
    authorization: enabled
         to
    security:
    authorization: disabled
  6. Restart MongoDB service
    systemctl restart mongod
  7. Login to MongoDB & change the featureCompatibilityVersion flag to 4.0 from 3.6 using below steps:
         db.adminCommand({setFeatureCompatibilityVersion: "4.0"})    
    [root@esaprimary ~]# mongo admin
    MongoDB shell version v4.0.19
    connecting to: mongodb://127.0.0.1:27017/admin?gssapiServiceName=mongodb
    Implicit session: session { "id" : UUID("43412d4b-952c-43c2-8525-173e3ce0f603") }
    MongoDB server version: 4.0.19

    > db.adminCommand({setFeatureCompatibilityVersion: "4.0"})
    { "ok" : 1 }

    > db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
    { "featureCompatibilityVersion" : { "version" : "4.0" }, "ok" : 1 }
  8. Modify /etc/mongod.conf to change 'authorization'
         Change below parameters
    security:
    authorization: disabled
         to
    security:
    authorization: enabled
  9. Restart MongoDB service
    systemctl restart mongod
  10. Remove the existing node.json file out of config-management folder.      
    mv /etc/netwitness/config-management/node.json /tmp/node.json.bkp
  11. Initiate ESA upgrade to target 11.6.0.0 version (Run on AdminServer).     
    upgrade-cli-client --upgrade --host-id <esa-node-uuid> --version 11.6.0.0 --verbose

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Event Stream Analysis /ESA Correlation Server
RSA Version/Condition: 11.6.0.0,11.6.1.0

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue