.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
RSA NetWitness 11.x Upgrade failed after Chef completed successfully
Issue
Upgrade failed after Chef completed successfully.Cause
Upgrade fails even though the chef run triggered by orchestration-server and node-infra-server completed successfully.The following error is displayed on the Admin node at /var/log/netwitness/orchestration-server/orchestration-server.log:
2021-09-20 23:41:23,305 [ 10.160.101.10] ERROR TaskManagement|Failed to process the job details of task '61491378879b9c4f40101e34'
org.springframework.web.client.RestClientException: Error while extracting response for type [com.rsa.asoc.orchestration.salt.client.domain.LocalResponse<java.util.Map<java.lang.String, java.lang.String>>] and content type [application/json]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)'); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:120)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:996)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:979)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:739)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:649)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:214)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:160)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.lambda$run$0(DefaultSaltClient.java:224)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:223)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.getServiceUuid(ServiceInstallTaskHandler.java:555)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.lambda$persistInstalledServices$4(ServiceInstallTaskHandler.java:310)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.persistInstalledServices(ServiceInstallTaskHandler.java:308)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.persistServiceHostInfo(ServiceInstallTaskHandler.java:249)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.processTaskResult(ServiceInstallTaskHandler.java:217)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.rsa.asoc.orchestration.task.TaskStateMachine.transitionToState(TaskStateMachine.java:175)
at com.rsa.asoc.orchestration.task.TaskStateMachine.execute(TaskStateMachine.java:88)
at com.rsa.asoc.orchestration.task.HostAsyncTaskQueue.processTask(HostAsyncTaskQueue.java:124)
at com.rsa.asoc.orchestration.task.HostAsyncTaskQueue.monitorAndExecuteTasks(HostAsyncTaskQueue.java:162)
at com.rsa.asoc.orchestration.task.AsyncTaskService.lambda$submitMonitorTasksAsRequired$6(AsyncTaskService.java:192)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)'); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:285)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:243)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:105)
... 40 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1455)
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1081)
at com.fasterxml.jackson.databind.deser.ValueInstantiator._createFromStringFallbacks(ValueInstantiator.java:371)
at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromString(ValueInstantiator.java:258)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:357)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:540)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:377)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:290)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:371)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4526)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3521)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:274)
... 42 common frames omitted
org.springframework.web.client.RestClientException: Error while extracting response for type [com.rsa.asoc.orchestration.salt.client.domain.LocalResponse<java.util.Map<java.lang.String, java.lang.String>>] and content type [application/json]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)'); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:120)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:996)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:979)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:739)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:649)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:214)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:160)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.lambda$run$0(DefaultSaltClient.java:224)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:223)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.getServiceUuid(ServiceInstallTaskHandler.java:555)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.lambda$persistInstalledServices$4(ServiceInstallTaskHandler.java:310)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.persistInstalledServices(ServiceInstallTaskHandler.java:308)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.persistServiceHostInfo(ServiceInstallTaskHandler.java:249)
at com.rsa.asoc.orchestration.salt.handler.ServiceInstallTaskHandler.processTaskResult(ServiceInstallTaskHandler.java:217)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.rsa.asoc.orchestration.task.TaskStateMachine.transitionToState(TaskStateMachine.java:175)
at com.rsa.asoc.orchestration.task.TaskStateMachine.execute(TaskStateMachine.java:88)
at com.rsa.asoc.orchestration.task.HostAsyncTaskQueue.processTask(HostAsyncTaskQueue.java:124)
at com.rsa.asoc.orchestration.task.HostAsyncTaskQueue.monitorAndExecuteTasks(HostAsyncTaskQueue.java:162)
at com.rsa.asoc.orchestration.task.AsyncTaskService.lambda$submitMonitorTasksAsRequired$6(AsyncTaskService.java:192)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)'); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:285)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:243)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:105)
... 40 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value (''nodeinfo' __virtual__ returned False: inconsistent use of tabs and spaces in indentation (nodeinfo.py, line 18)')
at [Source: (PushbackInputStream); line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->java.util.ArrayList[0]->java.util.LinkedHashMap["1eaa4d61-0a50-492f-9789-2e7df39f7f2d"])
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1455)
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1081)
at com.fasterxml.jackson.databind.deser.ValueInstantiator._createFromStringFallbacks(ValueInstantiator.java:371)
at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromString(ValueInstantiator.java:258)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:357)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:540)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:377)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:290)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:371)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4526)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3521)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:274)
... 42 common frames omitted
Resolution
1. SSH to the host on which the upgrade has failed.2. Clear the salt-minion cache using the below command.
rm -rf /var/cache/salt/minion
3. Restart the salt-minion service on the host.
systemctl restart salt-minion
4. SSH to the Admin node and restart the salt-master and salt-api services.
systemctl restart salt-master salt-api
5. Retry the upgrade.
Product Details
RSA Product Set: NetWitness PlatformRSA Product/Service Type: RSA NetWitness Platform
RSA Version/Condition: 11.6.x
Platform: CentOS
O/S Version: 7
Approval Reviewer Queue
Technical approval queue