Nifi pods going down

[kiranmudayan]

What steps will reproduce the bug?

1. Nifi Operator version 1.9.0
2. Nifi cluster version 1.17.0
3. Kubernetes version 1.29.0

What is the expected behavior?

Nifi cluster should be up and running.

What do you see instead?

Nifi pods continuosly getting restarted and the nifi-app.log shows the below error message :

2024-11-25 11:35:08,222 ERROR [main] org.apache.nifi.web.server.JettyServer Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchroni
zationException: java.lang.NullPointerException
org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.NullPointerException
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:381)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:197)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:42)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:835)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:477)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:890)
at org.apache.nifi.NiFi.(NiFi.java:172)
at org.apache.nifi.NiFi.(NiFi.java:83)
at org.apache.nifi.NiFi.main(NiFi.java:332)
Caused by: java.lang.NullPointerException: null
at java.net.URI$Parser.parse(URI.java:3041)
at java.net.URI.(URI.java:588)
at org.apache.http.client.utils.URIBuilder.(URIBuilder.java:82)
at org.apache.nifi.registry.flow.StandardFlowRegistryClient.addFlowRegistry(StandardFlowRegistryClient.java:66)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.addRegistry(VersionedFlowSynchronizer.java:478)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.inheritRegistries(VersionedFlowSynchronizer.java:470)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:323)
... 10 common frames omitted
2024-11-25 11:35:08,223 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.

Possible solution

No response

NiFiKop version

v1.9.0-release

Golang version

we are using the official 1.9.0 image of nifi operator

Kubernetes version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full ver
sion.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDat
e:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.5", GitCommit:"59755ff595fa4526236b0cc03aa2242d941a5171", GitTreeState:"clean", BuildDat
e:"2024-05-22T15:00:52Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.29) exceeds the supported minor version skew of +/-1

NiFi version

1.17.0

Additional context

Any help is appreciated.

[juldrixx]

Hello, it happens when the flow.xml.gz or flow.json.gz is desyncronized across the nodes of the cluster. You need to remove the files on the crashing node.

[kiranmudayan]

@juldrixx I tried renaming both the 'flow.xml.gz' and 'flow.json.gz' in nifi pods. But Getting the same error, also the files are not getting recreated on restart.

[juldrixx]

Do you have multiple nodes? Is there only one node that is crashing?
Do you have any data on it? If no, can you try to delete the PVCs and kill the pod associated to this node?
And can you try to upgrade the Operator to 1.11.4?

[kiranmudayan]

@juldrixx We have 3 node cluster. All the 3 nifi pods are crashing. We have data on it so deleting the PVC will cause more trouble. We have another environment which has nifi operator version 1.9.0 and nifi cluster 1.24.0.

We will try upgrading the nifi cluster to version 1.24.0 in the environment which has this issue.