Flink startup failed complaining that the JKS file is missing at so and so location. I said “Oh.. “ and looked at the location in the worker node, and I see the file there.
I compare the user of the java process and the owner of the file, and validate there is no permission issue.
Flink jobs run in a cluster. A cluster means there are multiple nodes. And I looking only at the one node that was listed as the worker node. You need to look at all the nodes.
The JKS files had not been copied over to these other nodes. Once all nodes where these flink jobs could potentially get deployed and run had the JKS file, the startup was fine.
