Controller fails to startup if DataDir cannot be created

[rseetham]

The controller has a property controller.data.dir. When the controller is started, we initialize a class called ControllerFilePathProvider. It reads this properpy and tried to load the PinotFS with that scheme. If it cannot be created, the controller start up fails. The dataDir is only used for backingup segments. Pinot can work without backing up segments. The server does not have this check at startup. The segment uploads fail.

We should remove this check. We use hdfs for the dataDir. There was an issue with the hdfs namenode so all segment uploads failed but Pinot was running and serving queries well. At that time, server could be restarted but controllers could not be restarted which is an issue.

[Jackie-Jiang]

I believe data dir is pointing to the deep store. Without it, server won't be able to download segment. So the check should be mandatory for the cluster to function.

[rseetham]

We had a cluster running. Our data dir points to hadoop. There was an issue with the hadoop name node calls failed for 4-5 hours until the issue was fixed. In that time, servers could be restarted. Data backups failed but the server was able to load data segments from the peer when needed. But the controllers could not be restarted. The restart would fail. We want to prevent this situation. When a cluster is running, we want to keep everything in a heathy state and serving queries even if the backup is not accessible.

[Jackie-Jiang]

@chenboat Can you share the experience of operating Pinot on HDFS?