[FLINK-37620][state/forst] ForSt Sync mode support remote storage

[fredia]

What is the purpose of the change

This PR makes ForSt Sync mode support remote storage and provide state.backend.forst.sync.enforce-local.

Brief change log

• Add state.backend.forst.sync.enforce-local to ForStOptions.
• Set remote directory for ForStSyncKeyedStateBackend when configured.
• Remove some useless code in ForStSyncKeyedStateBackend.java

Verifying this change

This change added tests and can be verified as follows:

• ForStStateBackendTest#testCreateKeyedStateBackend

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no )
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• bf5f208 UNKNOWN
• 3ce8c91 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[davidradl]

nit: is this true?
same job. -> same local job.

[Zakelly]

Overall LGTM. One suggestion: how about add descriptions explaining the state.backend.forst.sync.enforce-local under https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/ops/state/disaggregated_state/#forst-primary-storage-location

[masteryhx]

Thanks for the PR.
Just an extra comment: Should we also use ResourceGuard like FLINK-37597 when using sync mode to access remote storage ?

[fredia]

Should we also use ResourceGuard like FLINK-37597 when using sync mode to access remote storage ?

ResourceGuard is used to ensure that the snapshot threads are completed before ForStSyncKeyedStateBackend disposing, and ForStSyncKeyedStateBackend#dispose() has already achieved this.

IIUC, the remote storage accessing in ForStSyncKeyedStateBackend is done in the main task thread, maybe extra work is not necessary, correct me if I'm wrong :)