gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000

incrypto32 · 2025-05-12T12:12:05Z

Supporting multiple subgraphs in dev mode is a bit tricky since FileLinkResolver needs a base dir, this works well for a single subgraph since we can scope the resolver to the build directory of that subgraph, but when there are multiple subgraphs, we need to set the base_dir of the FileLinkResolver dynamically. This PR implements the mechanisms for that.

This is done by using a link_resolver_override thats passed upto the subgraph runner which switches to using it when available.

For supporting subgraph datasource, aliases are used users can declare their subgraph datasources in the manifests normally like below example.

dataSources:
  - kind: subgraph
    name: Factory
    network: base
    source:
      address: 'QmSource'
      startBlock: 1759510

When they run gnd they have to pass in the flag

--source "QmSource:<PATH_TO_MANIFEST_OF_SOURCE_SUBGRAPH"

lutter

This is nice; I feel that we need a more general concept around finding related subgraphs in the code, maybe some kind of LinkMapper that is pretty much a noop for IPFS, and knows where different subgraphs are in the filesystem for dev mode. If we had that, I think we could set these up in the respective main methods, and wouldn't have to guess as much in the link resolver.

lutter · 2025-05-20T22:40:52Z

graph/src/components/link_resolver/file.rs

+        // The manifest path is the path of the subgraph manifest file in the build directory
+        // We use the parent directory as the base directory for the new resolver
+        let base_dir = canonical_manifest_path
+            .parent()


I am a bit confused now, isn't this basically base_dir/deployment_str/.., i.e. base_dir ?

deployment_str can be something like "../subgraph2/subgraph.yaml" in that case the new base_dir is parent of "base_dir/../subgraph2/subgraph.yaml" which is "../subgraph2"

When deployment_str is an absolute path its simply the directory in which the subgraph.yaml is

Isn't deployment_str just the deployment hash because of the line let deployment_str = deployment.to_string(); ?

graph/src/components/link_resolver/file.rs

lutter · 2025-05-20T22:45:15Z

graph/src/components/link_resolver/mod.rs

+    /// For other resolvers, this method will simply return the current resolver
+    /// This is required because paths mentioned in the subgraph manifest are relative paths
+    /// and we need a new resolver with the right base directory for the specific subgraph
+    fn for_deployment(&self, deployment: DeploymentHash) -> Result<Box<dyn LinkResolver>, Error>;


Reading the comment, it might make sense to rather have a method for_manifest(manifest_path: PathBuf). By using the deployment here, we are still encoding a convention about where deployments live relative to each other. It would be nicer if that convention was only used to find manifests and everything else follows from there. Worst case, we could even store the path from which a manifest was read in the manifest itself to make it easier to get the path whereever it is needed. In a way, that would nicely express what changes with this PR: we used to know implicitly where manifests live because they were all on IPFS under their hash. Now, a manifest can live there, but also in the filesystem, and I think we should just express that explicitly

I changed the method signature to fn for_manifest(&self, manifest: DeploymentHash i had to keep the type to be DeploymentHash since it can be an alias name as well its not strictly a file path.

fn for_manifest(&self, manifest_path: &str)

node/src/dev/watcher.rs

core/src/subgraph/registrar.rs

lutter · 2025-05-22T23:45:26Z

graph/src/components/link_resolver/file.rs

+        // The manifest path is the path of the subgraph manifest file in the build directory
+        // We use the parent directory as the base directory for the new resolver
+        let base_dir = canonical_manifest_path
+            .parent()


Isn't deployment_str just the deployment hash because of the line let deployment_str = deployment.to_string(); ?

lutter · 2025-05-22T23:48:07Z

graph/src/components/link_resolver/file.rs

 }

 impl FileLinkResolver {
    /// Create a new FileLinkResolver
    ///
    /// All paths are treated as absolute paths.
-    pub fn new() -> Self {
+    pub fn new(base_dir: Option<PathBuf>, aliases: HashMap<String, PathBuf>) -> Self {


It would be clearer if aliases was a HashMap<DeploymentHash, PathBuf> which is really what it expresses: where in the filesystem one would find the files for a given deployment.

I find it a bit strange to do that, because aliases represents is a mapping from the String aliases that the user provided as cli arguments, yes it does get converted into DeploymentHash but with the recent change to

fn for_manifest(&self, manifest_path: &str) from fn for_deployment(&self, deployment: DeploymentHash)

Also in the cat method we would need to convert the Link again back to DeploymentHash which is a bit wierd, since Link was created by calling to_ipfs_link onDeploymentHash. In cat we use this to resolve aliases properly

graph/src/components/link_resolver/file.rs

…kResolver with the right base dir for a subgraph

ignore_graft_base This reverts commit b5bbf93.

incrypto32 changed the base branch from master to krishna/graph-dev May 12, 2025 12:12

incrypto32 changed the title ~~gnd: Support subgraph composition in dev mode~~ gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode May 12, 2025

incrypto32 force-pushed the krishna/graph-dev-composition-2 branch from 0826b2a to b5bbf93 Compare May 12, 2025 14:54

incrypto32 requested a review from lutter May 13, 2025 05:28

incrypto32 mentioned this pull request May 16, 2025

Graph node dev mode #5982

Merged

incrypto32 self-assigned this May 19, 2025

lutter approved these changes May 20, 2025

View reviewed changes

lutter approved these changes May 22, 2025

View reviewed changes

incrypto32 force-pushed the krishna/graph-dev branch from e5a9e23 to 00d6987 Compare June 10, 2025 11:37

incrypto32 force-pushed the krishna/graph-dev-composition-2 branch from a8a21a3 to bc4b351 Compare June 10, 2025 11:54

incrypto32 added 19 commits June 10, 2025 17:38

graph: Add clone_for_deployment to FileLinkResolver to create FileLin…

ec4b36d

…kResolver with the right base dir for a subgraph

graph: Add for_deployment to LinkResolverTrait

4af0e6d

core, graph: use for_deployment to get properly scoped resolver

b1e4a0c

graph: Implement aliases for file link resolver

b7dcdca

node: Make gnd work with multiple subgraphs

4263ebc

node: Support subgraph datasource in gnd

1616bf8

node: correct the default value for manfiest

3915686

core, node, graph: Ignore graft base in dev mode

32b7163

node: Allow providing a postgres url for gnd

317b381

node: Do not use pgtemp in windows

fdedb23

store: enable vendored feature for openssl crate

9a90dff

chain/ethereum: Return error when ipc is used in non unix platform

3012224

node: Refactor launcher

72e0da4

node/dev : Better error message when database directory doesn't exist

8e51bf2

node: refactor watcher

747f7a9

core, node, graph: Manipulate raw manifest instead of passing

2eccf06

ignore_graft_base This reverts commit b5bbf93.

node: Correct comments on redeploy_all_subgraphs

d99396f

node/gnd: Deploy all subgraphs first before wathcing files

80e7139

core, graph : Refactor LinkResolver trait

b349d60

incrypto32 force-pushed the krishna/graph-dev-composition-2 branch from bc4b351 to b349d60 Compare June 10, 2025 12:08

incrypto32 changed the base branch from krishna/graph-dev to krishna/gnd-base June 10, 2025 12:09

incrypto32 merged commit 655764d into krishna/gnd-base Jun 10, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000

gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000

Uh oh!

incrypto32 commented May 12, 2025 •

edited

Loading

Uh oh!

lutter left a comment

Uh oh!

lutter May 20, 2025

Uh oh!

incrypto32 May 21, 2025 •

edited

Loading

Uh oh!

lutter May 22, 2025

Uh oh!

Uh oh!

lutter May 20, 2025

Uh oh!

incrypto32 May 27, 2025

Uh oh!

incrypto32 May 27, 2025

Uh oh!

Uh oh!

Uh oh!

lutter May 22, 2025

Uh oh!

lutter May 22, 2025

Uh oh!

incrypto32 May 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000

gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000

Uh oh!

Conversation

incrypto32 commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lutter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

incrypto32 May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

incrypto32 commented May 12, 2025 •

edited

Loading

incrypto32 May 21, 2025 •

edited

Loading