Skip to content

Avoid cross-dependencies between components #1940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ghost opened this issue Feb 26, 2020 · 1 comment
Closed

Avoid cross-dependencies between components #1940

ghost opened this issue Feb 26, 2020 · 1 comment
Assignees
Labels
meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. type: tech debt A code change that does not add user value.

Comments

@ghost
Copy link

ghost commented Feb 26, 2020

Currently some components of Vector import and use code from other components. The list of such dependencies is the following:

  • sources-kubernetes depends on sources-file, transforms-json_parser, transforms-regex_parser;
  • sources-syslog depends on sources-socket;
  • sources-vector depends on sources-socket;
  • sinks-humio_logs depends on sinks-splunk_hec;
  • sinks-new_relic_logs dependes on sinks-http;
  • sinks-sematext_logs depends on sinks-elasticsearch.

I think in long term this might make it harder to refactor individual components, as change is one component might break some other (seemingly) unrelated component. So ideally we need to identify the common code and factor it out to lib crates or, if the code is too specific, at least as util modules.

The above relates only to implementation of components, but not to their tests. Using other components in tests I think is less an issue, although some tests for components might be changed to not depend on other components. In particular, I think most of the tests depending on json_parser can be modified to avoid such a dependency.

See also #1924.

@ghost ghost added type: tech debt A code change that does not add user value. meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. labels Feb 26, 2020
@ghost ghost added this to the Tech-debt payment #1: Code organization milestone Feb 26, 2020
@binarylogic binarylogic removed this from the Tech-Debt Payment #1: Code Organization milestone Jul 19, 2020
@ktff
Copy link
Contributor

ktff commented Jul 24, 2020

These have already been resolved:

  • syslog source
  • vector source
  • honeycomb sink
  • papertrail sink

kubernetes_logs source uses transforms:

  • merge whole so there is little to be gained in separating common code.
  • regex partially so there is common code that could be extracted, although it would be quite specific.
    • To extract common regex or not? (EDIT: No, I don't see any clean way of separating shared code from it )
  • json_parser partially so there is common code to be extracted, and it's something that can be shared, even specialized since tests use it in a same/similar way as kuberentes_logs source.

Others:

  • humio_logs sink uses whole splunk_hec sink
  • new_relic_logs sink uses whole http sink
  • sematext_logs sink uses whole elasticsearch sink

They are using whole of the other sink, as they are basically wrappers with some extra/default configuration. Their common code would be the whole sink so I think we wouldn't gain much.

cc. @binarylogic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. type: tech debt A code change that does not add user value.
Projects
None yet
Development

No branches or pull requests

2 participants