Logstash 2.4.1/5.0.x: Environments Variables parsed as Lists/Arrays for Logstash Configuration #6366
Comments
|
Just for the sake of hare-brained ideas... What happens if instead of you instead did (of course, you could just escape the brackets instead of using single quotes) Then you see (and I just verified this in a terminal): In theory, this may let you do: I know that this works in Curator. It may or may not work in Logstash. I figure it's worth a try, though. I think that the ENV variables are parsed first, into strings, which would potentially make the square brackets work when the config is parsed (after env var expansion). |
|
@untergeek That's a great idea; I tried it as per your suggestion, but still no dice. I also tried with escaping the square brackets but in the config it included the literal string. |
|
What if you omit the quotes around |
|
Maybe you did, and it made it a string anyway |
|
I am thinking of how to do this.
In terms of implementation, when we do validation is possibly where we
could support different data types from env.
For example, if a plugin has declares a setting to be a list of strings,
but provides some kind of env reference instead, we could define this to
take the Envirionment variable string (env is a map of string:string) and
split on comma.
If I try to imagine the code, I kind of want solve it by adding a new value
type in the config grammar that accepts this env ${blah} syntax and the
validation/setup would do the right thing to accept and validate that.
…On Tue, Dec 6, 2016 at 7:14 PM Aaron Mildenstein ***@***.***> wrote:
Maybe you did, and it made it a string anyway
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6366 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAIC6oRx89o-lCKwGw2BCUfs0jekaYYxks5rFiSWgaJpZM4LGCcX>
.
|
The configuration parser complains because it expects any value to be wrapped in double quotes. |
|
Bummer. That would do it.
…-- Aaron Mildenstein
On December 6, 2016 at 9:36:25 PM, Berg Lloyd-Haig ***@***.***) wrote:
@untergeek <https://github.com/untergeek>
What if you omit the quotes around "${ES_HOSTS}" and make it just hosts =>
${ES_HOSTS}?
The configuration parser complains because it expects any value to be
wrapped in double quotes.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6366 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA-R31X2PLbY5dEs15EbH2rwnzlsRdo4ks5rFjfJgaJpZM4LGCcX>
.
|
|
Hello, I am facing this problem, too. In my case, I'm trying to pass all kafka topics using an array as env variable. I tried all options defined above this post and Thanks! |
|
Any updates on this? This is really harshing my time....trying to pass the list as an environment variable to my ecs cluster of logstash servers. If no updates on the feature - anyone find a good work around (other than using a single elasticsaerch host ...) |
|
In issue #6665, 6 months ago, I proposed a way to do it, which is simple to implement and that I'm ready to implement (as I implemented first Logstash env var injection) :
What do you think about ? |
|
To whoever ends up implementing this, it would be fantastic if you could please also make it possible to pass in a nil value? (as opposed to an empty string)? It's a long story, but being able to pass in a nil would allow me to get around this issue logstash-plugins/logstash-input-beats#196. Thanks! |
|
Please do not send "+1" comments with no other content. This generates an ton of email for everyone. I have deleted all prior +1 comments. At the time of deletion, there were 4. If you feel compelled to "+1" something, please use Github issue reactions instead: |
|
Any updates on this issue yet? |
|
@fbaligand any updates on this? We really need this feature to keep our pipeline config clean. Also, unsure from the title whether this applies to logstash 6.x too? I'm hoping so 👍 |
|
Well, 1 year ago, I was ready to make a PR to implement the feature as explained in my previous comment. |
|
If we focus on just lists, we could probably make the proposed syntax possible ( For setting multiple hosts, for example, something like: and in Logstash: But what if the value would have a comma itself, such as passing a list to the date filter where one date pattern uses european-style decimal (with a comma), such as "YYYY-MM-DD HH:mm:ss,SSS" -- should users expect to need to escape the comma? What feedback can we provide? In the above date example, what's the expected interpretation: |
|
@jsvd What do you think? I struggle with finding a syntax that I think will satisfy what I think most users are wanting without creating a bunch of frustrating edge cases or alien syntax. |
|
IMHO, using environment variables is a little clunky because we can't use:
Could there be an alternative to using environment variables? Such as:
I'm guessing any such mechanism would require a change to the Logstash DSL. However, I feel this would give LS config authors the best flexibility in the long term. Cheers, |
|
I am running multiple instances of logstash in a docker swarm. Environment variables are an easy way for us to configure each different instance of logstash via our CI process, rather than requiring a persistent service. These variables are dynamically switched between at deploy time, so they can be hardcoded in separate files in VCS, and then switched between depending on branch. If we could put per environment configuration within logstash.yml, this may work... however I think adding a separate memcached/elastic instance for this will be overly complex. Perhaps something like 'logstash.dev.yml' and 'logstash.prod.yml' etc would do the job here, as we would be able to define per environment config overrides in any format, as long as logstash will look for/consume these files (preferrably by merging the base logstash.yml and logstash.env.yml). |
|
IE the only environment variable needed would be a simple string (LOGSTASH_ENV=dev, LOGSTASH_ENV=prod etc.), rather than complex formats. That way you have the benefit of versioning everything in a clean way, and avoiding formatting issues introduced by environment substitution. |
|
Hi @nick-george , First, in the future, probably other sources than environment variables will be available to interpolation using syntax |
|
Hi @jordansissel , To answer to your question about values that contain comma :
That's just my humble opinion :) |
|
@fbaligand if we scope this to solve providing a list of hosts from the environment instead of a generic list syntax, then I think it becomes easier to set user expectations. Do you want to scope this to just a list of hosts? or even a specific setting on a specific plugin? |
|
I think that the scope is every plugin option that is tied to environment. I also think it is important to manage empty array : if environment variable equals "" (empty string), then it is converted to empty array. Particularly useful for ca files list (in dev environment). And finally, as requested by @nick-george, I would find nice to manage special value "nil" to convert it to If implementation solves all these cases, I think we cover 99% of the needs. |
|
Hi! Any chance this would be implemented? It's been 3 years already and there is a pull request that has been there for almost a year. |
Since whitespace is illegal in URIs, we can safely use it as a delimiter when validating `list`-type `URI` params, enabling the expansion of an arbitrary list of URIs from a single Environment- or Keystore-variable. Resolves: elastic#8157 Resolves: elastic#6366
Since whitespace is illegal in URIs, we can safely use it as a delimiter when validating `list`-type `URI` params, enabling the expansion of an arbitrary list of URIs from a single Environment- or Keystore-variable. Resolves: elastic#8157 Resolves: elastic#6366
Since whitespace is illegal in URIs, we can safely use it as a delimiter when validating `list`-type `URI` params, enabling the expansion of an arbitrary list of URIs from a single Environment- or Keystore-variable. Resolves: elastic#8157 Resolves: elastic#6366
…2051) * plugin config: support space-deliminated URIs on list-type params Since whitespace is illegal in URIs, we can safely use it as a delimiter when validating `list`-type `URI` params, enabling the expansion of an arbitrary list of URIs from a single Environment- or Keystore-variable. Resolves: #8157 Resolves: #6366 * Doc: Create section for cross-plugin functionality and add space delimiters Co-authored-by: Karen Metts <[email protected]>
|
@yaauie |
My recent patch was limited to uri lists intentionally, because it is the only place where we can make a change to meaningfully address many use-cases that doesn't break in-the-wild configurations or require breaking changes in plugins, for three reasons:
But uris can be used to represent file paths, so this also gives us a path forward for plugins that wish to use one-to-many expansion of environment- and keystore-variables to populate file lists. Let's take a look at those plugins to see if they could meaningfully be moved over to use this functionality or in some similar way address those needs. |
|
Hi @yaauie , What do you think about? |
|
We were also just hit by this when trying to pass in multiple pipeline IDs to |
Uh oh!
There was an error while loading. Please reload this page.
I've been replacing the use of
logstash-filter-environmentplugin with the new feature: Using Environment Variables in the Configuration.I initially incorrectly diagnosed and reported the issue here ES hosts array: Support for parsing Environment Variables. @jordansissel advised I should file an issue in the correct project for this particular feature.
One of the things I am attempting to do is to supply an array for configuration item such as the hosts array in the
logstash-output-elasticsearchplugin. This is useful when the environment hostnames and IPs are ephemeral and the use of a service discovery via query ofetcdto determine IP and ports of online elasticsearch nodes during logstash startup is useful.I currently achieve this using sed to replace a placeholder string in the config on the docker container startup. It's not a major issue, but considering environment parsing functionality is built into Logstash now, it'd be great to leverage this instead of hacking the config prior to launch.
The text was updated successfully, but these errors were encountered: