Skip to content

www: add initial cloudflare log-collect tooling #987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

rvagg
Copy link
Member

@rvagg rvagg commented Nov 8, 2017

Includes the two commits in #976 which should land shortly.

As I said there:

Adds the log-collect stuff that's on there collecting logs for us from cloudflare and doing some basic filtering. There's also some stuff in there to convert the json format to nginx format but it's not being used for anything currently. The log-collect work is only partially complete since we're collecting but not using it for metrics generation yet. We still need to (a) confirm that we're successfully collecting everything with this procedure (logshare isn't the most robust tool unfortunately, or maybe it's their API, hence the sanity-check script in here), and (b) either do the json->nginx conversion for metrics, or make metrics scripts able to read the json files as they are. Either way, this PR reflects current state so is good to merge IMO.

I haven't tested this .. a bit of a stab in the dark, I'm not willing to run this against the live server.

I'm not sure that I'm going to be the person to finish this work but there's only a small number of people we can give access to the webserver. So what I'm going to have to do is spin up a separate server that has all of this log collecting stuff, metrics, old logs and some sensitive stuff (cloudflare key for one) and give someone, or someones access to it. I believe @refack volunteered to have a go at it when I raised this previously somewhere.

@rvagg
Copy link
Member Author

rvagg commented Nov 8, 2017

forgot about #679 which is essentially the same. I've closed that but there's some notes there that are relevant to this

@rvagg
Copy link
Member Author

rvagg commented Nov 28, 2019

see #2025

@rvagg rvagg closed this Nov 28, 2019
@rvagg rvagg deleted the rvagg/cloudflare-log-collect branch November 28, 2019 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant