Cloudflare log processing for metrics

[rvagg]

We're in a bit of limbo here, mainly because it's been fully on me and we know how that's working out lately!

Current status: we're load balancing with CF now so there's a fallback server in place so if the main one goes down it'll switch to serving the backup. But we still need to fully detach the /downloads/ content from CF so that it can serve it from cache. The blocker is that we need to be able to produce download metrics for http://nodejs.org/metrics/ so we need to switch from processing nginx logs to CF logs to produce those.

I have a script in place that fetches the CF logs every hour (it's not a deterministic process so it has to account for failures and retries so we need to verify that it's actually working now that it's been running for a while), I also have a script that'll convert their line-delimited JSON to the same format (roughly) as our nginx logs, potentially as a final solution or a step toward a final solution. I just haven't don all the bits that wire everything up together.

If someone wants to help with this, they'll either need access to nodejs.org, and there are only a few of those, or I'll have to provide example files and details of existing tools and file locations that can be used to replicate the process. I'm happy to work with someone on this but it's going to be tedious work on my end to replicate it off the server.

[refack]

I'll take this on (unless someone with access wants to).
In this case I'll need the samples.

[refack]

I'm happy to work with someone on this but it's going to be tedious work on my end to replicate it off the server.

Send me the code and a sample of input and output. I'll manage.

[refack]

Is this the API you're using? https://support.cloudflare.com/hc/en-us/articles/216672448-Enterprise-Log-Share-REST-API

[jbergstroem]

@refack yes.

[Trott]

Closing due to long period of inactivity. Feel free to re-open if this is a thing. I'm just trying to close stuff that has been ignored for sufficiently long that it seems likely it's not something we're going to get to.

[rvagg]

-> #987