feat: use new kibana api for pushing monitors

[vigneshshanmugam]

• Use the new Kibana API - [Uptime] New RESTful monitors / projects API kibana#142418
• This PR adds the support for the new Bulk API endpoints while maintaining the support for the previous API endpoints that is part of 8.5.x stack.
• Support for older and new versions is determined by hitting at the Kibana API endpoint and pushing the monitors.
• Monitors are created and deleted in chunks of 100, Might need further tuning later down the line as pushing 100's of monitor could be error prone due to the size of the bundled journey and request payload might be affected when there is a proxy layer in front of Kibana.
• The CLI progress has changed to keep it similar for both the legacy and the bulk API endpoints.

Testing

• Build the project - npm run build
• Create a brand new synthetics project using npm init <dir> command.
• Link the local synthetics package using npm link.
• Push a synthetics project - SYNTHETICS_API_KEY=<> npm run push
• Try on both older versions and also on latest snapshots 8.6.0.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-05T16:00:42.156+0000

• Duration: 15 min 35 sec

Test stats 🧪

Test	Results
Failed	0
Passed	206
Skipped	2
Total	208

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[dominiqueclarke]

I just tested this in cloud, and I noticed that none of the monitors pushed via this branch ran on the service, and there was no feedback as to why.

This did not happen on main for the same monitor configuration.

I know this is still in progress, but requesting changes just to make sure we re-test that the monitors are actually running on the service and private locations.

I tested this with a sample payload from the existing agent passed to the new api, and verified that the monitor executed on the service and returned back up.

[dominiqueclarke]

When you first use the v2 api for the first time, monitors are marked as added instead of updated. I think we should adjust the logic so that if the monitors are returned from the GET request, they are marked as updated, even if they don't yet have a hash. This is especially important, because the non existance of a hash is not a reliable way of determining whether a monitor exists or not. When monitors are edited from the UI (in the case of enabling/disable for example), the hash is reset to '' to ensure that when configuration is repushed it marks it as needing an update.

You can easily recreate this issue btw, even if using the new api with a fresh cluster, by pushing a batch of monitors, disabling one monitor from the UI, and repushing. The monitor will be marked as added, instead of updated.

[dominiqueclarke]

When changing monitors to enabled: false globally in the synthetics.config.ts file, monitors are marked as unchanged as the update is not pushed. Also, when setting tags, params or playrwight_options, screenshot globally, monitors are marked as unchanged.

When changing private_locations, schedule, locations, or throttling, the monitors are marked as changed.

    /**
     * Configure global monitor settings
     */
    monitor: {
      schedule: 60,
      locations: ['us_central'],
      privateLocations: ["BEEP"],
      enabled: false,
      tags: ["a", "tag"],
     params: { "a": "param" },
    },

Also, something more confusing is happening with enabled more generally. I have this monitor alongside the other test monitors

import { journey, step, monitor } from '@elastic/synthetics';

const createJourney = (name) => {  // TS types
  monitor.use({
    throttling: {
      download: 10,
      upload: 5,
      latency: 10,
    },
   enabled: false
  })
  journey(name, ({ page, params }) => {
    step('launch application', async () => {
      await page.goto(params.url);
    });
  });
};

['a', 'b'].forEach(a => createJourney(a));

When setting enabled false on this monitor, not changing any other monitors, something very weird happens. The resulting schemas sent to the api are

[
   {
      "throttling":{
         "download":10,
         "upload":5,
         "latency":10
      },
      "schedule":60,
      "locations":[
         "us_central"
      ],
      "privateLocations":[
         "BEEP"
      ],
      "params":{
         "url":"https://elastic.github.io/synthetics-demo/"
      },
      "playwrightOptions":{
         "ignoreHTTPSErrors":true,
         "headless":true
      },
      "name":"adding and removing multiple tasks been 0",
      "id":"adding and removing multiple tasks been 0",
      "type":"browser",
      "tags":[
         
      ],
      "hash":"vBvJUZb4Crs9dYqBLNHA4EnmXMH+yN0Fg0z9yO7/ESI=",
      "content":"UEsDBBQACAAIAAAAIQAAAAAAAAAAAAAAAAAkAAAAam91cm5leXMvYWR2YW5jZWQtZXhhbXBsZS5qb3VybmV5LnRzlVLdbpswFL7PUxxZk2ZUCku2q2WZ9v8C666qSnXhJHgxtmefZKER7z4DJqJZW2032AfO93POR57DT7NzGpu3+Q+PzuelqaWWv3ZYKOG2mJe4z32jqUKShc8JPS3yiPG5KPdCF1he4kHUVmEWv2TkZ7K2xhEcR4UUPKGFFtbO1MA+oBI+cE7Y2fIEmgEoI8qP1n4PoDSUoiyvhN+OZVFhsf1m3PRdGcw504yvZqNU9pfPywqVDeMGxbVxwBUSSFjBq2U43sE8HBcXSW8juue3wYAkaTQIXUJhOpq+NGvwUm8UAgVhuEPUsIAXR9nepsCPYMUG0/B0ovbQJrB63/OGEYz2BN1Gr/BAQZ19Mfolgd0ReKGCIw1NUAdssHPaYboVcmbl/b1gKYiwuwL4hHNkrVCU6AKn+C0k9R4yZQpBxnFWzVmyjO14sFgQH9oGVEbBzmejCTXxJMnIfMJRM+LaeE4y4tM5s51TsWWSW2wZJ44N50nGLnbnzBb1qHiW7WNUg6kHeelNn5bD2uy7ot4pkjZG5Yes/i+pHreCa9bdYB5CGG6L0+01u/nn5Zwos/AbfhVFxTk9SPOR7Z1F8MxiOuLr+c2TQQx+37DT9trZH1BLBwgG13YozwEAABEEAABQSwMEFAAIAAgAAAAhAAAAAAAAAAAAAAAAACQAAABqb3VybmV5cy9hZHZhbmNlZC1leGFtcGxlLWhlbHBlcnMudHOVVMFu2zAMvfcrCKFAFSDNsGuLBFuL7bRbdxsGRFWYWqgiCRK9Jgj876NkJ7ETt0AviU2Kj4/v0TKb4CPBHhJhmAJuA2qCBtbRb0B8Q6sSGf0l7RxVyE9J3F9p7xKB9Wr1PYQnroM5yKBecAp1tBOYL2B/BQVRCqtqpytQIYgpKMbRII9HANSbMgS5ePbiycsMcM+Zhn+bQyu1Wv1W6XXYijgy7LXkcyUM1/v81yzHOraQxoWaGKzX33qtyEcpSm7m8O2W/MqLwufAtM3RLqAsBC5zIWJKUvxwhFFczpIh0xNaltlHJiBqO8uxW2uyqKa8HEVeG1dGz0zbSUaGj0h1dGOjLFmIfr8GFgsg3NJcdBKJ5UDolDBSbviL2Tz62lEn+qC7zokz7XWF+hWoUtRaUMapVOKVUprsjj0pZQ0wlw06Sh+4kxHSuDuDeTr5262V7elSOyu95GTCcj6gbAlfeFE4//Sxt1zvq9wfc83WlSlvOhlv2Pwy8uhQnUEHM/voF5xWmCj63ScodRVnhD5Q16pntEd132N1PG2OR0vh6TvZBkXVXDnNBHy8u7Pmz9e/YlDccXuoibw74ZgTyHNJDau0NexLzaIh1wwwZiUnB98dw1X+H8ZhtAfSE5lXpdx3HOndKtP8Or77OXW+Jzl25lMOHZTMz73bMbf9D1BLBwj8Gxp6/QEAAGcFAABQSwECLQMUAAgACAAAACEABtd2KM8BAAARBAAAJAAAAAAAAAAAACAApIEAAAAAam91cm5leXMvYWR2YW5jZWQtZXhhbXBsZS5qb3VybmV5LnRzUEsBAi0DFAAIAAgAAAAhAPwbGnr9AQAAZwUAACQAAAAAAAAAAAAgAKSBIQIAAGpvdXJuZXlzL2FkdmFuY2VkLWV4YW1wbGUtaGVscGVycy50c1BLBQYAAAAAAgACAKQAAABwBAAAAAA=",
      "filter":{
         "match":"adding and removing multiple tasks been 0"
      }
   },
   {
      "throttling":{
         "download":10,
         "upload":5,
         "latency":10
      },
      "schedule":60,
      "locations":[
         "us_central"
      ],
      "privateLocations":[
         "BEEP"
      ],
      "params":{
         "url":"https://elastic.github.io/synthetics-demo/"
      },
      "playwrightOptions":{
         "ignoreHTTPSErrors":true,
         "headless":true
      },
      "name":"a",
      "id":"a",
      "type":"browser",
      "tags":[
         
      ],
      "hash":"C3Gij1/Fld8KFLhDTP25dY+GQRqOtK29VVGQ1Q1iIGE=",
      "content":"UEsDBBQACAAIAAAAIQAAAAAAAAAAAAAAAAAXAAAAam91cm5leXMvb25lLmpvdXJuZXkudHNVkD9PAzEMxff7FFamVDpdAImlFYiFhZ0JMZic2wvk4pD4WlXVfXfS+yOVxYqfnn/PjjHwzUMKdN6a90wpm5Z7F9zvQNZj+iHT0tHkc5COxNlshLI8mGUmGw7ULE0juXJ95CRwWaE1ZKFYQ8/BCScYYZ+4B/VCHnPh3ZDVrjpiApsIhd7mcXgCHbCnDTw9w6WCldMMmfS1B5AusYh34bCFWQFo+RQ8Y7uF+7t60YY4K4+r4EtMsOerZ1LGUsfNrtRl9ym5Bn2BiIfyiJiwz8WzLgPTcVp5HILtAGP0zqI4DqoGLJdZ0DdmADyhk4nWHFhYz8RmSH7KXfOvddxVHwoLR32pz2bP6RVtpzVOvH9/VLTi/wNQSwcI3bM26RQBAADIAQAAUEsBAi0DFAAIAAgAAAAhAN2zNukUAQAAyAEAABcAAAAAAAAAAAAgAKSBAAAAAGpvdXJuZXlzL29uZS5qb3VybmV5LnRzUEsFBgAAAAABAAEARQAAAFkBAAAAAA==",
      "filter":{
         "match":"a"
      }
   }
]

Notice how monitor adding and removing multiple tasks been 0 is included in the request, despite not changing that monitor.

[vigneshshanmugam]

When monitors are edited from the UI (in the case of enabling/disable for example), the hash is reset to '' to ensure that when configuration is repushed it marks it as needing an update.

Fixed 7e1a505

When changing monitors to enabled: false globally in the synthetics.config.ts file, monitors are marked as unchanged as the update is not pushed. Also, when setting tags, params or playrwight_options, screenshot globally, monitors are marked as unchanged.

We don't support those under monitor options. The Typings are wrong which will be addressed in a separate PR.

1. enabled - Needs to be supported as its super handy.
2. params|playwright_options|screenshot - They are available as separate top level fields to keep it compatible with how we run journeys and also to avoid repetition. So it would be as

{
  params: {},
  playwrightOptions: {},
  monitor: {. } // Not supported under here. 
}

3. tags - Tags are used as filtering in the top level for running journeys. So we might have to rethink how we want to support this.

Good that you brought up these concerns, Let move the discussion here #657 (comment) and unblock this PR.

Also, something more confusing is happening with enabled more generally. I have this monitor alongside the other test monitors

I am not able to reproduce it, Tried with different combinations. Would like to understand more if I have missed something. Lets sync and address it.

[shahzad31]

Smoke tested !!

By adding few hundred monitors !!
By deleting few of them !!
By editing few from code and ui !!

Things seems to be working as expected !!

[shahzad31]

(node:64665) ExperimentalWarning: stream/web is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
⚠ If you are using lightweight monitors with less than 1 minute resolution, these will be saved to the nearest supported frequency
> Pushing monitors for project: prod_overview_monitors
> bundling 1 monitors
✖ Erroring or updating 1 monitors (502ms)
   > Failed to save or update monitor. Configuration is not valid: monitor(production-logging-aws-af-south-1)
       Invalid value "undefined" supplied to "content"

it's failing for me on 8.5 with this error

[shahzad31]

....

[dominiqueclarke]

I can't recreate the error that Shahzad is seeing in 8.5, but I do have this error

✖ Error
   > Unsupported Heartbeat option: monitor(an-id3)
       The following Heartbeat options are not supported for http project monitors in 8.5.2: hash. You monitor was not created or updated.

   > Unsupported Heartbeat option: monitor(stuff)
       The following Heartbeat options are not supported for icmp project monitors in 8.5.2: hash. You monitor was not created or updated.

   > Unsupported Heartbeat option: monitor(http-monitor-with-host)
       The following Heartbeat options are not supported for http project monitors in 8.5.2: hash. You monitor was not created or updated.

   > Unsupported Heartbeat option: monitor(gmail-smtp)
       The following Heartbeat options are not supported for tcp project monitors in 8.5.2: hash. You monitor was not created or updated.

In 8.5.2, we did not support hash and so it's hitting our error handling. Is there any way we can omit it hash when pushing via the legacy flow?

Example heartbeat.yml

heartbeat.monitors:
- type: http
  id: 'an-id3'
  name: Elastic Homepage
  urls: ["https://www.google.com"]
  tags: 
    - org:elastic
  check.request.headers: 
    Content-Type: 'text/plain'
- type: icmp
  id: stuff
  name: Cloudflare DNS
  hosts: ["1.1.1.1"]
  tags: ["service:dns", "org:cloudflare"]
- type: http
  name: A name
  id: http-monitor-with-host
  urls: ["http://www.doordash.com"] 
  tags: ["service:dns"]
- type: tcp
  id: gmail-smtp
  name: GMail SMTP
  hosts: ["smtp.gmail.com:587"]
  tags: ["service:smtp", "org:google"]

[shahzad31]

also broken for 8.4.0

> bundling 2 monitors
TypeError: monitor.hash is not a function
    at buildMonitorSchema (/Users/shahzad-16/elastic/synthetics/src/push/monitor.ts:129:21)
    at async pushLegacy (/Users/shahzad-16/elastic/synthetics/src/push/index.ts:252:15)
    at async push (/Users/shahzad-16/elastic/synthetics/src/push/index.ts:69:12)
    at async Command.<anonymous> (/Users/shahzad-16/elastic/synthetics/src/cli.ts:208:7)

[vigneshshanmugam]

✖ Erroring or updating 1 monitors (502ms)
Failed to save or update monitor. Configuration is not valid: monitor(production-logging-aws-af-south-1)
Invalid value "undefined" supplied to "content"

This happens if you are trying to push Lightweight monitors prior to 8.5 as all are treated as Browser monitors.

In 8.5.2, we did not support hash and so it's hitting our error handling. Is there any way we can omit it hash when pushing via the legacy flow?

Fixed.

[shahzad31]

LGTM !!

I think timer could be adjusted to be in seconds insead of mili !!