help request: service health check interval

[hgaong]

Description

service

{
  "id": "557101231575663294",
  "create_time": 1741587810,
  "update_time": 1742293721,
  "name": "test-api",
  "upstream": {
    "nodes": [
      {
        "host": "172.31.19.1",
        "port": 80,
        "weight": 100
      },
      {
        "host": "172.31.20.9",
        "port": 80,
        "weight": 100
      }
    ],
    "retries": 0,
    "timeout": {
      "connect": 30,
      "send": 30,
      "read": 30
    },
    "type": "roundrobin",
    "checks": {
      "active": {
        "concurrency": 2,
        "healthy": {
          "http_statuses": [
            200,
            302,
            301,
            404
          ],
          "interval": 1,
          "successes": 2
        },
        "http_path": "/apisix/hearbeat",
        "port": 80,
        "timeout": 1,
        "type": "http",
        "unhealthy": {
          "http_failures": 2,
          "http_statuses": [
            429,
            500,
            501,
            502,
            503,
            504,
            505
          ],
          "interval": 1,
          "tcp_failures": 2,
          "timeouts": 3
        }
      }
    },
    "scheme": "http",
    "pass_host": "pass",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1000,
      "size": 640
    }
  },
  "plugins": {
    "ip-restriction": {
      "_meta": {
        "disable": true
      },
      "blacklist": [
        "108.136.33.21"
      ]
    },
    "limit-req": {
      "_meta": {
        "disable": true
      },
      "burst": 0,
      "key": "http_x_forwarded_for",
      "key_type": "var",
      "rate": 10,
      "rejected_code": 429,
      "rejected_msg": "ffffff"
    },
    "ua-restriction": {
      "_meta": {
        "disable": true
      },
      "bypass_missing": false,
      "denylist": [
        "curl"
      ]
    }
  }
} 

Then I configured around 600 routes, each of which referenced this service. The observed situation is that each node of the upstream service has about 20 heartbeat requests per second, which is inconsistent with the configured interval of 1 second. why ?

Environment

• APISIX version (run apisix version):3.11
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

The observed situation is that each node of the upstream service has about 20 heartbeat requests per second

How did you tally this information? Maybe try adding upstream.checks.active.req_headers configuration to make sure all requests are issued by APISIX's health checks

[hgaong]

The observed situation is that each node of the upstream service has about 20 heartbeat requests per second

How did you tally this information? Maybe try adding upstream.checks.active.req_headers configuration to make sure all requests are issued by APISIX's health checks

Through SkyWalking and the logs of my service, I have counted 20 heartbeat requests per second. It can be confirmed that only APISix will call this path "http_path": "/apisix/heartbeat".

[Baoyuantop]

Can you provide the configuration file for APISIX? It would help to reproduce the problem.

[hgaong]

Can you provide the configuration file for APISIX? It would help to reproduce the problem.
config.yaml

apisix:
  node_listen: 9080              # APISIX listening port
  enable_ipv6: false

deployment:
  admin:
    allow_admin:               # https://nginx.org/en/docs/http/ngx_http_access_module.html#allow
      - 0.0.0.0/0              # We need to restrict ip access rules for security. 0.0.0.0/0 is for test.

    admin_key:
      - name: "admin"
        key: secret                 # apikey
        role: admin                 # admin: manage all configuration data

  etcd:
    host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
      - "http://etcd:2379"          # multiple etcd address
    prefix: "/apisix"               # apisix configurations prefix
    timeout: 30                     # 30 seconds

plugin_attr:
  skywalking:
    service_name: APISIX
    service_instance_name: "APISIX Instance Name"
    endpoint_addr: http://sw:12800

[Baoyuantop]

How did you deploy APISIX? Are there multiple instances running?

[hgaong]

How did you deploy APISIX? Are there multiple instances running?

I deployed two Apisix nodes, but the Spring Boot service log shows that heartbeat requests only come from the Apisix node with traffic. And when I turned off Apisix's health check, all heartbeat requests from the Spring Boot service disappeared. When I reopened it, heartbeat requests appeared again.

[hgaong]

How did you deploy APISIX? Are there multiple instances running?

I deployed two Apisix nodes, but the Spring Boot service log shows that heartbeat requests only come from the Apisix node with traffic. And when I turned off Apisix's health check, all heartbeat requests from the Spring Boot service disappeared. When I reopened it, heartbeat requests appeared again.

Another interesting phenomenon is that when I modify the heartbeat path under the apisix service to/apisix/heartbeat/ss, high-frequency /apisix/heartbeat requests still exist. When I turn off the apisix health check, both high-frequency /apisix/heartbeat requests and low-frequency high-frequency /apisix/heartbeat/ss will disappear simultaneously.

[hgaong]

When I restarted Apisix, the abnormal heartbeat disappeared. Guess it's caused by residual heartbeat tasks

[Baoyuantop]

You need to check the apisix log to confirm the frequency of health check requests.

[Baoyuantop]

If there is still a problem, please open it again.