InferX platform 0.1.0 deployment

System Requirement

• OS: Linux Kernel > 5.8.0, it has been tested in Ubuntu 20.04 and 22.4
• Processor: X86-64/Amd64
• Docker: > 17.09.0
• Enable virtualization technology in BIOS (Usually in Security tab of BIOS)
• Memory: >=64GB
• Cuda: >= 12.5

Components overview

InferX platform includes InferX runtime and InferX platform service.

• InferX runtime: It is released as a tgz package at the repo release https://github.com/inferx-net/inferx/releases/tag/0.1.0
• InferX platform service: It will be installed with docker compose as https://github.com/inferx-net/inferx/blob/main/docker-compose.yml. There are following container images:
  1. inferx/inferx_dashboard:v0.1.0: The inferx webui dashboard.
  2. inferx/inferx_one:v0.1.0: The inferx platform services such as rest api gateway,scheduler, etc.
  3. inferx/spdk-container:v0.1.0: Optional. This is simple wrapper of https://spdk.io/. It is only needed when using InferX blob store.
  4. quay.io/keycloak/keycloak:latest: Keycloak image which used for Authentication.
  5. postgres:14.5: Standard postgres container. It stores inferx audit log, secret and keycloak configuration
  6. quay.io/coreos/etcd:v3.5.13: Inferx configurations such as tenant, namespace and model functions.

Install Steps

1. Install Inferx runtime package
   • downlaod inferx.tar.gz from https://github.com/inferx-net/inferx/releases/tag/0.1.0
   • unzip the package to /opt folder

cd /opt
sudo tar zxvf inferx.tar.gz

Create or update the docker configuration: /etc/docker/daemon.json as example. please add the section:

        "inferx": {
            "path": "/opt/inferx/bin/inferx"
        }

Restare docker

sudo systemctl restart docker

or

sudo systemctl restart docker.service

2. Running InferX service with docker composer

Clone this repo and Run the docker composer as makefile

git clone [email protected]:inferx-net/inferx.git
cd inferx
make run

3. Configure Keycloak: There are 2 options to configure the keycloak
   • Option#1: use keyclaok postgres data. Download keycloak database file from https://github.com/inferx-net/inferx/releases/download/0.1.0/postgres_keycloak.tar.gz and unzip that in /opt/inferx/data folder
   • Option#2: follow the steps at https://github.com/inferx-net/inferx/wiki/keycloak-configuration
4. Restart he inferx service

make stop
make run

now you can access inferx dashboard from http://localhost:81/

Submit User model to InferX platform

please follow the steps at https://github.com/inferx-net/inferx/wiki/Submit-User-model-to-InferX-platform

InferX support high perform blob store. To configure that, please reference document https://github.com/inferx-net/inferx/wiki/InferX-Snapshot-and-restore-configuration.