Skip to content

Commit 076dd28

Browse files
ishanarya0ishanarya0
authored
docs: create entropy doc (#66)
* docs: add intro, installation & glossary * docs: add concepts * docs: add modules * docs: add CLI reference * docs: add job description * docs: add architecture details * docs: add module writing guide * docs: fix state struct description * docs: move http & cli into one * chore: remove layout.md
1 parent 73bad78 commit 076dd28

13 files changed

+1109
-0
lines changed

‎docs/concepts/architecture.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Architecture
2+
3+
This document describes the high-level architecture of Entropy to enable anyone interested in contributing understarnd it's flow quickly.
4+
5+
## Flow of logic
6+
7+
First let's have a bird's eye view on how the logic flows in Entropy. This diagram below will give you a high-level understanding of how an action performed by an Actor interacts with various components of Entropy.
8+
9+
![](architecture.png)
10+
11+
Now, let's discuss what happends in each of the action above one by one.
12+
13+
- 1: Entropy's `core` interacts with the Module, to sanitise and validate the raw configuration. This also includes the `Plan` phase of resource life cycle, which attaches all the pending actions to the resource.
14+
- 2: Next, is an interaction with the `Database`, where the resource is saved.
15+
- 3: Now, all the resources with a status pending, are picked up from the Database.
16+
- 4: Entropy `core` again interacts with the respective `Module`, to trigger the `Sync` phase of the resource life cycle. A `job-queue model` is used to handle sync operations. In this part, Module can call external APIs.
17+
- 5: The updated state returned by the Module in the previous step is saved in the Database.
18+
19+
Note: The actor can interact with the core either through APIs call or using Entropy's CLI.
20+
21+
## Code Layout
22+
23+
Here is how the basic layout of Entropy looks like:
24+
25+
```
26+
+ entropy
27+
|--+ cli/
28+
| |--+ serve.go {Load configs, invoke server.Serve()}
29+
| |--+ migrate.go {Load configs, execute store migration}
30+
|--+ core/
31+
| |--+ resource/
32+
| | |--+ resource.go {Resource type, Store iface, pure functions on Resource}
33+
| | |--+ state.go
34+
| |--+ module/
35+
| | |--+ module.go {Module iface, module related Error types}
36+
| | |--+ module_ext.go
37+
| | |--+ action.go
38+
| | |--+ registry.go
39+
| |--+ mocks/
40+
|--+ modules/
41+
| |--+ firehose/ {Module implementation}
42+
| |--+ kubernetes/ {Module implementation}
43+
|--+ pkg/
44+
| |--+ errors/ {Custom errors, with Code, Cause and message}
45+
| | |--+ errors.go
46+
| |--+ helm/ {Helm client and APIs}
47+
| | |--+ cliet.go
48+
| | |--+ kube_rest.go
49+
| | |--+ release.go
50+
| | |--+ status.go
51+
| |--+ kube/ {Kubernetes client and APIs}
52+
| | |--+ client.go
53+
| | |--+ config.go
54+
| |--+ logger/
55+
| | |--+ logger.go
56+
| |--+ metric/
57+
| | |--+ metric.go
58+
| |--+ version/
59+
| | |--+ version.go
60+
| |--+ worker/ {Worker Implementation}
61+
| | |--+ example/
62+
| | |--+ mocks/
63+
| | |--+ pgq/
64+
| | |--+ worker.go
65+
| | |--+ worker_option.go
66+
| | |--+ job.go
67+
|--+ internal/
68+
| |--+ server/
69+
| | |--+ v1/
70+
| | | |--+ server.go {Resource CRUD handlers}
71+
| | | |--+ mappers.go {Resource CRUD handlers}
72+
| | |--+ server.go {Server setup, Serve() function, etc.}
73+
| |--+ store/
74+
| | |--+ postgres {Implement resource.Store, module.Store using Postgres}
75+
|--+ docs/
76+
|--+ main.go {Setup cobra+viper & add commands}
77+
78+
```
79+
80+
***Some highlights:***
81+
82+
Domain oriented packages are inside `core/` (resource and module)
83+
84+
`internal/` for keeping packages that should not be imported by any other projects (e.g., server, store, etc.)
85+
86+
`pkg/` for truly reusable (independent of entropy specific things) packages.
87+
88+
Mocks for interfaces are defined close to the interface definition in an isolated `mocks/` package insede `core/`.

‎docs/concepts/architecture.png

289 KB
Loading

‎docs/concepts/job.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Job
2+
3+
A job is an action that needs to be performed by Entropy asynchronously. It has a kind which maps it to a JobFn and a payload that is passed to the JobFn. Jobs are picked by worker treads for execution.
4+
5+
The Job struct looks like this:
6+
7+
```
8+
type Job struct {
9+
// Specification of the job.
10+
ID string `json:"id"`
11+
Kind string `json:"kind"`
12+
RunAt time.Time `json:"run_at"`
13+
Payload []byte `json:"payload"`
14+
15+
// Internal metadata.
16+
Status string `json:"status"`
17+
CreatedAt time.Time `json:"created_at"`
18+
UpdatedAt time.Time `json:"updated_at"`
19+
20+
// Execution information.
21+
Result []byte `json:"result,omitempty"`
22+
AttemptsDone int64 `json:"attempts_done"`
23+
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
24+
LastError string `json:"last_error,omitempty"`
25+
}
26+
```
27+
28+
## Sanitise
29+
30+
Sanitise the job fields.
31+
32+
```
33+
func (j *Job) Sanitise() error
34+
```
35+
36+
## Attempt
37+
38+
Attempt attempts to safely invoke `fn` for this job. Handles success, failure and panic scenarios and updates the job with result in-place.
39+
40+
```
41+
func (j *Job) Attempt(ctx context.Context, now time.Time, fn JobFn)
42+
```

‎docs/concepts/modules.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Modules
2+
3+
Module are responsible for achieving desired external system states based on a resource in Entropy.
4+
5+
This is how the module interface looks like:
6+
7+
```
8+
type Module interface {
9+
Plan(ctx context.Context, spec Spec, act ActionRequest) (*resource.Resource, error)
10+
11+
Sync(ctx context.Context, spec Spec) (*resource.State, error)
12+
}
13+
```
14+
15+
Every Module has a `Plan` and a `Sync` method which plays it's part in the resource lifecycle.
16+
17+
Entropy currently support firehose and kubernetes modules, with more lined up.

‎docs/concepts/resource-life-cycle.md

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Resource Life Cycle
2+
3+
## Resource
4+
5+
A resource represents the current state and desired state of real-world entities. A resource definition look like
6+
7+
```
8+
type Resource struct {
9+
Resource metadata.
10+
URN string
11+
Name string
12+
Kind string
13+
Project string
14+
Labels map[string]string
15+
Version int
16+
CreatedAt time.Time
17+
UpdatedAt time.Time
18+
19+
Resource spec & current-state.
20+
Spec Spec
21+
State State
22+
}
23+
24+
type Spec struct {
25+
Configs map[string]interface{}
26+
Dependencies map[string]string
27+
}
28+
29+
type State struct {
30+
ModuleData json.RawMessage
31+
Status string
32+
Output Output
33+
}
34+
35+
type Output map[string]interface{}
36+
```
37+
38+
The `Resource` definition is self explanatory. It has the `Spec` field which holds the `Configs` and `Dependencies` of a resource. The `State` field has three parts, `Status` holds the current status of a resource, `Output` holds the outcome of the latest action performed while `Data` holds the transactory information which might be used to perform actions on the reosurce.
39+
40+
For instance, a [firehose](https://github.com/odpf/firehose) resource looks like:
41+
42+
```
43+
{
44+
"name": "foo",
45+
"parent": "bar",
46+
"kind": "firehose",
47+
"spec": {
48+
"configs": {
49+
"log_level": "INFO"
50+
},
51+
"dependencies": {
52+
"deployment_cluster": "urn:odpf:entropy:kubernetes:godata"
53+
}
54+
},
55+
"state": {
56+
"status": "STATUS_PENDING"
57+
}
58+
}
59+
```
60+
61+
## Resource Lifecycle - The Plan & Sync Approach
62+
63+
We use a Plan and Sync approach for the resource lifecycle in Entropy. For illustration, we will take you through each of the steps in the lifecycle for a firehose resource.
64+
65+
### 1. Create a resource
66+
67+
```
68+
POST /api/v1beta1/resources
69+
70+
{
71+
"name": "foo",
72+
"parent": "bar",
73+
"kind": "firehose",
74+
"configs": {
75+
"log_level": "INFO"
76+
},
77+
"dependencies": {
78+
"deployment_cluster": "urn:odpf:entropy:kubernetes:godata"
79+
}
80+
}
81+
```
82+
83+
### 2. Plan phase
84+
85+
Plan validates the action on the current version of the resource and returns the resource with config/status/state changes (if any) applied. Plan DOES NOT have side-effects on anything other thing than the resource.
86+
87+
Here is the resource returned
88+
89+
```
90+
{
91+
"urn": "urn:odpf:entropy:firehose:bar:foo",
92+
"created_at": "2022-04-28T11:00:00.000Z",
93+
"updated_at": "2022-04-28T11:00:00.000Z",
94+
"name": "foo",
95+
"parent": "bar",
96+
"kind": "firehose",
97+
"configs": {
98+
"log_level": "INFO"
99+
},
100+
"dependencies": {
101+
"deployment_cluster": "urn:odpf:entropy:kubernetes:godata"
102+
},
103+
"state": {
104+
"status": "STATUS_PENDING",
105+
"output": {},
106+
"data": {
107+
"pending": ["helm_release"]
108+
}
109+
}
110+
}
111+
```
112+
113+
### 3. The Sync Phase
114+
115+
Sync is called repeatedly by Entropy core until the returned state has `StatusCompleted`.Module implementation is free to execute an action in a single Sync() call or split into multiple steps for better feedback to the end-user about the progress.
116+
117+
A job-queue model is used to handle sync operations. Every mutation (create/update/delete) on resources will lead to enqued jobs which will be processed later by workers.
118+
119+
### 4. Get resource (After Sync completion)
120+
121+
```
122+
{
123+
"urn": "urn:odpf:entropy:firehose:bar:foo",
124+
"kind": "firehose",
125+
"name": "foo",
126+
"project": "bar",
127+
"created_at": "2022-04-28T11:00:00.000Z",
128+
"updated_at": "2022-04-28T11:00:00.000Z",
129+
"configs": {
130+
"log_level": "INFO"
131+
},
132+
"dependencies": {
133+
"deployment_cluster": "urn:odpf:entropy:kubernetes:godata"
134+
},
135+
"state": {
136+
"status": "COMPLETED",
137+
"output": {
138+
"app": "entropy-firehose-bar-foo"
139+
}
140+
}
141+
}
142+
```
143+
144+
### 5. Execute Action
145+
146+
```
147+
POST /api/v1beta1/resources/urn:odpf:entropy:firehose:bar:foo/execute
148+
149+
{
150+
"action": "increase_log_level"
151+
}
152+
```
153+
154+
This will trigger Plan again, and leave the resource in `STATUS_PENDING` state. These resource will be later picked by the Sync in entropy core.
155+
156+
```
157+
{
158+
"urn": "urn:odpf:entropy:firehose:bar:foo",
159+
"kind": "firehose",
160+
"name": "foo",
161+
"project": "bar",
162+
"created_at": "2022-04-28T11:00:00.000Z",
163+
"updated_at": "2022-04-28T11:00:00.000Z",
164+
"configs": {
165+
"log_level": "WARN"
166+
},
167+
"dependencies": {
168+
"deployment_cluster": "urn:odpf:entropy:kubernetes:godata"
169+
},
170+
"state": {
171+
"status": "PENDING",
172+
"output": {
173+
"app": "entropy-firehose-bar-foo"
174+
},
175+
"data": {
176+
"pending": ["helm_release"]
177+
}
178+
}
179+
}
180+
```

0 commit comments

Comments
 (0)