Feat/integrate portal (WIP)

.env.example

@@ -125,7 +125,7 @@ PLUGINS_DIRECTORY
 # ╚═╝╚═════╝ ╚══════╝     ╚═════╝╚══════╝╚═╝╚══════╝╚═╝  ╚═══╝   ╚═╝   
 
 # The URL of the app portal
-VITE_APP_PORTAL_URL
+NEXT_PUBLIC_APP_URL=http://localhost:3000
 
 # The root url of the IDE server
 SERVER_ROOT_URL
@@ -182,12 +182,15 @@ PORTAL_SHADOW_DATABASE_URL
 NEXT_PUBLIC_APP_URL
 
 # URL of the IDE API
-NEXT_PUBLIC_API_URL
+NEXT_PUBLIC_API_URL=http://localhost:3030
 
 # The root url of the IDE server
-IDE_SERVER_UR
+IDE_SERVER_URL=http://localhost:3030
 
-APP_URL
+APP_URL=http://localhost:3000
+
+# URL for the API server
+API_URL=http://localhost:3030
 
 #  Stripe stuff
 NEXT_PUBLIC_STRIPE_PUB_KEY

README.md

@@ -90,76 +90,15 @@ Triggers tell nodes to start asynchronous tasks. Some nodes can process data wit
 
 # ⚙️ Installation
 
-## 📚 [Documentation / Guide](https://magick-docs.vercel.app/)
+Please follow the installation guide for your operating system:
 
-### Prerequisites
+- [macOS Installation Guide](docs/installation/macos.md)
+- [Linux Installation Guide](docs/installation/linux.md)
+- [Windows Installation Guide](docs/installation/windows.md)
 
-Before you start, ensure you have the following software installed on your machine:
-
-- **git**: Version control system, required for cloning the repository.
-  - [Download git](https://git-scm.com/downloads)
-- **node.js 18+**: JavaScript runtime, needed for running the application.
-  - [Download Node.js](https://nodejs.org/en/download/)
-- **Docker**: Enables you to run the project within containers for easier setup and distribution.
-  - [Download Docker Desktop](https://www.docker.com/products/docker-desktop)
-
-Follow the respective installation guides to set up each piece of software.
-
-- Install **pipx**: A tool for installing and running Python applications in isolated environments.
-
-On Unix and macOS: Open a terminal and run the following command:
-
-```bash
-python3 -m pip install --user pipx
-python3 -m pipx ensurepath
-```
-
-On Windows: Open a command prompt and execute:
-
-```bash
-py -m pip install --user pipx
-py -m pipx ensurepath
-```
-
-Install Poetry (Manages python packages)
-
-```bash
-pipx install poetry
-```
-
-Verify Installation
-
-```bash
-pipx --version
-```
-
-Once installed, proceed to set up Magick.
-
-## Setup
-
-Clone and set up Magick
-
-```bash
-git clone https://github.com/Oneirocom/Magick
-cd Magick
-npm install
-poetry install --no-root
-npm run dev
-```
-
-## Run Magick!
-
-```bash
-npm run dev
-```
-
-Note: Installation is automatic. Most Node projects require `npm install` - With Magick, dependencies will automatically be installed for you. Linux users may need to enter sudo password to install some dependencies.
-
-#### Build
-
-Build will take some time initially. When everything is ready, the client will be ready at [localhost:4200](http://localhost:4200/home)
-
-_Please be aware Magick is under heavy development which may cause breaking changes._
+For detailed development documentation and architecture overview, see:
+- [Development Guide](docs/development/README.md)
+- [Architecture Overview](docs/architecture/README.md)
 
 ## Database
 
@@ -182,8 +121,7 @@ The following documents should help you with setup:
 
 Magick uses [Feathers 5](https://feathersjs.com/) for backend, which in turn uses [Knex](https://knexjs.org/) for making database queries. We will offer a better database configuration experience in the future. For now, you will need to manually configure the database connection in the [`.env` file](.env) and then run the migration script.
 
-```
-cd apps/server
+```cd apps/server
 npm run migrate
 ```
 

docs/architecture.md

@@ -0,0 +1,84 @@
+# Service Architecture
+
+## Service Hierarchy
+
+### 1. Infrastructure Services (Required)
+
+- **PostgreSQL Databases**
+  - Main DB (`localhost:5432`)
+  - Shadow DB (`localhost:5433`)
+  - Purpose: Stores spells, agents, and application data
+- **Redis** (`localhost:6379`)
+  - Purpose: Caching and pub/sub for real-time updates
+- **S3Mock** (`localhost:9000`)
+  - Purpose: File storage for assets and templates
+
+### 2. Core Backend Services
+
+- **IDE/Agent Server** (`http://localhost:3030`)
+  - Primary backend service
+  - Handles agent execution, spell management
+  - Provides API endpoints for agent operations
+  - Dependencies: PostgreSQL, Redis
+
+### 3. Frontend Services
+
+- **Portal** (`http://localhost:3000`)
+  - Main user interface
+  - Template gallery and management
+  - Project management
+  - Dependencies: IDE Server, Authentication
+
+## URL Structure
+
+### Development Environment
+
+```
+Portal Frontend: http://localhost:3000
+IDE/Agent Server: http://localhost:3030
+PostgreSQL Main: localhost:5432
+PostgreSQL Shadow: localhost:5433
+Redis: localhost:6379
+S3Mock: localhost:9000
+```
+
+### Production Environment
+
+```
+Portal Frontend: https://app.magickml.com
+IDE/Agent Server: https://api.magickml.com
+```
+
+## Service Dependencies
+
+```mermaid
+graph TD
+    A[Portal Frontend] --> B[IDE/Agent Server]
+    B --> C[PostgreSQL Main]
+    B --> D[PostgreSQL Shadow]
+    B --> E[Redis]
+    B --> F[S3Mock]
+    A --> G[Clerk Auth]
+```
+
+## Startup Sequence
+
+1. Infrastructure Services: `npm run portal:up`
+   - Starts PostgreSQL, Redis, and S3Mock containers
+2. Database Initialization:
+   - `npm run db:init` (Main database)
+   - `npm run portal:db:init` (Portal database + templates)
+3. Backend Server: `npm run dev:server`
+4. Frontend Portal: `npm run portal:dev`
+
+## Environment Configuration
+
+The system uses several URL configurations that must be consistent:
+
+- `NEXT_PUBLIC_APP_URL`: Portal frontend URL
+- `NEXT_PUBLIC_API_URL`: IDE/Agent server URL
+- `IDE_SERVER_URL`: IDE server's self-reference
+- `APP_URL`: Internal portal URL
+- `API_URL`: Internal API URL
+
+For local development, these all use the localhost URLs shown above. In production, they use the corresponding production URLs.

docs/architecture/README.md

@@ -0,0 +1,177 @@
+# Architecture Overview
+
+## System Components
+
+### 1. Frontend (Client)
+
+- React-based web application
+- Visual node-based programming interface
+- Real-time collaboration features
+- State management and data flow
+- Plugin system integration
+
+### 2. Backend (Server)
+
+- Feathers.js framework
+- RESTful API endpoints
+- WebSocket support
+- Authentication and authorization
+- Database interactions
+- Plugin management
+
+### 3. Database
+
+- PostgreSQL with pgvector extension
+- Stores:
+  - User data
+  - Project configurations
+  - Spell definitions
+  - Agent states
+  - Plugin data
+
+### 4. Plugin System
+
+- Modular architecture
+- Hot-reloadable plugins
+- Standardized interfaces
+- Custom node types
+- Service integrations
+
+## Data Flow
+
+```mermaid
+graph TD
+    A[Client] -->|HTTP/WebSocket| B[Server]
+    B -->|Query/Update| C[Database]
+    B -->|Load/Execute| D[Plugins]
+    D -->|Results| B
+    B -->|Response| A
+```
+
+## Key Concepts
+
+### Spells
+
+- Visual programming workflows
+- Composed of nodes and connections
+- JSON-serializable format
+- Executable logic units
+
+### Nodes
+
+- Atomic processing units
+- Input/output sockets
+- Type system
+- Custom implementations
+
+### Agents
+
+- Autonomous entities
+- State management
+- Event handling
+- Multi-modal interactions
+
+## Security Architecture
+
+### Authentication
+
+- JWT-based auth
+- Role-based access control
+- API key management
+- Session handling
+
+### Data Protection
+
+- Input validation
+- SQL injection prevention
+- XSS protection
+- CORS configuration
+
+## Scalability Considerations
+
+### Horizontal Scaling
+
+- Stateless server design
+- Load balancer ready
+- Database connection pooling
+- Caching strategies
+
+### Performance
+
+- Optimized database queries
+- Efficient WebSocket usage
+- Resource monitoring
+- Memory management
+
+## Development Architecture
+
+### Local Development
+
+- Hot reloading
+- Development database
+- Debug tooling
+- Testing framework
+
+### Deployment
+
+- Docker containers
+- Environment configuration
+- Database migrations
+- Health monitoring
+
+## Integration Points
+
+### External Services
+
+- OAuth providers
+- Cloud services
+- AI/ML services
+- Webhook handlers
+
+### API Architecture
+
+- RESTful endpoints
+- WebSocket events
+- GraphQL (planned)
+- Plugin APIs
+
+## Future Considerations
+
+### Planned Improvements
+
+- Enhanced caching
+- Distributed processing
+- Real-time collaboration
+- Advanced monitoring
+
+### Architectural Goals
+
+- Maintainability
+- Extensibility
+- Performance
+- Security
+
+## Technical Specifications
+
+### Technology Stack
+
+- TypeScript/JavaScript
+- React
+- Node.js
+- PostgreSQL
+- Docker
+
+### Development Tools
+
+- npm/yarn
+- Jest
+- ESLint
+- Prettier
+- TypeScript
+
+## Additional Resources
+
+- [API Documentation](../api/)
+- [Database Schema](../database/)
+- [Plugin Development](../plugins/)
+- [Security Guidelines](../security/)