Architecture

Engineering decisions and system design

This page documents the architecture philosophy behind InClouds and the technical design of its core system components. It is written for engineers evaluating whether the approach maps to their infrastructure requirements.

Why TOSCA before Terraform

Terraform is an execution engine. It knows how to provision resources on a given provider — it does not know what the system is supposed to do, what constraints it must satisfy, or what the relationships between components mean at a semantic level.

TOSCA separates intent from execution. A TOSCA model describes the topology, capabilities, and requirements of a system independently of any provider. When the model is complete and valid, Terraform HCL is derived from it — not authored.

This matters when you need to retarget a deployment from AWS to GCP, or run the same workload across two providers simultaneously. The topology stays constant. Only the compilation target changes.

# TOSCA (source of truth)
node_templates:
  api_server:
    type: tosca.nodes.Compute
    capabilities:
      host:
        properties:
          num_cpus: 8
          mem_size: 32 GB

# ↓ Compiled to provider-specific HCL
# aws → aws_instance.api_server
# gcp → google_compute_instance.api_server
# azure → azurerm_linux_virtual_machine.api_server

Separation of intent vs. execution

Imperative IaC encodes both intent and execution in the same artifact. Changing the execution target requires touching the same files that express the intent. This creates coupling that makes multi-cloud systems brittle.

InClouds enforces a strict boundary: TOSCA expresses intent, the compilation pipeline expresses execution. Policies and constraints (tagging, naming, CIDR allocation, IAM boundaries) are applied at compile time, not baked into individual resource blocks.

The result is an infrastructure codebase where topology authors and platform engineers work in separate layers with explicit contracts between them.

# Intent layer (TOSCA)
groups:
  network_isolation:
    type: tosca.groups.Root
    members: [api_server, db_primary]
    policies:
      - network_boundary

# Execution layer (generated)
# VPC, subnets, security groups, and
# routing rules are derived — not authored.

Graph-based modeling vs. imperative IaC

Infrastructure has inherent graph structure. Services connect to databases. Load balancers route to compute pools. IAM roles bind to service accounts. Encoding this as flat HCL loses the semantic relationships.

InClouds models infrastructure as a directed property graph. Nodes are typed components. Edges are typed relationships. Graph traversal algorithms validate dependency closure, detect cycles, compute deployment order, and enforce placement constraints.

This enables capabilities that flat IaC cannot express: topology diffing, reachability analysis, component substitution, and systematic impact analysis when a component type changes.

relationships:
  - type: tosca.relationships.ConnectsTo
    source: api_server
    target: pg_primary
    properties:
      protocol: postgresql
      port: 5432
    interfaces:
      Configure:
        post_configure_source:
          implementation: scripts/inject_dsn.sh

# Graph traversal verifies:
# - pg_primary capability HostedOn satisfied
# - Port not already allocated
# - Network path exists between endpoints

System components

How the platform is built

Retrieval

TOSCA Component RAG

When authoring a topology, component types are resolved from a vector-indexed catalog of TOSCA type definitions. Semantic search retrieves the closest matching node type given a natural-language description or partial schema. Retrieved types are validated against their schema before insertion into the graph.

Implementation notes

Vector store: Milvus / pgvector
Embedding: text-embedding-ada-002 compatible
Schema validation: TOSCA Simple Profile 1.3
Fallback: exact match on TOSCA type identifier

Processing

Background queue layer

Compilation from TOSCA to Terraform is CPU-bound and involves graph traversal, policy evaluation, and HCL codegen. This runs off the request path via a durable queue. Jobs are idempotent and keyed on the model hash. Clients poll for completion or receive a webhook on readiness.

Implementation notes

Queue: NATS JetStream / RabbitMQ
Retry: exponential backoff with dead-letter
Idempotency: content-addressed model hash
Status: streaming via SSE on job ID

Generation

Graph → Terraform pipeline

The compilation pipeline performs topological sort on the graph, resolves provider-specific type mappings, applies policy rules, and generates HCL modules per component cluster. Each module is independently testable. The pipeline is deterministic given the same model and target provider.

Implementation notes

Provider targets: AWS, GCP, Azure
Module boundaries: derived from graph clusters
State: Terraform remote state per cluster
Drift: graph diff against live state via provider APIs

Validation

Constraint solver

Before compilation, the graph is checked for validity: dependency closure, capability satisfaction, cyclic dependencies, and constraint evaluation. Constraint definitions follow TOSCA's built-in constraint clauses extended with custom operators for network CIDRs, IAM scope, and availability zone placement.

Implementation notes

Dependency closure: DFS on requirement/capability graph
Cycle detection: Tarjan's algorithm
Constraint evaluation: custom expression evaluator
Output: structured error list with node path context

Data flow

Request to deployment


  ┌─────────────────────────────────────────────────────────────────────────────┐
  │                          InClouds Platform                                  │
  │                                                                             │
  │   ┌──────────────┐    ┌───────────────────┐    ┌──────────────────────┐    │
  │   │  Editor UI   │───▶│   TOSCA Model API  │───▶│   Graph Store (GDB)  │    │
  │   │  (WebSocket) │    │   (REST + GraphQL) │    │   (typed prop graph) │    │
  │   └──────────────┘    └───────────────────┘    └──────────────────────┘    │
  │                                │                           │                │
  │                     ┌──────────▼──────────┐               │                │
  │                     │   RAG Component      │               │                │
  │                     │   Retrieval Service  │               │                │
  │                     │   (TOSCA type index) │               │                │
  │                     └──────────────────────┘               │                │
  │                                                             │                │
  │                     ┌──────────────────────────────────────▼──────────┐     │
  │                     │   Compilation Queue (NATS JetStream)             │     │
  │                     │   Job: model_hash → provider_target              │     │
  │                     └─────────────────────┬────────────────────────────┘    │
  │                                           │                                 │
  │                     ┌─────────────────────▼────────────────────────────┐   │
  │                     │   Compiler Workers                                │   │
  │                     │                                                   │   │
  │                     │   1. Topological sort (Kahn's algorithm)          │   │
  │                     │   2. Constraint evaluation                        │   │
  │                     │   3. Provider type mapping                        │   │
  │                     │   4. HCL codegen per cluster                      │   │
  │                     └─────────────────────┬────────────────────────────┘   │
  │                                           │                                 │
  │              ┌─────────────┬─────────────▼──────────────┐                 │
  │              │             │                             │                  │
  │      ┌───────▼────┐ ┌──────▼─────┐              ┌───────▼────┐            │
  │      │ AWS HCL    │ │ GCP HCL    │              │ Azure HCL  │            │
  │      │ (modules/) │ │ (modules/) │              │ (modules/) │            │
  │      └────────────┘ └────────────┘              └────────────┘            │
  └─────────────────────────────────────────────────────────────────────────────┘