Set Up Local Development

Instill Core is currently under rapid development. The project is built with the microservice architecture. To develop each service independently, we assign profiles to each service in the docker-compose-latest.yml file in Core. This allows us to selectively enable services for various purposes, e.g., debugging, development.

📘

Services are associated with profiles through the profiles attribute, which takes an array of profile names. A service will be started only if one of its profile names is activated. A service without profiles will always be started.

Core's profiles are defined as exclusion profiles, i.e., instead of launching a service and its dependencies, they'll launch all the services except the specified one. This optimizes the development experience, where a service is built locally in order to apply some changes in the source code while its dependencies tend to keep unchanged.

The following profiles are available on Core:

• all - start all services
• exclude-api-gateway - start all the services but api-gateway
• exclude-artifact - start all the services but artifact-backend
• exclude-console - start all the services but console
• exclude-mgmt - start all the services but mgmt-backend
• exclude-model - start all the services but model-backend
• exclude-pipeline - start all the services but pipeline-backend

Use one of the profile names to develop the corresponding service:

# In the `instill-core` repository folder
make latest PROFILE=all # or other profile name

The following guideline shows a specific example of how to develop the pipeline-backend service. For more details, you can read the service's contribution guidelines.

Start dependent services for pipeline-backend

On the local machine, clone the instill-core repository in your workspace, move to the repository folder, and launch all the dependent services:

# Clone the core repository
git clone https://github.com/instill-ai/instill-core.git
cd instill-core

# Use profile `exclude-pipeline` to launch all dependent services
# for `pipeline-backend`
make latest PROFILE=exclude-pipeline

Run dev pipeline-backend

Clone the pipeline-backend repository in your workspace and move to the repository folder:

git clone https://github.com/instill-ai/pipeline-backend.git
cd pipeline-backend

Build & run the dev image

make build-dev
make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

Start the pipeline-backend server

docker exec -it pipeline-backend bash
go run ./cmd/migration
go run ./cmd/init
go run ./cmd/main

Start the Temporal worker

docker exec -it pipeline-backend bash
go run ./cmd/worker

Run the integration tests

During local development, you can run the integration test to make sure your latest pipeline-backend works as intended:

docker exec -it pipeline-backend bash
make integration-test API_GATEWAY_URL=api-gateway:8080 DB_HOST=pg-sql

Remove the dev container

make rm

Shut down all dependent Core services

# In the `instill-core` repository folder
make down