

>This ariticle was written by Ajeet Singh Raina and Wey Gu, originally published on [Docker blog](https://www.docker.com/blog/distributed-cloud-native-graph-database-nebulagraph-docker-extension/).

Graph databases have become a popular solution for storing and querying complex relationships between data. As the amount of graph data grows and the need for high concurrency increases, a distributed graph database is essential to handle the scale.

Finding a distributed graph database that automatically shards the data, while allowing businesses to scale from small to trillion-edge-level without changing the underlying storage, architecture of the service, or application code, however, can be a challenge. 

In this article, we’ll look at NebulaGraph, a modern, open source database to help organizations meet these challenges.


# Meet NebulaGraph
[NebulaGraph](https://nebula-graph.io) is a modern, open source, cloud-native graph database, designed to address the limitations of traditional graph databases, such as poor scalability, high latency, and low throughput. NebulaGraph is also highly scalable and flexible, with the ability to handle large-scale graph data ranging from small to trillion-edge-level.

NebulaGraph has built a thriving community of more than [1000 enterprise users](https://nebula-graph.io/cases) since 2018, along with a rich ecosystem of tools and support. These benefits make it a cost-effective solution for organizations looking to build graph-based applications, as well as a great learning resource for developers and data scientists.

The NebulaGraph cloud-native database also offers [Kubernetes Operators](https://github.com/vesoft-inc/nebula-operator) for easy deployment and management in cloud environments. This feature makes it a great choice for organizations looking to take advantage of the scalability and flexibility of cloud infrastructure.

# Architecture of the NebulaGraph database
NebulaGraph consists of three services: the Graph Service, the Storage Service, and the Meta Service (Figure 1). The Graph Service, which consists of stateless processes (nebula-graphd), is responsible for graph queries. The Storage Service (nebula-storaged) is a distributed (Raft) storage layer that persistently stores the graph data. The Meta Service is responsible for managing user accounts, schema information, and Job management. With this design, NebulaGraph offers great scalability, high availability, cost-effectiveness, and extensibility.

![Nebulagraph database architecture infographic](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-database-architecture-infographic.png)
Figure 1: Overview of NebulaGraph services.

# Why NebulaGraph?
NebulaGraph is ideal for graph database needs because of its architecture and design, which allow for high performance, scalability, and cost-effectiveness. The architecture follows a separation of storage and computing architecture, which provides the following benefits:

- **Automatic sharding**: NebulaGraph automatically shards graph data, allowing businesses to scale from small to trillion-edge-level data volumes without having to change the underlying storage, architecture, or application code.
- **High performance**: With its optimized architecture and design, NebulaGraph provides high performance for complex graph queries and traversal operations.
- **High availability**: If part of the Graph Service fails, the data stored by the Storage Service remains intact.
- **Flexibility**: NebulaGraph supports property graphs and provides a powerful query language, called Nebula Graph Query Language (nGQL), which supports complex graph queries and traversal operations. 
- **Support for APIs**: It provides a range of APIs and connectors that allow it to integrate with other tools and services in a distributed system.

# Why run NebulaGraph as a Docker Extension?
In production environments, NebulaGraph can be deployed on Kubernetes or in the cloud, hiding the complexity of cluster management and maintenance from the user. However, for development, testing, and learning purposes, setting up a NebulaGraph cluster on a desktop or local environment can still be a challenging and costly process, especially for users who are not familiar with containers or command-line tools.

This is where the NebulaGraph Docker Extension comes in. It provides an elegant and easy-to-use solution for setting up a fully functional NebulaGraph cluster in just a few clicks, making it the perfect choice for developers, data scientists, and anyone looking to learn and experiment with NebulaGraph.

## Getting started with NebulaGraph in Docker Desktop
### Setting up
Prerequisites: [Docker Desktop 4.10](https://docs.docker.com/desktop/) or later.

### Step 1: Enable Docker Extensions
You’ll need to enable Docker Extensions under the Settings tab in Docker Desktop. Within Docker Desktop, confirm that the **Docker Extensions** feature is enabled (Figure 2). Go to **Settings > Extensions** and select **Enable Docker Extensions**.

![Docker desktop extensions settings](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/docker-desktop-extensions-settings-980x557.png)
Figure 2: Enabling Docker Extensions within the 
Docker Desktop.

All Docker Extension resources are hidden by default, so, to ensure its visibility, go to **Settings > Extensions** and check the **Show Docker Extensions system containers**.

### Step 2: Install the NebulaGraph Docker Extension
The NebulaGraph extension is available from the Extensions Marketplace in Docker Desktop and on [Docker Hub](https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext). To get started, search for NebulaGraph in the Extensions Marketplace, then select **Install** (Figure 3).

![Nebulagraph extensions marketplace](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-extensions-marketplace-980x624.png)
Figure 3: Installing NebulaGraph from the Extensions Marketplace.

This step will download and install the latest version of the NebulaGraph Docker Extension from Docker Hub. You can see the installation process by clicking Details (Figure 4).


![Nebulagraph installation progress docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-installation-progress-docker-desktop-980x624.png)
Figure 4: Installation progress.

### Step 3: Waiting for the cluster to be up and running
After the extension is installed, for the first run, it normally takes fewer than 5 minutes for the cluster to be fully functional. While waiting, we can quickly go through the **Home** tab and **Get Started** tab to see details of NebulaGraph and NebulaGraph Studio, the WebGUI Utils.

We can also confirm whether it’s ready by observing the containers’ status from the **Resources** tab of the Extension as shown in Figure 5.

![Nebulagraph resources docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-resources-docker-desktop-980x624.png)
Figure 5: Checking the status of containers.

### Step 4: Get started with NebulaGraph
After the cluster is healthy, we can follow the Get Started steps to log in to the NebulaGraph Studio, then load the initial dataset, and query the graph (Figure 6).

![Nebulagraph get started docker desktop](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-get-started-docker-desktop-980x624.png)
Figure 6: Logging in to NebulaGraph Studio.

## Step 5: Learn more from the starter datasets 
In a graph database, the focus is on the relationships between the data. With the starter datasets available in NebulaGraph Studio, you can get a better understanding of these relationships. All you need to do is click the Download button on each dataset card on the welcome page (Figure 7).

![starter datasets](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebulagraph-starter-datasets-980x937.png)
Figure 7: Starter datasets.

For example, in the ```demo_sns``` (social network) dataset, you can use the following query to find new friend recommendations by identifying second-degree friends with the most mutual friends:

```
1 Einstein
```

![Nebula console demo sns](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-sns-980x782.png)
Figure 8: Query results shown in the Nebula console.

Instead of just displaying the query results, you can also return the entire pattern and easily gain insights. For example, in Figure 9, we can see LeBron James is on two mutual friend paths with Tim:

![Nebula console demo sns results graph](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-sns-results-graph-980x853.png)
Figure 9: Graphing the query results.


Another example can be found in the ```demo_fraud_detection``` (graph of loan) dataset, where you can perform a 10-degree check for risky applicants, as shown in the following query:
```Cypher
MATCH 
p_=(p:`applicant`)-[*1..10]-(p2:`applicant`)
WHERE id(p)=="p_200" AND 
p2.`applicant`.is_risky == "True"
RETURN p_ LIMIT 100
```

The results shown in Figure 10 indicate that this applicant is suspected to be risky because of their connection to ```p_190```.

![Nebula console demo fraud detection graph](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/docker-demo/nebula-console-demo-fraud-detection-graph-980x783.png)
Figure 10: Results of query showing fraud detection risk.

By exploring the relationships between data points, we can gain deeper insights into our data and make more informed decisions. Whether you are interested in finding new friends, detecting fraudulent activity, or any other use case, the starter datasets provide a valuable starting point.

We encourage you to download the datasets, experiment with different queries, and see what new insights you can uncover, then share with us in the NebulaGraph community.

# Try NebulaGraph for yourself

To learn more about NebulaGraph, visit our [website](https://nebula-graph.io), [documentation site](https://docs.nebula-graph.io/), star our [GitHub repo](https://github.com/vesoft-inc/nebula), or join our [community chat](https://community-chat.nebula-graph.io/).

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))



NebulaGraph Docker Extension provides an elegant and easy-to-use solution for setting up a fully functional NebulaGraph cluster in just a few clicks, making it the perfect choice for developers, data scientists, and anyone looking to learn and experiment with NebulaGraph.

Distributed Cloud-Native Graph Database with NebulaGraph Docker Extension


![Nebula Operator](https://user-images.githubusercontent.com/57335825/126621008-a011782d-98d8-4814-97cd-5a25a07318e7.png)

Hi, everybody! As you know, [Nebula Operator](https://github.com/vesoft-inc/nebula-operator) has been open-source, and we published an [introduction](https://nebula-graph.io/posts/nebula-operator-overview/) to it. Today, I would like to introduce my practice on Nebula Operator on cloud.

# About Nebula Operator

What is Nebula Operator? You can refer to [Nebula Operator Overview: Automated Operation on Kubernetes](https://nebula-graph.io/posts/nebula-operator-overview/). Today, this article will focus on the practice on Nebula Operator. I hope it could help you get started with it soon and enjoy NebulaGraph.

# Practicing Nebula Operator on Cloud

In this section, I will introduce the process of deploying and using Nebula Operator on cloud. Alibaba Cloud services are used, but the process is applicable to the similar services of other cloud service providers.

## Installation Tools

Install the following tools on your on-premises machine:

* [Kubectl](https://kubernetes.io/zh/docs/tasks/tools/)
* [Helm](https://helm.sh/zh/docs/intro/install/)
* [Docker](https://docs.docker.com/get-docker/)

For the installation information, see their documentations.

## Creating Kubernetes Cluster on Cloud

Nebula Operator is implemented for Kubernetes applications, so Kubernetes is necessary for practicing Nebula Operator.

This practice was performed on a managed Kubernetes cluster on Alibaba Cloud, so the first thing is to log on to the Alibaba Cloud Container Service for Kubernetes (ACK) console and then to create a managed Kubernetes cluster. The configurations must meet your needs. 

NOTE: To facilitate access to the Kubernetes API Server from Internet, Expose API Server with EIP was enabled for this practice. Whether to enable it or not is determined by your needs. If it is not enabled, you need to make sure that your on-premises machine can connect to the Kubernetes cluster. Configure other parameters as required.

When the Kubernetes cluster starts, in the Connection Information tab, copy the Public Access content in the code block to the $HOME/.kube/config file of your on-premises machine.

And then on your on-premises machine, run the following command to check the connection to your Kubernetes cluster.

```plain
$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
cn-beijing.192.168.250.13    Ready    <none>   51m   v1.20.4-aliyun.1
cn-beijing.192.168.250.185   Ready    <none>   51m   v1.20.4-aliyun.1
cn-beijing.192.168.250.89    Ready    <none>   51m   v1.20.4-aliyun.1
```

## Installing Dependencies for Nebula Operator

Install all the dependencies listed in this section for Nebula Operator.

### Installing CertManager

Run the following command to install [CertManager](https://cert-manager.io/).

```plain
 Install CertManager
$ helm install cert-manager cert-manager --repo https://charts.jetstack.io \
    --namespace cert-manager --create-namespace --version v1.3.1 \
    --set installCRDs=true
# Wait a moment and then check whether CertManager is started successfully.
$ kubectl -n cert-manager get pod
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7998c69865-jfw9x              1/1     Running   0          93s
cert-manager-cainjector-7b744d56fb-846w9   1/1     Running   0          93s
cert-manager-webhook-7d6d4c78bc-ssk4w      1/1     Running   0          93s
```

### Installing OpenKruise

Run the following command to install [OpenKruise](https://openkruise.io/).

```plain
# Install OpenKruise
$ helm install kruise \
    https://github.com/openkruise/kruise/releases/download/v0.8.1/kruise-chart.tgz
# Wait for a moment and then check whether OpenKruise is started successfully.
$ kubectl -n kruise-system get pod
NAME                                         READY   STATUS    RESTARTS   AGE
kruise-controller-manager-6797f89d9b-ppv65   1/1     Running   0          49s
kruise-controller-manager-6797f89d9b-wlkbd   1/1     Running   0          49s
kruise-daemon-7rljq                          1/1     Running   0          49s
kruise-daemon-8kd8d                          1/1     Running   0          49s
kruise-daemon-n6tdw                          1/1     Running   0          49s
```

## Adding Nebula Operator Charts

Run the following command to add [Nebula Operator Charts](https://vesoft-inc.github.io/nebula-operator/charts).

```plain
# Add the Nebula Operator Charts repository.
$ helm repo add nebula-operator https://vesoft-inc.github.io/nebula-operator/charts
# Update the repository.
$ helm repo update
```

## Installing Nebula Operator

If you cannot pull the `gcr.io` or `k8s.gcr.io` image, use mirrors. For example, in this practice, the following mirrors were used.  

|Original Images|Mirrors|
|:----|:----|
|[gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0](http://gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0)|kubesphere/kube-rbac-proxy:v0.8.0|
|[k8s.gcr.io/kube-scheduler:v1.18.8](http://k8s.gcr.io/kube-scheduler:v1.18.8)|kubesphere/kube-scheduler:v1.18.8|

You can run the following command to view all the configurable parameters.

```plain
$ helm show values nebula-operator/nebula-operator
```

In this practice, I followed this procedure to install Nebula Operator.

```plain
# Install Nebula Operator
$ helm install nebula-operator nebula-operator/nebula-operator \
    --namespace nebula-operator-system --create-namespace --version 0.1.0 \
    --set image.kubeRBACProxy.image=kubesphere/kube-rbac-proxy:v0.8.0 \
    --set image.kubeScheduler.image=kubesphere/kube-scheduler:v1.18.8
# Wait for a moment and check whether Nebula Operator is started successfully.
$ kubectl -n nebula-operator-system get pod
NAME                                                             READY   STATUS    RESTARTS   AGE
nebula-operator-controller-manager-deployment-6968547fff-k62b4   2/2     Running   0          19s
nebula-operator-controller-manager-deployment-6968547fff-lhpdx   2/2     Running   0          19s
nebula-operator-scheduler-deployment-7c5fc7945-hbkv8             2/2     Running   0          19s
nebula-operator-scheduler-deployment-7c5fc7945-sxc7w             2/2     Running   0          19s
```

If Cluster Domain of Kubernetes are customized, modify the preceding installation command by adding configuration of `kubernetesClusterDomain`. 

```plain
# Install Nebula Operator and replace <<YourCustomCLusterDomain>> with your own cluster domain.
$ helm install nebula-operator nebula-operator/nebula-operator \
    --namespace nebula-operator-system --create-namespace --version 0.1.0 \
    --set image.kubeRBACProxy.image=kubesphere/kube-rbac-proxy:v0.8.0 \
    --set image.kubeScheduler.image=kubesphere/kube-scheduler:v1.18.8 \
    --set kubernetesClusterDomain=<<YourCustomCLusterDomain>>
```

## Deploying NebulaGraph Cluster

So far, Nebula Operator has been ready. Now, let's deploy a NebulaGraph cluster for handling graph data. 

The first one is to know StorageClass information, which will be used to configure the persistent volume storage for the NebulaGraph cluster.

```plain
$ kubectl get sc
NAME                       PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-available    diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-efficiency   diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-essd         diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-ssd          diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   100m
alicloud-disk-topology     diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   100m
```

As you can see from the result of the preceding command, five types of StorageClass are supported on Alibaba Cloud. For this practice, alicloud-disk-ssd was used. If you want to use the services provided by another cloud service provider, choose an applicable StorageClass type.

NOTE: Each cloud service provider may set limits on the storage capacity for your application. For example, on Alibaba Cloud, [the limit on SSD](https://partners-intl.aliyun.com/help/doc-detail/25513.htm) is from 20 GiB to 32,768 GiB. Therefore, you must specially consider this configuration when deploying a NebulaGraph cluster.

To view all the configurable parameters, you can run the following command.

```plain
$ helm show values nebula-operator/nebula-cluster
```

In this practice, all these commands were run to deploy a NebulaGraph cluster.

```plain
# Name the NebulaGraph cluster.
$ export NEBULA_CLUSTER_NAME=nebula

# Create a namespace for the NebulaGraph cluster.
$ export NEBULA_CLUSTER_NAMESPACE=nebula

# Specify StorageClass type for the NebulaGraph cluster. In this example, alicloud-disk-ssd is used.
$ export STORAGE_CLASS_NAME=alicloud-disk-ssd

# Specify the storage size for the three processes of NebulaGraph.
$ export STORAGE_SIZE_GRAPHD=20Gi
$ export STORAGE_SIZE_METAD=20Gi
$ export STORAGE_SIZE_STORAGED=20Gi

# Deploy the NebulaGraph cluster.
$ helm install ${NEBULA_CLUSTER_NAME} nebula-operator/nebula-cluster \
    --namespace ${NEBULA_CLUSTER_NAMESPACE} --create-namespace --version 0.1.0 \
    --set nameOverride=${NEBULA_CLUSTER_NAME} \
    --set nebula.storageClassName="${STORAGE_CLASS_NAME}" \
    --set nebula.graphd.storage="${STORAGE_SIZE_GRAPHD}" \
    --set nebula.metad.storage="${STORAGE_SIZE_METAD}" \
    --set nebula.storaged.storage="${STORAGE_SIZE_STORAGED}"

# Wait a moment and check whether the NebulaGraph cluster is started successfully.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get nebulacluster
NAME     GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   2                2              3               3             3                  3                4m10s
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get pod
NAME                READY   STATUS    RESTARTS   AGE
nebula-graphd-0     1/1     Running   0          96s
nebula-graphd-1     1/1     Running   0          96s
nebula-metad-0      1/1     Running   0          97s
nebula-metad-1      1/1     Running   0          97s
nebula-metad-2      1/1     Running   0          97s
nebula-storaged-0   1/1     Running   0          97s
nebula-storaged-1   1/1     Running   0          97s
nebula-storaged-2   1/1     Running   0          97s
```

Of course, you can also run the following command to upgrade the cluster to have five nebula-storaged process instances. 

```plain
# Upgrade the NebulaGraph cluster.
$ helm upgrade ${NEBULA_CLUSTER_NAME} nebula-operator/nebula-cluster \
    --namespace ${NEBULA_CLUSTER_NAMESPACE} --create-namespace --version 0.1.0 \
    --set nameOverride=${NEBULA_CLUSTER_NAME} \
    --set nebula.storageClassName="${STORAGE_CLASS_NAME}" \
    --set nebula.graphd.storage="${STORAGE_SIZE_GRAPHD}" \
    --set nebula.metad.storage="${STORAGE_SIZE_METAD}" \
    --set nebula.storaged.storage="${STORAGE_SIZE_STORAGED}" \
    --set nebula.storaged.replicas=5
#  Wait for a moment and check whether the cluster is started successfully.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get nebulacluster
NAME     GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   2                2              3               3             5                  5                6m12s
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get pod
NAME                READY   STATUS    RESTARTS   AGE
nebula-graphd-0     1/1     Running   0          2m30s
nebula-graphd-1     1/1     Running   0          2m30s
nebula-metad-0      1/1     Running   0          2m30s
nebula-metad-1      1/1     Running   0          2m30s
nebula-metad-2      1/1     Running   0          2m30s
nebula-storaged-0   1/1     Running   0          2m30s
nebula-storaged-1   1/1     Running   0          2m30s
nebula-storaged-2   1/1     Running   0          2m30s
nebula-storaged-3   1/1     Running   0          52s
nebula-storaged-4   1/1     Running   0          52s
```

For more information, see [Install Nebula Cluster with helm](https://github.com/vesoft-inc/nebula-operator/blob/master/doc/user/nebula_cluster_helm_guide.md) in NebulaGraph Database Manual.

## Connecting to NebulaGraph Cluster

If the NebulaGraph cluster is started successfully, here comes the last step: Connect to the cluster.

### Connecting from Inside of Kubernetes

Firstly, on Kubernetes, run the following command to start [NebulaGraph Console](https://github.com/vesoft-inc/nebula-console). 

```plain
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nebula-console
spec:
  containers:
    - name: nebula-console
      image: vesoft/nebula-console:v2-nightly
      command:
      - sleep
      - "1000000"
EOF
```

And then, run this command to connect to the cluster via the created NebulaGraph Console.

```plain
$ kubectl exec -it nebula-console -- \
    nebula-console -u u -p p --addr ${NEBULA_CLUSTER_NAME}-graphd-svc.${NEBULA_CLUSTER_NAMESPACE}.svc --port 9669
2021/06/23 06:21:22 [INFO] connection pool is initialized successfully
Welcome to NebulaGraph!
(u@nebula) [(none)]> show hosts
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| Host                                                                  | Port | Status   | Leader count | Leader distribution  | Partition distribution |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-3.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-4.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "Total"                                                               |      |          | 0            |                      |                        |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
Got 4 rows (time spent 7669/9367 us)
Wed, 23 Jun 2021 06:21:26 UTC
```

### Connecting from Outside of Kubernetes

If you want to connect to the services inside Kubernetes from outside, use `hostPort`, `hostNetwork`, `Ingress`, `LoadBalancer`, and so on. In this practice, I took advantages of the functionalities of Alibaba Cloud and used `LoadBalancer`.

***DO NOT: This operation will expose your NebulaGraph Cluster to the public, so do not use it in the production environment.***

Firstly, change the type of `nebula-graphd` to `LoadBalancer`, and then view `EXTERNAL-IP`. 

```plain
# Change the type of nebula-graphd to LoadBalancer.
$ kubectl patch -n ${NEBULA_CLUSTER_NAMESPACE} svc ${NEBULA_CLUSTER_NAME}-graphd-svc \
    -p '{"spec": {"type": "LoadBalancer"}}'
# View EXTERNAL-IP. If "pending" is shown, wait a moment and try again.
$ kubectl -n ${NEBULA_CLUSTER_NAMESPACE} get svc nebula-graphd-svc
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                          AGE
nebula-graphd-svc   LoadBalancer   172.16.85.222   x.x.x.x         9669:31460/TCP,19669:32579/TCP,19670:31481/TCP   27m
```

Now, you can use the `EXTERNAL-IP` address (for example, `x.x.x.x` in this example) to connect to the cluster.

```plain
$ export EXTERNAL_IP=x.x.x.x
$ docker run -it --rm vesoft/nebula-console:v2-nightly -u u -p p --addr ${EXTERNAL_IP} --port 9669
2021/06/23 06:42:17 [INFO] connection pool is initialized successfully
Welcome to NebulaGraph!
(u@nebula) [(none)]> show hosts
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| Host                                                                  | Port | Status   | Leader count | Leader distribution  | Partition distribution |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-3.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-4.nebula-storaged-headless.nebula.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "Total"                                                               |      |          | 0            |                      |                        |
+-----------------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
Got 4 rows (time spent 3747/60433 us)
Wed, 23 Jun 2021 06:42:21 UTC
```

# Enjoy

**You made it!**

**Have fun with NebulaGraph!**

Join our [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) if you want to discuss with the rest of the NebulaGraph community!


A step-by-step guide on deploying Nebula Operator on public cloud services.

Practicing Nebula Operator on Cloud


![Using Ansible to Automate Deployment of NebulaGraph Cluster](https://www-cdn.nebula-graph.com.cn/nebula-blog/ansible-new-world.jpeg)

# Background

To do tests on NebulaGraph, we need to frequently deploy it on servers, so we need a tool to improve the efficiency of deployment. The tool need to meet these major requirements:

* It can allow a non-root account to deploy NebulaGraph, so that we can usecgroupsto limit resource usage.
* It can modify the configuration files on the operating machines, and then distribute the configurations to the cluster for testing the parameters.
* It can be called by using scripts to ease integrating it to the testing platform or tool later.

About tool selection, there are several options. Fabric and Puppet are veterans and Ansible and SaltStack are rising stars.

The Ansible project on GitHub has earned more than 40 thousand stars. It was acquired by Red Hat in 2015, and its community is very active. A lot of open-source projects adopt Ansible for deployment. For example,[kubespray](https://github.com/kubernetes-sigs/kubespray?fileGuid=w4fEAGmypkwIQ89x)of Kubernetes and[tidb-ansible](https://github.com/pingcap/tidb-ansible?fileGuid=w4fEAGmypkwIQ89x)of TiDB. Therefore, we decided to use Ansible to deploy NebulaGraph.

# Introduction to Ansible

## Features

Ansible is an open-source project, but [Ansible Tower](https://www.ansible.com/products/tower?fileGuid=w4fEAGmypkwIQ89x), an automate deployment tool, is commercial. Ansible has these features:

* SSH protocol by default. Compared with SaltStack, Ansible manages machines in an agentless manner.
* Flexibility. Playbook, role, and module are used to define the deployment steps.
* Idempotent behavior.
* Supporting modular development with a variety of modules.

Some features have obvious advantages and disadvantages:

* Using SSH protocol:It enables Ansible to deploy a cluster on most machines with the password authentication mechanism by default and its disadvantage is that the performance will be compromised.
* Using playbook to define deployment steps and using Jinja2 as the template engine. For those who are familiar with these technologies, they are easy, but for new players, the learning curve is high.

In summary, Ansible is suitable for the batch deployment on a small batch of machines in anagentless manner, and this is right where we are.

## How to Deploy

Usually, to deploy a cluster offline, the machines are assigned three roles:

**Control node**: Ansible is installed on the control node and manages machines over the SSH protocol.

**Resource machine**: The tasks that need Internet access such as downloading RPM packages are executed on the resource machine.

**Managed node**: The managed nodes run the services. They can be in an isolated network. The services are deployed by Ansible.

![ansible workflow](https://user-images.githubusercontent.com/38887077/120767347-b4b75700-c54d-11eb-80c7-793435889d05.png)

## How to Execute Tasks

In Ansible, there are three levels of tasks:

* Module
* Role
* Playbook

Modules are the basic Ansible task units. There are two types of modules: CoreModule and CustomerModule.

A task is executed as follows:

1. In the control node, Ansible generates a new Python file based on the module code and the passed parameters.
2. The control node copies the Python file to the`tmp`folder in the managed nodes over SSH.
3. The control node executes the Python file over SSH.
4. The managed nodes return the execution result to the control node and then the folder is removed from the managed nodes.

![ansible workflow](https://user-images.githubusercontent.com/38887077/120781586-c99ae700-c55b-11eb-99e7-b80f19441ac2.png)

```plain
# Do not delete the tmp folder.
export ANSIBLE_KEEP_REMOTE_FILES=1
# Run -vvv to view the debug information.
ansible -m ping all -vvv
<192.168.8.147> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="nebula"' -o ConnectTimeout=10 -o ControlPath=/home/vesoft/.ansible/cp/d94660cf0d -tt 192.168.8.147 '/bin/sh -c '"'"'/usr/bin/python /home/nebula/.ansible/tmp/ansible-tmp-1618982672.9659252-5332-61577192045877/AnsiballZ_ping.py && sleep 0'"'"''
```

The debug information is returned as above.  The file named `AnsiballZ_ping.py`is the Python file generated based on the module. We can run this file on the managed node to do a check of the file.

```plain
python3 AnsiballZ_ping.py
#{"ping": "pong", "invocation": {"module_args": {"data": "pong"}}}
```

The result is the standard output of a Python file and Ansible processes the result additionally.

A role contains a series of tasks defined in modules. It uses register to pass context parameters.

Here is a typical example:

1. Creating a directory.
2. Continuing installation if the directory is created successfully, otherwise, canceling the deployment.


A playbook bridges the managed nodes and the roles.

Ansible can use inventories to group remote machines (managed nodes) and use different roles to perform deployment on the machines of different groups.The installation and deployment task can be completed flexibly.

After a playbook is defined, if we need to adapt the deployment to different environments, only changes to the machine configuration in the inventory are necessary.

## Customizing Modules

* Customizing filter

Ansible uses Jinja2 as the template engine.[Filter](https://ansible-docs.readthedocs.io/zh/stable-2.0/rst/playbooks_filters.html?fileGuid=w4fEAGmypkwIQ89x)of Jinja2 can be used.

```plain
# Uses the default filter. By default, 5 is output.
ansible -m debug -a 'msg={{ hello | default(5) }}' all
```

Sometimes, we need to customize`filter`to operate variables. For NebulaGraph, a typical scenario is configuring the`meta_server_addrs`parameter for the IP addresses of nebula-metad servers. For a cluster where only one nebula-metad process is deployed, the value of the parameter is in the`metad:9559`format. For those where three nebula-metad processes are deployed, the value of the parameter is in the`metad1:9559,metad2:9559,metad3:9559`format.

In an Ansible playbook project, create a new directory named`filter_plugins`and then a Python file named`map_format.py`in it. Copy the following content to the`.py`file.

```python
# -*- encoding: utf-8 -*-
from jinja2.utils import soft_unicode

def map_format(value, pattern):
    """
    e.g.  
    "{{ groups['metad']|map('map_format', '%s:9559')|join(',') }}"
    """
    return soft_unicode(pattern) % (value)
class FilterModule(object):
    """ jinja2 filters """
    def filters(self):
        return {
            'map_format': map_format,
        }
```

`{{ groups['metad'] | map('map_format', '%s:9559') | join(',') }}`is the value that we need.

* Customizing module

A custom module needs to be compliant with the Ansible framework, including obtaining parameters, standard returns, and error returns.  After customization, we need to configure`ANSIBLE_LIBRARY`in`ansible.cfg`to enable Ansible to obtain the custom module. For more information, see[https://ansible-docs.readthedocs.io/zh/stable-2.0/rst/developing_modules.html](https://ansible-docs.readthedocs.io/zh/stable-2.0/rst/developing_modules.html?fileGuid=w4fEAGmypkwIQ89x).

## Practice of Ansible on NebulaGraph

Startup of NebulaGraph is not complicated, so it is very simple to use Ansible to deploy a NebulaGraph cluster.

1. Downloading the RPM package.
2. Copying the RPM package to managed nodes. Unzipping the package and moving it to the target folder.
3. Updating the configuration files.
4. Running shell scripts to start the cluster.

## Using Common Role

NebulaGraph has three process types: nebula-graphd, nebula-metad, and nebula-storaged. They can be named and started in the same manner, so the common role can be applied to these processes.

By using this mechanism, the maintenance gets easier and the services can be more fine-grained. For example, if we want to deploy the nebula-storaged process on machine A, B, and C and to deploy the nebula-graphd process on machine C only, the nebula-graphd configuration file will be distributed to machine C, but not machine A and B.

```plain
# install/task/main.yml, a common role. Variables are used. 
- name: config {{ module }}.conf
  template:
    src: "{{ playbook_dir}}/templates/{{ module }}.conf.j2"
    dest: "{{ deploy_dir }}/etc/{{ module }}.conf"
    
# nebula-graphd/task/main.yml, the graphd role. A value is assigned to the variable.
- name: install graphd
  include_role:
    name: install
  vars:
    module: nebula-graphd
```

In the playbook, the machine group for the nebula-graphd process runs the graphd role. For example, no nebula-graphd process is deployed on machine A and B, so its configuration file will not be passed to them.

When a NebulaGraph cluster is deployed in this way, you cannot run`nebula.service start all`to start it, because some machines do not have the`nebula-graphd.conf`file. Similarly, we can use parameters in a playbook to define machine groups and to pass the parameters.

```plain
# playbook start.yml
- hosts: metad
  roles:
    - op
  vars:
    - module: metad
    - op: start
- hosts: storaged
  roles:
    - op
  vars:
    - module: storaged
    - op: start
- hosts: graphd
  roles:
    - op
  vars:
    - module: graphd
    - op: start

```

It works like running the startup script over SSH multiple times. The  execution efficiency is not as high as`start all`, but the services can be started and stopped more flexibly.
Using the common role

![ansible demo](https://user-images.githubusercontent.com/38887077/120771106-715ee780-c551-11eb-9ce5-b381503f09d2.png)

## Using vars_prompt to End Playbook

If we want to delete the binary files but not to delete the data directory, we can add`vars_prompt`in the`remove`playbook to double confirm the task. If the task is double confirmed, the data will be deleted and the playbook will be ended.

```plain
# playbook remove.yml
- hosts: all
  vars_prompt:
    - name: confirmed
      prompt: "Are you sure you want to remove the Nebula-Graph? Will delete binary only  (yes/no)"
 
  roles:
    - remove
```

However, in the role, the value for the double confirmation will be verified.

```plain
# remove/task/main.yml
---
- name: Information
  debug:
    msg: "Must input 'yes', abort the playbook "
  when:
    - confirmed != 'yes'
- meta: end_play
  when:
    - confirmed != 'yes
```

The playbook will be executed as shown in the following figure. In the double confirmation step, if no 'yes' is input, the playbook will be aborted, which will delete the binary files only but not the data of the NebulaGraph cluster.

![final result](https://user-images.githubusercontent.com/38887077/120771759-285b6300-c552-11eb-9b00-2c10de1eb80f.png)


This post shares with you a way to deploy nebula graph cluster with ansible.

Using Ansible to Automate Deployment of NebulaGraph Cluster


![Step by Step Tutorial: From Data Preprocessing to Using Graph Database](https://user-images.githubusercontent.com/57335825/117272063-626f1180-ae8d-11eb-9a53-d81b89a61fd9.png)

This article is contributed by Jiayi98, a NebulaGraph user. She shared her experience in deploying NebulaGraph offline and preprocessing a dataset provided by LDBC. It is a beginner-friendly step-by-step guide to learn NebulaGraph.

This is not standard stress testing, but a small-scale test. Through this test, I got familiar with the deployment of NebulaGraph, its data import tool, its graph query language, Java API, and data migration. Additionally, now I have a basic understanding of its cluster performance.

# Preparation

Internet connection is necessary for the following preparations.

1. Download an RPM file of Docker: [https://docs.docker.com/engine/install/centos/#install-from-a-package](https://docs.docker.com/engine/install/centos/#install-from-a-package)
2. Download a TAR file of Docker Compose: [https://github.com/docker/compose/releases](https://github.com/docker/compose/releases)
3. Pull the following images from [https://hub.docker.com/search?q=vesoft&type=image](https://hub.docker.com/search?q=vesoft&type=image) and run docker save `image name` to save them to tar archives: nebula-metad, nebula-graphd, nebula-storaged, nebula-console, nebula-graph-studio, nebula-http-gateway, nebula-http-client, nginx, and nebula-importer.
4. Copy and modify the YAML file from [https://github.com/vesoft-inc/nebula-docker-compose/blob/docker-swarm/docker-stack.yaml](https://github.com/vesoft-inc/nebula-docker-compose/blob/docker-swarm/docker-stack.yaml)
5. On the nebula-graph-studio GitHub page ([https://github.com/vesoft-inc/nebula-web-docker](https://github.com/vesoft-inc/nebula-web-docker)), download its RPM file.

# Installation

1. Install Docker.

```
$ rpm -ivh <rpm package>
$ systemctl start docker --Starts Docker
$ systemctl status docker --Views Docker status
```

2. Install Docker Compose.

```
$ mv docker-compose /usr/local/bin/ -- Moves Docker Compose file to /usr/local/bin
$ chmod a+x /usr/local/bin/docker-compose --Modifies the file permissions
$ docker-compose -version
```

3. Import the images.

```
$ docker load <tar archives of the images>
$ docker image ls
```

4. On the manager node, run the following command to initialize the Docker Swarm cluster.

```
$ sudo docker swarm init --advertise-addr <manager machine ip>
```

5. According to the prompt, on another machine, join the swarm as a worker node.

```
$ docker node ls
```

* The following error may occur when a worker node joins a swarm.

`Error response from daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.16.9.129:2377: connect: no route to host"`

You can try disabling the firewall as follows to solve this problem.

```
$ systemctl status firewalld.service
$ systemctl disable firewalld.service
```

6. On the manager node, modify `docker-stack.yml` and create `nebula.env`.

```
-- nebula.env
TZ=UTC
USER=root
```

* In the YAML file, the hostnames of machines must be different. If errors occur during the startup, please check your YAML file, which should be blamed for most errors. If you want to upgrade NebulaGraph from v1 to v2, replacing the images in the YAML file is enough.
 
7. On the manager node, deploy a NebulaGraph stack.

```
$ docker stack deploy <stack name> -c docker-stack.yml
```

Here is how I debugged the deployment:

```
$ docker service ls --Views service status
$ docker service ps <NAME/ID> --Lists the tasks of a specified service
$ docker stack ps --no-trunc <stack name> --Lists the tasks in the stack
```

8. Install NebulaGraph Studio.

The source code in the folder is for NebulaGraph v1. If you are using NebulaGraph v2, find the source code in the subfolder v2.

```
$ cd nebula-web-docker
```

OR

```
$ cd nebula-graph-studio/v2
$ docker-compose up -d --Builds and starts the Studio service.
```

In the command, `-d` is added to run the container for the service in the background.

When the service starts, in the browser address bar, type `http://ip address:7001`

# Test

The dataset in this test is provided by LDBC.

### Prepare

1. Pull the source code from [https://github.com/ldbc/ldbc_snb_datagen/tree/stable](https://github.com/ldbc/ldbc_snb_datagen/tree/stable).To generate data for scale factor 1-1000, use the stable branch.
2. Download hadoop-3.2.1.tar.gz from [http://archive.apache.org/dist/hadoop/core/hadoop-3.2.1/](http://archive.apache.org/dist/hadoop/core/hadoop-3.2.1/).
3. Preprocess the LDBC dataset.

### Preprocess LDBC Dataset

**Please make sure that the NebulaGraph version that you are using supports `“｜”` as separator.**

For an LDBC dataset, the IDs and indexes of the vertices and edges are not compatible with those in NebulaGraph. The vertex IDs must be processed to be unique keys.

In my case, a prefix was used for each vertex ID. For example, for a person vertex, a `p` was added to change the original ID `933` to `p933`. To try my CDH, I used Spark to preprocess the data and stored the data on HDFS for importing them into NebulaGraph with Nebula Exchange. 

### Hardware Specifications

![Hardware Specifications](https://user-images.githubusercontent.com/57335825/117277279-52a5fc00-ae92-11eb-81ca-eee50e00819a.png)

NOTE: An HDD is not recommended for NebulaGraph. However, I do not have an SSD. The test result proved that HDDs perform badly.

### Service Distribution

Three nodes for the services:
- 192.168.1.10: meta, storage
- 192.168.1.12: graph, meta, storage
- 192.168.1.60: graph, meta, storage

Two graph spaces were created: 
1. csv: With 10 partitions
    1. Original data: About 42 MB
    2. More than 7,000 vertices and 400 thousand edges
2. test: With 100 partitions
    1. Original data: About 73 GB
    2. More than 282 million (282,612,309) vertices and 1.10 billion (1,101,535,334) edges

When the data was imported to NebulaGraph, about 76 GB storage space was occupied, of which about 2.2 GB was occupied by WAL files.

I did not do a test on data import. Some data was imported with Nebula Importer, and the rest was imported with Nebula Exchange.

![Data Import Results](https://user-images.githubusercontent.com/57335825/117277827-c8aa6300-ae92-11eb-9cbc-ad61f5246cbf.png)

# Do a Test

How to do the test:

1. Choose 1,000 vertices and obtain the average response time of 1,000 queries.

![Testing Results](https://user-images.githubusercontent.com/57335825/117278816-ad8c2300-ae93-11eb-8c3c-a34ebc005afe.png)

* In the three-hop test, it was detected as "Timeout" because I set the timeout parameter to 120 seconds. Later, I performed a three-hop query on the terminal and found more than 300 seconds were needed.

I really hope this article could do some help to those who are new to NebulaGraph. I am grateful for all the technical support from the community and the NebulaGraph team.

NebulaGraph is really supportive of its users' attempts to learn it. I have gained a lot in the learning process.


This article is contributed by Jiayi98, a NebulaGraph user. She shared her experience in deploying NebulaGraph offline and preprocessing a dataset provided by LDBC.

Step by Step Tutorial: From Data Preprocessing to Using Graph Database


![Nebula Operator Overview](https://user-images.githubusercontent.com/57335825/116524538-16aaee00-a90a-11eb-9299-bcd025af1609.png)

Before introducing Nebula Operator, you should know what the Operator is.

An Operator aims to package, deploy, and manage Kubernetes applications. By extending the functionality of the Kubernetes API, it creates, configures, and manages complex application instances on behalf of a user. It is built based on the Custom Resource Definition (CRD) and its associated Controller, covering domain or application-specific knowledge to automate the entire life cycle of the software it manages.

In Kubernetes, controllers of the control plane implement control loops that repeatedly compare the desired state of a cluster with its actual state. If the actual state does not match the desired state, the controller continues managing internal business logic until the desired state is achieved.

![reconcile loop](https://user-images.githubusercontent.com/57335825/116524709-45c15f80-a90a-11eb-87a1-447527d07f7b.png)

Nebula Operator abstracts deployment management of NebulaGraph as CRD. By combining multiple built-in API objects including `StatefulSet`, `Service`, and `ConfigMap`, the routine management and maintenance of NebulaGraph are coded as a control loop. When a CR instance is submitted, Nebula Operator drives the database cluster to the final state according to the control process.

# Functionalities of Nebula Operator

## Custom Resource Definition

Here is the CRD file for the deployment of a NebulaGraph cluster. From it, you can see the core functionalities of Nebula Operator.

```yaml
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
  namespace: default
spec:
  graphd:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
    replicas: 1
    image: vesoft/nebula-graphd
    version: v2.0.0
    storageClaim:
      resources:
        requests:
          storage: 2Gi
      storageClassName: gp2
  metad:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
    replicas: 1
    image: vesoft/nebula-metad
    version: v2.0.0
    storageClaim:
      resources:
        requests:
          storage: 2Gi
      storageClassName: gp2
  storaged:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
    replicas: 3
    image: vesoft/nebula-storaged
    version: v2.0.0
    storageClaim:
      resources:
        requests:
          storage: 2Gi
      storageClassName: gp2
  reference:
    name: statefulsets.apps
    version: v1
  schedulerName: default-scheduler
  imagePullPolicy: IfNotPresent
```

In the spec section, three items must be focused, that is, graphd, metad, and storaged. They represent Graph Service, Meta Service, and Storage Service respectively. The controllers will check the state of StatefulSet, Service, and ConfigMap in sequence in the control loop. If a dependent API object is not created or NebulaGraph component service regulation is abnormal, the control loop will be terminated and wait for the next one. This process is repeated again and again.

At present, only the essential configuration parameters for the operation are provided as CRDs, including resources, replicas, image, and schedulerName. In the future, more configuration parameters will be provided for Nebula Operator to meet your scenarios.

![reconcile](https://user-images.githubusercontent.com/57335825/116524916-88833780-a90a-11eb-8f75-c1f6227c8da6.png)

## Scaling

Scaling out Storage is divided into two stages. In the first stage, you need to wait for the status of all newly created Pods to be Ready. In the second stage, the BALANCE DATA command is executed. The execution of BALANCE DATA is defined in the StorageBalancer CRD. Through both stages, the scaling process of the controller replicas is decoupled from the balancing data process to realize the customization of the balancing data task such as adding task execution time and executing it at low traffic. Such an implementation can effectively reduce the impact of data migration on online services, which is in line with NebulaGraph principle: Balancing data is not fully automated and when to balance data is decided by users.

![scale out](https://user-images.githubusercontent.com/57335825/116524993-a05abb80-a90a-11eb-9a8a-91a617d021e4.png)

Scaling in and out Storage are the opposite processes. To scale in Storage, some nodes must be removed safely, which can be completed by running the BALANCE DATA REMOVE $host_list command internally. After the nodes are removed, the size of Storage Pods will be shrunk.

NOTE: The following figures show how Storage is scaled in, but it cannot be used as a deployment reference. In a high-availability scenario, three online replicas are necessary.

![scale in](https://user-images.githubusercontent.com/57335825/116525050-b5cfe580-a90a-11eb-9a8a-ab869655a478.png)

![scale in](https://user-images.githubusercontent.com/57335825/116525103-c41e0180-a90a-11eb-8952-6c90e9c01628.png)

## Balance Scheduling

For scheduling, Nebula Operator provides two options: the default scheduler and the custom scheduler based on the scheduler extender interface.

Topology spread constraints of the default scheduler can be used to spread Pods evenly across your cluster. By default, the key of node label is set to kubernetes.io/hostname in Nebula Operator. It will be configurable in the future. The affinity-based scheduling policies are not applied because such policies control how Pods are packed or scattered.  "PodAffinity" is to pack multiple Pods into a specific topology domain, which is called "packed", while "PodAntiAffinity" is to ensure that only one Pod can be scheduled into a single topology domain, which is called "scattered". However, these scheduling policies cannot meet the scenarios that require Pods to be spread as evenly as possible, especially in a distributed scenario, where high-availability needs to be spread across multiple topology domains.

Of course, if you are using an earlier version of Kubernetes and cannot experience the topology spread constraints feature, you can use Nebula-Scheduler. Its core logic is to ensure that the Pods of each component can be evenly spread across the specified topology domains.

## Workload Controllers

Nebula Operator is going to support a variety of workload controllers. By configuring reference, you can choose a workload controller. Currently, besides the native StatefulSet, Nebula Operator supports the AdvancedStatefulSet feature provided by Community Edition of OpenKruise. Users can use the advanced features of Nebula Operator such as in-place upgrade and designating children nodes for a specific node to meet their own business needs. Of course, parameter configurations in Operator are necessary to enable these features. Currently, only parameters for in-place upgrade are available.

```yaml
  reference:
    name: statefulsets.apps
    version: v1
```  

## Other Functionalities

Other functionalities such as WebHook supporting configuration verification in high-availability mode and updating configuration of custom parameters are not introduced in this article. These features aim to allow Nebula Operator to manage NebulaGraph clusters more securely and conveniently on behalf of you. For more information, refer to the documentation on GitHub.

# FAQ

> Can Nebula Operator be used outside of Kubernetes?

No. Operator depends on Kubernetes. It is an extension of the Kubernetes API. It is a K8s tool.

> How to ensure the stability and availability of upgrade and scaling? If the upgrade or scaling fails, can a cluster roll back? 

We recommend that you back up your data before the operation to make sure that your cluster can roll back after a failure. At present, Nebula Operator does not support backing up data before the operation, but it will do after iterations. 

> Can I use Nebula Operator to manage a NebulaGraph v1.x cluster?

No. NebulaGraph v1.x does not support internal domain name resolution. Nebula Operator needs this feature, so it is not compatible.

> Can the use of local storage ensure the stability of a cluster?

No guarantee now. If local storage is used, it means that Pods bind specific nodes, and currently, the Operator cannot fail over after a local storage node goes down. However, using network storage does not have such a problem.

> When can Nebula Operator support the upgrade? 

Nebula Operator needs to be in line with NebulaGraph. When NebulaGraph supports rolling upgrade, Nebula Operator will have this feature. 

Nebula Operator is now open sourced and can be accessed via GitHub: [https://github.com/vesoft-inc/nebula-operator](https://github.com/vesoft-inc/nebula-operator).

Interested in Nebula Operator? Join the NebulaGraph [Slack channel](https://join.slack.com/t/nebulagraph/shared_invite/zt-7ybejuqa-NCZBroh~PCh66d9kOQj45g) to discuss with the community!


Nebula Operator is a plug-in to deploy, operate and maintain NebulaGraph automatically on K8s， which makes NebulaGraph a real cloud-native graph database.

Nebula Operator Overview: Automated Operation on Kubernetes


![How to Deploy NebulaGraph on Kubernetes](https://user-images.githubusercontent.com/57335825/83609181-bf9e6c80-a532-11ea-8f6d-8132efb66198.png)

# What is Kubernetes

Kubernetes is an open-source platform for managing containerized applications on multiple hosts on a cloud platform. Kubernetes is designed to enable users to deploy containerized applications simply and efficiently. It provides users with a set of mechanisms to deploy, plan, update, and maintain their applications.

Architecturally, Kubernetes provides a series of building blocks for users to deploy, maintain, and scale applications. The components that make up Kubernetes are loosely coupled and scalable, aiming to support a variety of workloads. Scalability is primarily enabled by the Kubernetes APIs, which are primarily used as internal components for scaling and as containers running on Kubernetes.

![Kubernetes API](https://oscimg.oschina.net/oscnet/up-69f689e312c8968a3cbef1a5cb7d2075f0d.png)

Kubernetes consists mainly of the following core components:

- `etcd`. Stores the states of the entire cluster.
- `kube-apiserver`. Provides the only entry for operating the resources, and provides operations of validation, authentication, access control, and API registry and finding.
- `kube-controller-manager`. Watches the state of a cluster, for example, fault detection, autoscaling, and rolling updates.
- `kube-scheduler`. Schedules the resources. The scheduler assigns the Pods to the corresponding nodes according to the pre-defined scheduling policy. 
- `kubelet`. Maintains the life cycle of the containers and manages the Volume and the network.
- `kube-proxy`. Provides Service with the service discovery and load balancing within the cluster.

# Kubernetes and Databases

Database containerization is a big hit right now, so what can Kubernetes bring to databases?

- **Failure recovery**: Kubernetes provides the failure recovery feature. If a database is deployed on Kubernetes, when the application goes down, Kubernetes enables it to restart automatically or migrate to another node in the cluster.
- **Storage management**: Kubernetes supports various storage solutions, so a database deployed on it can transparently use different storage solutions.
- **Load balancing**: Kubernetes Service provides load balancing capabilities, so it can balance external access to different replicas of database instances.
- **Horizontal scalability**: Depending on the resource utilization of the database cluster, Kubernetes can scale the number of replicas to improve resource utilization.

So far, many databases, such as MySQL, MongoDB, and TiDB, work well on Kubernetes clusters.

# NebulaGraph in Practice on Kubernetes

NebulaGraph is a distributed open-source graph database. It has three major components: nebula-graphd for Query Engine, nebula-storaged for data storage, and nebula-metad for metadata. Kubernetes brings the following benefits to NebulaGraph: 

- Kubernetes can balance the loads between different replicas of nebula-graphd, nebula-metad, and nebula-storaged. They can discover each other automatically through the domain service of Kubernetes.
- With StorageClass, users do not need to perceive the design of PVC (PersistentVolumeClaim) and PV (PersistentVolume). Kubernetes can transparently access the local volumes or cloud storage.
- The deployment of a NebulaGraph cluster on Kubernetes takes only seconds and Kubernetes can also enable upgrade of the cluster not perceived by users.
- Kubernetes enables automatic recovery of a NebulaGraph cluster. If a single replica crashes, Kubernetes can automatically recover it without human intervention.
- Kubernetes can scale a NebulaGraph cluster elastically based on the resource utilization to improve its performance.

Now, let me introduce this practice in detail.

## Cluster Deployment

### Software And Hardware Requirements 

These are the specifications of the hardware and operating system involved in this practice: 

- The operation system is CentOS-7.6.1810 x86_64.
- Virtual machine configuration:
  - 4 CPU
  - 8G memory
  - 50G system disk
  - 50G data disk A
  - 50G data disk B
- Kubernetes version: v1.14 or later
- NebulaGraph version: v2.0.0-rc1
- Data storage: Local PV
- CoreDNS version: 1.6.0 or later

### Planning of the Cluster

This table lists how the cluster is composed. 

| Server IP | Nebula Services | Role |
| --- | --- | --- |
| 192.168.0.1 |  | master |
| 192.168.0.2 | graphd, metad-0, storaged-0 | node |
| 192.168.0.3 | graphd, metad-1, storaged-1 | node |
| 192.168.0.4 | graphd, metad-2, storaged-2 | node |

### Components to Be Deployed

- Helm 3
- Local volume and the plugin for the local volume
- NebulaGraph

## Install Helm 3

Helm is a package manager for Kubernetes. Helm can ease the deployment of an application on Kubernetes. I will not cover the details of Helm in this article. If you are interested, please refer to [Quickstart Guide of Helm](https://www.hi-linux.com/posts/21466.html). In this practice, Helm 3 is used.

1. Download and Install Helm 3: Open a terminal and run these lines.

```bash
$ wget https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz
$ tar -zxvf helm/helm-v3.5.2-linux-amd64.tgz
$ mv linux-amd64/helm /usr/bin/helm
```

2. View the Helm Version

To view the version of Helm, run `helm version`. In this example, the following line is returned. 

```cpp
version.BuildInfo{Version:"v3.5.2", 
GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", 
GitTreeState:"dirty", GoVersion:"go1.15.7"}
}
```

## Configure the Local Volume

On each server, complete these configurations: 

1. Create a mount point named `/mnt/disks`.

```bash
$ sudo mkdir -p /mnt/disks
```

2. Format the data disks. 

```bash
$ sudo mkfs.ext4 /dev/diskA 
$ sudo mkfs.ext4 /dev/diskB
```

3. Mount the data disks on the mount point.

```bash
$ DISKA_UUID=$(blkid -s UUID -o value /dev/diskA) 
$ DISKB_UUID=$(blkid -s UUID -o value /dev/diskB) 
$ sudo mkdir /mnt/disks/$DISKA_UUID
$ sudo mkdir /mnt/disks/$DISKB_UUID
$ sudo mount -t ext4 /dev/diskA /mnt/disks/$DISKA_UUID
$ sudo mount -t ext4 /dev/diskB /mnt/disks/$DISKB_UUID

$ echo UUID=`sudo blkid -s UUID -o value /dev/diskA` /mnt/disks/$DISKA_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab
$ echo UUID=`sudo blkid -s UUID -o value /dev/diskB` /mnt/disks/$DISKB_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab
```

4. Install the plugin for the local volume. 

```bash
$ curl https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/archive/v2.4.0.zip
$ unzip v2.4.0.zip
```

5. Modify the `classes` section in `v2.4.0/helm/provisioner/values.yaml`: Replace `hostDir: /mnt/fast-disks` with `hostDir: /mnt/disks` and delete the `#` from `# storageClass: true`. And then run the following command:

```bash
$ helm install local-static-provisioner --namespace default sig-storage-local-static-provisioner/helm/provisioner

# View the deployment of local-static-provisioner
$ helm list
NAME                        NAMESPACE   REVISION    UPDATED                                 STATUS      CHART               APP VERSION
local-volume-provisioner    default     1           2021-02-10 11:06:34.3540341 +0800 CST   deployed    provisioner-2.4.0   2.4.0
```

## Deploy a NebulaGraph Cluster

### Download nebula-charts

```bash
# Download nebula-charts
$ helm repo add nebula-charts https://vesoft-inc.github.io/nebula-docker-compose/charts
$ helm pull nebula-charts/nebula
$ tar -zxvf nebula-0.2.0.tgz
```

### Set Up the Kubernetes Nodes

The following table lists all the nodes of the Kubernetes cluster. Some nodes must be labeled for scheduling. In this example, I labeled `192.168.0.2`, `192.168.0.3`, and `192.168.0.4` with `nebula: "cloud"`.

| Server IP | kubernetes roles | nodeName |
| --- | --- | --- |
| 192.168.0.1 | master | 192.168.0.1 |
| 192.168.0.2 | node | 192.168.0.2 |
| 192.168.0.3 | node | 192.168.0.3 |
| 192.168.0.4 | node | 192.168.0.4 |

The commands are as follows.

```bash
$ kubectl  label node 192.168.0.2 nebula="cloud" --overwrite 
$ kubectl  label node 192.168.0.3 nebula="cloud" --overwrite
$ kubectl  label node 192.168.0.4 nebula="cloud" --overwrite
```

### Change Default Values

Here is the hierarchy of the nebula-charts directory. 

```bash
master/kubernetes/
└── helm
    ├── Chart.yaml
    ├── templates
    │   ├── configmap.yaml
    │   ├── deployment.yaml
    │   ├── _helpers.tpl
    │   ├── NOTES.txt
    │   ├── pdb.yaml
    │   ├── serviceaccount.yaml
    │   ├── service.yaml
    │   └── statefulset.yaml
    └── values.yaml

1 directory, 11 files
```
You can change the default values in the `charts/nebula/values.yaml` file to meet your requirements.

### Install NebulaGraph with Helm

```bash
$ helm install nebula charts/nebula 
# View the deployment status
$ helm status nebula
NAME: nebula
LAST DEPLOYED: Fri Feb 19 12:58:16 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
NebulaGraph Cluster installed!

1. Watch all containers come up.
  $ kubectl get pods --namespace=default -l app.kubernetes.io=nebula -w
# View the status of the NebulaGraph cluster deployed on K8s
$ kubectl get pods --namespace=default -l app.kubernetes.io=nebula
NAME                             READY   STATUS    RESTARTS   AGE
nebula-graphd-676cfcf797-4q7mk   1/1     Running   0          6m
nebula-graphd-676cfcf797-whwqp   1/1     Running   0          6m
nebula-graphd-676cfcf797-zn5l6   1/1     Running   0          6m
nebula-metad-0                   1/1     Running   0          6m
nebula-metad-1                   1/1     Running   0          6m
nebula-metad-2                   1/1     Running   0          6m
nebula-storaged-0                1/1     Running   0          6m
nebula-storaged-1                1/1     Running   0          6m
nebula-storaged-2                1/1     Running   0          6m

```

#### Connect to the Graph Service

```bash
$ kubectl get service nebula-graphd
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                          AGE
nebula-graphd   NodePort   10.105.47.116   <none>        9669:31646/TCP,19669:30554/TCP,19670:32386/TCP   22m

# Use nebula-console to test the Graph service.
$ docker run --rm -ti --entrypoint=/bin/sh vesoft/nebula-console:v2-nightly

# Connect to the Graph service with the NodePort mode
/ $ nebula-console -addr 192.168.0.4 -port 31646 -u root -p vesoft
2021/02/19 05:04:55 [INFO] connection pool is initialized successfully

Welcome to NebulaGraph v2.0.0-rc1!

(root@nebula) [(none)]> show hosts;
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| Host                                                          | Port | Status   | Leader count | Leader distribution  | Partition distribution |
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-0.nebula-storaged.default.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-1.nebula-storaged.default.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "nebula-storaged-2.nebula-storaged.default.svc.cluster.local" | 9779 | "ONLINE" | 0            | "No valid partition" | "No valid partition"   |
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
| "Total"                                                       |      |          | 0            |                      |                        |
+---------------------------------------------------------------+------+----------+--------------+----------------------+------------------------+
Got 4 rows (time spent 2608/4258 us)
```

# FAQ

> How to create a Kubernetes cluster?

To create a highly-available Kubernetes cluster, refer to the docs: [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/).

> How to adjust the parameters for deploying a NebulaGraph cluster?

Use `--set` in the `helm install` command to overwrite the variables in the `values.yaml` file under the `nebula-charts` directory. For more information, refer to [https://helm.sh/docs/intro/using_helm/](https://helm.sh/docs/intro/using_helm/)

> Can I follow this process to deploy NebulaGraph v1.0.0+ on Kubernetes?

NebulaGraph v1.0.0+ does not support parsing internal domain names. To deploy a cluster of NebulaGraph v1.0.0+, you must modify the `charts/nebula/values.yaml` file as follows:

```bash
hostNetwork: true
metadEndpoints: []
  - 192.168.0.2:9559
  - 192.168.0.3:9559
  - 192.168.0.4:9559
```

> How to access the internal components of NebulaGraph from the outside of a K8s cluster?

In this example, the Graph service is accessed via the NodePort mode. You can also access it via the hostPort, hostNetwork, Ingress, or LoadBalancer mode. You can choose the appropriate option to meet your environment requirement.

> How to view the status of the deployed NebulaGraph cluster?

Run the `kubectl get pods --namespace=default -l app.kubernetes.io=nebula` command. Or use Kubernetes Dashboard to view the status of the cluster.

> How to use other types of storage solutions?

Refer to [https://kubernetes.io/zh/docs/concepts/storage/storage-classes/](https://kubernetes.io/zh/docs/concepts/storage/storage-classes/)

# You might also like

1. [NebulaGraph Architecture — A Bird’s View](https://nebula-graph.io/posts/nebula-graph-architecture-overview/)
1. [An Introduction to NebulaGraph's Storage Engine](https://nebula-graph.io/posts/nebula-graph-storage-engine-overview/)
1. [An Introduction to NebulaGraph’s Query Engine](https://nebula-graph.io/posts/nebula-graph-query-engine-overview/)

This post gives a brief introduction of what Kubernetes is and provides a step-by-step guide to deploying NebulaGraph on Kubernetes, aka k8s.

Tag