How could we model data in Tabular sources and ETL it to NebulaGraph? This article demonstrates an end-to-end example of doing so with dbt.

Graph Data Modeling and ETL with dbt in NebulaGraph Database

This article was written by a NebulaGraph community contributor *xrfinbj*. It introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

# Background

The company StoneWise had internal needs of using graph databases and the NebulaGraph database was confirmed to use through technical selection. We still needed to verify the query performance of the NebulaGraph database in real business scenarios. Therefore, we urgently needed to import data to NebulaGraph and validate the performance. We found that the documentation of importing data to NebulaGraph from Hive with Exchange was not very complete, so I decided to record the pitfalls we stepped in this process to avoid detours and traps.


This article was written based on NebulaGraph Documentions: [Import data from Hive](https://docs.nebula-graph.io/3.3.0/nebula-exchange/use-exchange/ex-ug-import-from-hive/)


# Environment

- NebulaGraph: nightly
- Deployment: Docker on Mac
- Hardware:
  - Disk: SSD
  - CPU、Memory: 16 G
- Data warehouse environment (Local data warehouse on Mac)
  - Hive 3.1.2
  - Hadoop 3.2.1
- Exchange: https://github.com/vesoft-inc/nebula-java/tree/v1.0/tools/exchange （Compile and generate a JAR package）
- Spark：spark-2.4.7-bin-hadoop2.7 (Configure `core-site.xml`, `hdfs-site.xml`, `hive-site.xml` in the `$HADOOP_HOME/etc/hadoop` directory, and configure `spark-env.sh` in the `$SPARK_HOME/conf` directory.)
- Scala code runner version 2.13.3 – Copyright 2002-2020, LAMP/EPFL and Lightbend, Inc.

# Configuration

1. NebulaGraph DDL

![ ](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/1.png)

2. Hive DDL
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/2.png)

3. The latest `nebula_application.conf` file.
Pay attention to these field mappings: `exec`, `fields`, `nebula.fields`, `vertex`, `source`, and `target`.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/3.png)
# Starting the Import

1. Make sure that the NebulaGraph service is started.
2. Make sure that Hive tables and data are ready.
3. Run `spark-sql cli` to see if the Hive tables and data are ready to ensure the Spark environment is OK.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/4.jpeg)

4. When the configuration is ready, execute Spark commands.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/5.png)
5. After successful import, you can check the amount of data imported with the db_dump tool.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/6.png)

# Pitfalls and Notes

- The first pitfall is that the spark-submit command does not add the `-h` option.

- The tagName in NebulaGraph is case-sensitive. the name in the configuration of tags should be the tag name of NebulaGraph.

- The Hive data type `int` and NebulaGraph `int` are not consistent. The Hive `bigint` corresponds to the NebulaGraph `int`.


## Other Notes

- INSERT operations have better performance than UPDATE because the underlying storage of NebulaGraph is KV Store. Repeated insertions are overwriting the data. 

- In addition to the information in the documentation, I also read the source code and made posts on the forum.

- For suggestions and recommendations on importing data and data compaction, see https://docs.nebula-graph.com.cn/manual-CN/3.build-develop-and-administration/5.storage-service-administration/compact/.

- I have verified the following two scenarios:

  - Importing data from Hive 2 (Hadoop 2) to NebulaGraph with Spark 2.4

  - Importing data from Hive 3 (Hadoop 3) to NebulaGraph with Spark 2.4

Note: Exchange does not support Spark 3 yet, and it reports an error after compiling, so it is hard to verify the Spark 3 environment.

There are still some questions.

- How to set the parameters `batch` and `rate.limit` in the `nebula_application.conf` file? 

- The principle of using Exchange to import data from Hive (I've started learning Spark recently.)

# Exchange Source Code Debugging

For information about Spark debugging, see https://dzone.com/articles/how-to-attach-a-debugger-to-apache-spark.

Through the Exchange source code learning and debugging, I found that only in conjunction with the source code can I understood the descriptions about importing SST files and the Download and Ingest in the document.

Debugging the source code also helped me find some configuration problems.

Step 1:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/7.png)
Step 2:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/8.png)
Step 3: IDEA configuration

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/9.png)

Step 4: Click `Debug` in IDEA
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Hive-Data-Import%20with%20NebulaGraph%20Exchange/10.png
)
# Suggestions and Gratitude

Thanks to NebulaGraph database, which can solve many practical problems in the business. The problems I met are nothing compared to the problems met by others. The problems encountered in the middle of the process received timely feedback from the community.

This blog introduces the process and notes of importing data from Hive to NebulaGraph with NebulaGraph Exchange. 

Hands-on Tutorial for Hive-Data-Import with NebulaGraph Exchange

NebulaGraph VEditor is a high-performance, highly customizable WYSIWYG visualization editing front-end library. NebulaGraph VEditor is based on SVG drawing, and it is easy to develop and customize the drawing by reasonably abstracting the code structure. It is extremely suitable for WYSIWYG editing and preview scenarios such as approval flow, workflow, kinship, ETL processing, and graph query. You can use VEditor to visually query, edit, and model graphs. 

After continuous iterations and polishing, VEditor has been relatively perfect and now its code has been open-sourced. In this topic, I'd like to share some ideas and thoughts about its design.

# Basic Features

- Customizable node and edge styles.
- Flat, simple, and clear code structure.
- Small map and magnetic lines.
- Shortcuts for common operations.
- History management.
- Lightweight, only 160 KB in size before compression.

# Design Ideas


When I first came across the graph editor library, I just wanted to find one for customization and animation. After using many flowchart libraries, I found that most of the designs have bloated interfaces and complex classes. As a front-end developer, it was against my philosophy of writing simple, streamlined, and low-coupling code. So I decided to write a lightweight library myself to meet my needs.

The design concept of VEditor is to make it lighter for developers to use, reduce learning APIs, and reduce reliance on various libraries based on customizability and comprehensibility.

# Architecture

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/VEditor/1.png)

The overall architecture is mainly through events to do the dependency management between entities, it is also recommended to get the state changes of the whole flowchart mainly through events.

Among them, the rendering process is semi-automatic rendering, and the rendering needs to be triggered manually after changing the flowchart data. Other state operations on the canvas will trigger the user-defined shape-rendering function to render nodes and lines. 

# Rendering Process

VEditor uses SVG to render the canvas. The declarative use of SVG makes the internal structure visible externally, which is convenient for developers to customize the rendering. Users can directly rewrite the relevant SVG styles outside and can directly use the SVG DOM (Document Object Model) to manipulate mouse events and animate nodes.

In terms of shape rendering, user-defined rendering functions are registered mainly through the exposed Shape interface. From this perspective, VEditor can perform rendering based on any rendering technique, as long as the rendering interface returns an SVGDOM, which can be an SVGElement or ForeignObject. I highly recommend managing SVC rendering through the DOM framework, such as React and Vue. In some cases, you can even wrap a Canvas to render WebGL nodes, which greatly extends the customizability.

In addition to nodes, node anchors and lines also support the object rendering after the corresponding interface is registered as Shape. In actual business scenarios, we use this feature to implement dynamic addition, deletion, and change of algorithm parameter anchors, the input and output anchors of OLTP queries (figure 1), filtering of edges in graph visualization, and step rendering (figure 2). You are welcome to apply for Explorer free trial. Just click [here](https://nebula-graph.com.cn/products/explorer/).

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/VEditor/2.png)

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/VEditor/3.png)

# Data Structure Design

The data structure of VEditor is similar to most libraries, but it will not destroy the user's object reference, that is, when the user mounts the relevant data to the node or line object, it will be aligned and reserved, so that it will be convenient for the user to mount the relevant data to vertices after vertices or edges are configured. Therefore, the Redo, Undo, etc. operations of the history will store the user's data as snapshots.

# Performance Design

It is well known that SVG has a much worse performance than Canvas for rendering at small resolutions, which is a disadvantage that comes with increased ease of use. This is especially obvious when initializing a large number of more complex or animated nodes. To address this situation, the data rendering part of VEditor uses an asynchronous process, putting the rendering of anchors into the next event loop to avoid the browser-forced redrawing caused by the large number of bounding boxes fetched during the synchronization process. After the drawing is finished, the corresponding nodes are cached to avoid repeated fetching.

When adding nodes or lines, the DOM feature of SVG will let the browser do dirty rendering automatically, so the performance of incremental rendering is not much different from Canvas, but mainly slower when interacting and animating, which leads to a lot of DOM redrawing. The current design performance target is 1000 nodes with complex shapes for smooth rendering, which is easy to achieve in flowchart editing scenarios.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/VEditor/4.jpeg)


# Interaction Design

VEdtior provides Dagre-based directed graph layouts by default. It automatically centers all nodes after Dagre is optimized and called. At the same time, it provides an adaptive size function. Unlike other libraries, the coordinates of the current nodes will be reset to the adaptive position, and the adaptive position can be restored directly after the user saves the current data.

VEditor's mini-map adopts canvg rendering, which directly converts SVG to Canvas, which can guarantee the accuracy of mini-map and reduce the performance loss at the same time. In terms of interaction, VEditor allows you to zoom in and out of the canvas, and drag and drop elements.

# Future Plans

VEditor is expected to be a data editor and renderer for any field by expanding the scenarios where it can be used. In addition, we will continue to improve the performance and user experience of VEditor while supporting the following features in the future. 

- Box selector and multi-select operations.

- Undirected graphs and double arrows

- Further performance optimizations

👉 GitHub open source address: [https://github.com/vesoft-inc/nebulagraph-veditor](https://github.com/vesoft-inc/nebulagraph-veditor)

# You Might Also Like
[Graph Database Visualization: NebulaGraph Explorer](https://www.nebula-graph.io/products/explorer)


NebulaGraph VEditor is a high-performance, highly customizable WYSIWYG visualization editing front-end library.

Meet NebulaGraph VEditor, an Open-Source Graph Editing Library


# What is geospatial data?

Geospatial data is information related to geospatial entities, such as points, lines, and polygons.

NebulaGraph 2.6 supports geospatial data. You can store, compute, and retrieve geospatial data in NebulaGraph. Geography is a data type supported in NebulaGraph. It is composed of latitude and longitude that represents geospatial data.

## How to use geospatial data in NebulaGraph?

**Create Schema**

The following example shows how to create tags. You can create edge types in the same way.

NebulaGraph currently supports three types of geospatial data: Point, LineString, and Polygon. The following shows how to create geography types and how to insert geospatial data.

```ngql
CREATE TAG any_shape(geo geography);
CREATE TAG only_point(geo geography(point));
CREATE TAG only_linestring(geo geography(linestring));
CREATE TAG only_polygon(geo geography(polygon));
```

When no geography type is specified, it means that you can store data of any type; when a type is specified, it means that only geospatial data of that type can be stored, such as `geography (point)`, which means that you can only store spatial information of points.

**Insert Data**

Insert data in the `geo` column of the `any_shape` tag.

```ngql
INSERT VERTEX any_shape(geo) VALUES "101":(ST_GeogFromText("POINT(120.12 30.16)"));
INSERT VERTEX any_shape(geo) VALUES "102":(ST_GeogFromText("LINESTRING(3 8, 4.7 73.23)"));
INSERT VERTEX any_shape(geo) VALUES "103":(ST_GeogFromText("POLYGON((75.3 45.4, 112.5 53.6, 122.7 25.5, 93.9 28.6, 75.3 45.4))"));
```

Insert data in the `geo` column of the `only_point` tag.

```ngql
INSERT VERTEX only_point(geo) VALUES "201":(ST_Point(120.12，30.16)"));;
```

Insert data in the `geo` column of the `only_linestring` tag.

```ngql
INSERT VERTEX only_linestring(geo) VALUES "302":(ST_GeogFromText("LINESTRING(3 8, 4.7 73.23)"));
```

Insert data in the `geo` column of the `only_polygon` tag.

```ngql
INSERT VERTEX only_polygon(geo) VALUES "403":(ST_GeogFromText("POLYGON((75.3 45.4, 112.5 53.6, 122.7 25.5, 93.9 28.6, 75.3 45.4))"));
```

When the data inserted does not meet the requirements of the specified type, the data insertion fails.

```ngql
(root@nebula) [geo]> INSERT VERTEX only_polygon(geo) VALUES "404":(ST_GeogFromText("POINT((75.3 45.4))"));
[ERROR (-1005)]: Wrong value type: ST_GeogFromText("POINT((75.3 45.4))")
```

We can see that the geospatial data insertion method is rather peculiar, and is very different from the insertion of basic types such as `int`, `string`, and `bool`.

Let's take `ST_GeogFromText("POINT(120.12 30.16)")` as an example, `ST_GeogFromText` is a geographic location information parsing function, which accepts a string type of geographic location data in WKT (Well-Known Text) standard format.

`POINT(120.12 30.16)` represents a geographic point with longitude 120°12′E and latitude 30°16′N; the `ST_GeogFromText` function parses and constructs a geography data object from the WKT parameter, and then the `INSERT` statement stores it in the NebulaGraph in WKB (Well-Known Binary) standard.

## Geospatial functions

The geospatial functions supported by NebulaGraph can be divided into the following main categories:

- Constructing functions

  - `ST_Point(longitude, latitude)`: Constructs a `geography point` object based on a latitude and longitude pair.

- Parsing functions

  - `ST_GeogFromText(wkt_string)`: Parses `geography` objects from the WKT text.

  - `ST_GeogFromWKB(wkb_string)`: Parses `geography` objects from the WKB text. # Not yet supported, because NebulaGraph does not yet support binary strings.

- Format setting functions

  - `ST_AsText(geography)`: Outputs the `geography` object in the WKT text format.

  - `ST_AsBinary(geography)`: Outputs the `geography` object in the WKB text format. # Not yet supported, because NebulaGraph does not yet support binary strings.

- Conversion functions

  - `ST_Centroid(geography)`: Calculates the center of gravity of a `geography` object, which is a `geography point` object.

- The predicate function

  - `ST_Intersects(geography_1, geography_2)`: Determines whether two `geography` objects intersect.

  - `ST_Covers(geography_1, geography_2)`: Determines if the first `geography` object completely covers the second.

  - `ST_CoveredBy(geography_1, geography_2)`: The inverse of ST_Covers.

  - `ST_DWithin(geography_1, geography_2, distance_in_meters)`: Determines if the shortest distance between two `geography` objects is less than the given distance.

- The metric function

  - `ST_Distance(geography_1, geography_2)`: Calculates the distance between two `geography` objects.

These function interfaces follow the OpenGIS Simple Feature Access and ISO SQL/MM standards. For details, see [NebulaGraph doc](https://docs.nebula-graph.io/3.3.0/3.ngql-guide/6.functions-and-expressions/14.geo/).

## Geospatial Index

**What is a geospatial index?**

Geospatial indexes are indexes that can be used to quickly filter data based on the predicate `ST_Intersects` and `ST_Covers` functions. 

NebulaGraph uses the Google S2 library as the geospatial index. 

The S2 library projects the Earth's surface into a tangent square, then recursively quadruples each square surface of the square n times, and uses a space-filling curve, the Hilbert curve, to connect the centers of these small square lattices.

When n is infinitely large, this Hilbert curve almost fills the square.

The S2 library uses a Hilbert curve of order 30.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Exploring-Geospatial-Data-with-NebulaGraph/1.jpeg)

The following figure shows that the Earth is filled with Hilbert curves.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/Exploring-Geospatial-Data-with-NebulaGraph/2.jpeg)


It can be seen that the Earth's surface is divided into cells by these Hilbert curves. For any geographic shape on the earth's surface, such as a city, a river, or a person's location, we can use several of these cells to completely cover the geographic shape.

Each cell is identified by a unique int64 CellID. Thus, the spatial index of a geographic object is the set of S2 cells that are constructed to completely cover the geographic shape.

When constructing an index of a geospatial object, a collection of different S2 cells that completely cover the indexed object is constructed. The indexing query based on spatial predicate functions quickly filters out a large number of irrelevant geographic objects by finding the intersection between the set of S2 cells that cover the queried object and the S2 cells that cover the indexed object.


**Create a geography index**

```ngql
CREATE TAG any_shape_geo_index on any_shape(geo)
```

For geospatial data with the type `point`, it can be represented by an S2 cell of order 30, so a point corresponds to one index entry; for geospatial data with the type `inestring` and `polygon`, we use multiple S2 cells of different levels to cover it, so it will correspond to multiple index entries.

Spatial indexing is used to speed up the lookup of all geo predicates, for example:

```ngql
LOOKUP ON any_shape WHERE ST_Intersects(any_shape.geo, ST_GeogFromText("LINESTRING(3 8, 4.7 73.23)"));
```

When there is no spatial index on the `geo` column of `any_shape`, this statement will first read all the data of `any_shape` into memory and then use it to calculate whether it intersects with the point (3.0, 8.0), which is generally more expensive. When the amount of data in `any_shape` is large, the computation overhead will be unacceptable.

When the `geo` column of `any_shape` has a spatial index, the statement will first use the spatial index to filter out most of the data that intersected by the line, but there will still be some that may be intersected when read into memory, so there is still one more calculation to be done. In this way, the spatial index quickly filters out most of the data that is not likely to intersect at a small cost, and a small percentage is filtered, greatly reducing the computational overhead.

You can store, compute, and retrieve geospatial data in NebulaGraph. Geography is a data type supported in NebulaGraph.

Exploring Geospatial Data with NebulaGraph

Data is significant in our world today. High-growth organizations are reaping the[ benefits of data-driven decisions](https://online.hbs.edu/blog/post/data-driven-decision-making). Just look around and you'll not fail to see that almost any top company you come across is highly dependent on data. Almost every dope product out here is heavily reliant on data, constantly changing for the better and each change driven by data.
At the heart of this enormous data utilization is data processing, one of the core ingredients that helps organizations quickly make sense of data and use it to accomplish the kind of swift progress that we are witnessing. Online data processing, in particular, is critical for any business that depends on real-time operations. The inputting, manipulating, and outputting of data must be efficient and in most cases demands that the output data is produced immediately after the input data has been entered. Online data processing plays a critical role across areas such as online shopping and online banking. In some cases, online data processing is also used to process offline data, such as when a company processes payroll information.
Robust data processing requires powerful systems, and luckily we have super systems in the market. OLTP and OLAP are the two major system types that are unfortunately often mistaken to be the same thing altogether. Let's put OLTP vs OLAP into perspective and help you make the distinction, especially regarding where to use either.



# What is OLTP?
OLTP stands for Online Transaction Processing. In a nutshell, it is a system that processes transactions (inserts, updates, deletes) on data. This system enables several people to execute multiple database transactions in real time over the internet. An OLTP system gathers and stores transactional data in a database. OLTP databases contain multiple records and fields that maintain each transaction.  These types of processing systems have numerous uses in our daily financial and non-financial transactions, including hotel reservations, in-store purchases, ATMs, text messaging, and password changes.  

An example of OLTP at work would be a banking system where you are transferring money from one account to another. The source account would have money subtracted (a debit transaction), and the destination account would have money added (a credit transaction). The OLTP system would then take care of making sure both of those changes are made to the relevant records in the database.

While OLTP systems are typically based on Relational Database Management Systems (RDBMS), which use the Structured Query Language (SQL) to interact with the data, there are also **NoSQL** solutions such as a [graph database](https://www.nebula-graph.io/posts/what-is-a-graph-database) that can be used for OLTP as well. These systems use a different paradigm than RDBMS, but they can still be used to process online transactions.

In general, OLTP systems need to be able to support a high volume of concurrent transactions with low latency. This means that changes need to be made to the data quickly and efficiently in order to keep up with the demands of users. A good example is the [use of graph databases for real-time recommendations](https://www.nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation) where things can change very fast. Consequently, OLTP systems need to be designed carefully in order to meet these performance requirements.

# What is OLAP?

Online analytical processing, or OLAP, is an approach to answering multi-dimensional queries of data. OLAP systems perform multidimensional data analysis on massive amounts of data. This system uses data from a centralized data store, a data mart, or an OLAP data warehouse. An OLAP cube is the most vital component of OLAP systems because it allows users to run queries, analyze and report on [multidimensional data](https://www.geeksforgeeks.org/multidimensional-data-model/). 

The OLAP cube also extends the format of the databases by adding data dimensions and layers. This allows OLAP to perform complex queries that involve multiple columns of data. Decision makers and data analysts can use OLAP to create custom reports that process the data into useful information for decision-making.  OLAP systems have multiple applications in business intelligence, data mining, and complex data analysis. They are also used in business reporting functions, such as sales forecasting, financial analysis, and budgeting.  For example, a sales manager could examine previous years' sales totals by month, quarter and region, looking for trends that might indicate changes in customer behavior.

Data in OLAP databases are organized into dimensions and measures. Dimensions are the attributes of data, such as time, location or product type. Measures are the facts about those attributes, such as sales totals or inventories. Users can interact with dimensions to filter and drill down into measures of interest. For example, a user might select "North America" as a dimension to filter a measure such as "Sales Totals". OLAP tools typically help users to identify patterns and trends that would be difficult to discern from looking at raw data tables.

# OLAP & OLTP in graph databases

Essentially, a graph database can be partitioned into OLAP and OLTP parts. The OLAP part is concerned with offline analysis whereas OLTP is mainly for online processing. 

There are two key frameworks in the OLAP partition: graph computing and graph streaming. Graph computing is used for analyzing data based on graph structures, and this is quite similar in concept to the OLAP in traditional relational databases. The difference is the iterative algorithm, which is not available in the traditional database. For example, the Google PageRank algorithm applies constant iterative computing to determine the relevance of a web page. And so graph databases will be best suited for such a need. The graph streaming framework is basically a mix of basic computing and streaming computing. The latency here is in seconds and the computing is triggered by events.

The OLTP section of the graph database can have many use cases online which are entirely different from the OLAP use cases.

Refer to the illustration below:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OPLTP-vs-OLAP-what-are-the-differences/Figure1.png)


## Examples of systems that support OLTP

Common examples of OLTP systems include.
* **Titan:** Titan is a highly scalable [graph database](https://itcompanies.net/blog/what-is-graph-database) that allows multiple users to access and update a single graph concurrently. 
* **NebulaGraph:** NebulaGraph is an extremely fast open source [graph database](https://nebula-graph.io/) that uses RocksDB as its storage engine - the only open source graph database that can process graphs with trillions of edges and vertices.
* **OrientDB**：This OLTP system manages relationships using direct pointers that provide consistent performance.
* **Neo4j**: Neo4j is an enterprise-strength OLTP that provides advanced security, native graph storage, and scalable, fast architecture. 

## Examples of systems that support OLAP

Here are a few examples of OLAP.
* **IBM Cognos**: This is an integrated, web-based OLAP system with a toolkit for data analysis, score carding, reporting, and performance analysis.
* **Integrate.io**: This toolkit has code, no-code, and low-code capabilities for developing data pipelines. It allows users to create data for business intelligence.
* **Apache Kylin**: This multidimensional open-source analytics tool provides an SQL interface that supports large datasets.  

## The main differences difference between OLAP and OLTP: Processing type

In order to effectively decide which system to use between OLTP and OLAP, it is vital to understand the key differences between them. Of course the key difference is that OLTP is designed for transactions, while  OLAP is great for analysis and insights. 
Here are the main differences worth knowing.

**1. Data source**

OLAP databases are multidimensional and thus support complex queries from historical and current data sets. OLTP databases can be set up as a data warehouse to provide aggregated data for OLTP. On the other hand, OLTP systems source their data from traditional relational databases that handle massive amounts of real-time transactions.

**2. Processing time**

OLAP handles read-intensive workloads that involve massive amounts of data and this means the response and processing time for OLAP will depend on the amount of data that is being processed. Therefore, it may take seconds, minutes, or even hours, making it slower than OLTP. 
OLTP is a transaction processing tool that handles simple read-write operations using a structured query language (SQL). Such processes require fewer computing resources, and thus transactions and responses are derived in milliseconds.   
**3. Backup**

OLTP handles current transactional data. Such data is updated and modified frequently and thus requires regular backup to maintain data integrity constraints. Furthermore, OLTP requires frequent backup to adhere to governance and legal requirements. However, OLAP doesn’t handle so much current data, which gives room for less frequent backups.  
**4. Purpose**

OLTP runs and controls business operations in real time. It makes it simple to perform insertions, deletions, and updates in a database. On the other hand, OLAP handles complex data analysis to discover hidden insights and support business decisions. OLAP queries involve large amounts of data.   
**5. Storage requirements**

OLTP deals with historical transactional data that requires a small storage capacity, especially if the data is archived. In contrast, OLAP handles massive data because it aggregates multiple large datasets. These processes require a larger storage capacity.
**6. Database design and data view**

OLTP databases store daily business transactions and thus are designed for efficiency. On the other hand, OLAP provides a multidimensional view of business data and is thus designed for data analysis.  
**7. Audience**

OLTP databases are market-oriented. They increase the end user’s productivity by providing data for data-critical users such as clerks and tellers. In contrast, OLAP databases are customer oriented and provide data for data-knowledge users, like top executives, business managers, and data analysts.  

The table below summarizes the key differences between OLTP and OLAP.


|  | OLAP |  OLTP |
| --- | --- | --- |
| **Data Source** | Sources data from OLTP databases | Sources data from the traditional relational databases |
| **Processing time**| Processing time depends on the amount of data being processed | Responses are derived in milliseconds |
| **Backup** | Less frequent backups  | Frequent backups |
| **Purpose**  | Complex data analysis | Simple transactional operations |
| **Storage requirements** | Requires large storage capacity | Needs less storage capacity |
| **Database design and data view**| Designed for analysis | Designed for efficiency |
| **Audience** | Market-oriented data for data-critical users | Customer-oriented data for data-knowledge users. |


### **Drawbacks of OLTP**


**1. Dependency**: OLTP systems can be heavily dependent on staff, which can call for a lot of investment to train new staff members.

**2. Information loss risks**: OLTP systems often contain sensitive information, such as credit card numbers or social security numbers. This information is at risk of being compromised if the system is not properly secured.

**3. Data inconsistencies**: OLTP systems allow concurrent data modifications, which can lead to data inconsistencies. For example, two staff members may attempt to modify the same record at the same time, resulting in one of the changes being overwritten. Additionally, concurrent data modifications can make it difficult to track changes made to the data over time.

**4. Number of queries:** OLTP systems are designed to support a limited number of queries, making them unsuitable for business intelligence or decision support applications.

 

### **Drawbacks of OLAP**




**1. Speed: OLAP systems can be slow**. This is because they have to process large amounts of data, which can take time.

**2. Complex data organization**: OLAP systems require data to be organized into star schemas or snowflakes, which can be complicated. This means that the data must be arranged in a specific way, which can be time-consuming and difficult.

**3. Number of dimensions**: An OLAP cube cannot accommodate large numbers of dimensions. This means that if there are too many dimensions, the system will not be able to cope.

**4. Cube changes**: If any changes are made to the OLAP cube, the entire cube must be updated, which can be time-consuming and frustrating.

# Verdict: Which is best for you, OLAP or OLTP?

When it comes to which to use between OLTP and OLAP, the question should never be about which one is the best but rather how to leverage both for complex data tasks. OLTP focuses on operational data while OLAP focuses on analytics. For example, you can use OLAP to gather insights from massive data, then use these insights to create efficient transactions with  OLTP. 

Using both systems gives you the advantage of being able to have better control of your data and derive holistic value. Using only one where you need both can be limiting, as you will more or less be forcing a single system to perform functions that it’s not designed for. But if you are purely interested in data analysis, then by all means go for OLAP. If you are exclusively keen on transactions and nothing more, it’s perfectly okay to go for OLTP. 


Let's look at how OLTP and OLAP are different in Graph databases. And get an idea of which is best for you. 

OLTP vs OLAP: What Are The Differences?

Based on the actual demand and subsequent scalability, the NebulaGraph database is used for technical selection. The batch import performance in the actual business scenarios of NebulaGraph needs to be first verified.

This article shares the best practice of using NebulaGraph Spark Connector by performing import jobs via Spark on Yarn distributed tasks and with CSV files on HDFS as the data source.

# About NebulaGraph Spark Connector

The following figure shows the use cases and advantages of NebulaGraph Spark Connector. For more details, see [NebulaGraph Spark Connector](https://docs.nebula-graph.io/3.0.1/nebula-spark-connector/).

<pic>


# Environment Information

- Hardware environment 

| **Name**     |  **Value**  | **Recommend** |
| :----------- | :------: | -------: |
| Local Disk (SSD) |   2 T    | At least 2 T |
| CPU          | 16 C * 4 |    128 C |
| Memory         |  128 GB  |    128 GB |

- Software environment

| **Name**               | **版本号**  |
| :--------------------- | :---------: |
| NebulaGraph           |    3.0.0    |
| NebulaGraph Spark Connector |    3.0.0    |
| Hadoop                 | 2.7.2U17-10 |
| Spark                  |   2.4.5U5   |

- Data size

| **Name**     | **Value** |
| :----------- | :----: |
| Data size      | 200 GB  |
| Vertex | 930 million |
| Edge   | 970 million |

# Deployment Plan

- Method: Distributed, 3 nodes
- See [Deploy a NebulaGraph cluster with RPM/DEB package on multiple servers](https://docs.nebula-graph.com.cn/3.0.1/4.deployment-and-installation/2.compile-and-install-nebula-graph/deploy-nebula-graph-cluster/)

There are roughly three steps for the deployment.

1. Download the RPM package of NebulaGraph and install it.
2. Modify configuration files in batch.
3. Start the cluster service.

The above-mentioned three steps are performed with the `root` user. For non-root users, use `sudo` to execute the commands.

## Download and Install the NebulaGraph RPM Package

Execute the following commands:

```bash
wget https://os-cdn.nebula-graph.com.cn/package/3.0.0/nebula-graph-3.0.0.el7.x86_64.rpm
wget https://oss-cdn.nebula-graph.com.cn/package/3.0.0/nebula-graph-3.0.0.el7.x86_64.rpm.sha256sum.txt
rpm -ivh nebula-graph-3.0.0.el7.x86_64.rpm
```

Note: The default installation path is `/usr/local/nebula`. Make sure the memory resources are sufficient.

## Modify Configuration Files in Batch

```bash
sed -i 's?--meta_server_addrs=127.0.0.1:9559?--meta_server_addrs=172.1x.x.15:9559,172.1x.x.176:9559,172.1x.1x.149:9559?g' *.conf
sed -i 's?--local_ip=127.0.0.1?--local_ip=172.1x.1x.149?g' *.conf
sed -i 's?--meta_server_addrs=127.0.0.1:9559?--meta_server_addrs=172.1x.x.15:9559,172.1x.x.176:9559,172.1x.1x.149:9559?g' *.conf
sed -i 's?--local_ip=127.0.0.1?--local_ip=172.1x.x.15?g' *.conf
sed -i 's?--meta_server_addrs=127.0.0.1:9559?--meta_server_addrs=172.1x.x.15:9559,172.1x.x.176:9559,172.1x.1x.149:9559?g' *.conf
sed -i 's?--local_ip=127.0.0.1?--local_ip=172.1x.x.176?g' *.conf
```

Note: The IP addresses in the above commands are internal IP addresses, used for communication between nodes. 


## Start the NebulaGraph Cluster Service

```bash
/usr/local/nebula/scripts/nebula.service start all
```

The above command is to start the cluster service, the following is to check the status of the cluster service.

```bash
ps aux|grep nebula
```

The returned result shows 3 processes for each service.

```bash
/usr/local/nebula/bin/nebula-metad --flagfile /usr/local/nebula/etc/nebula-metad.conf
/usr/local/nebula/bin/nebula-graphd --flagfile /usr/local/nebula/etc/nebula-graphd.conf
/usr/local/nebula/bin/nebula-storaged --flagfile /usr/local/nebula/etc/nebula-storaged.conf
```

Note: If the returned result shows less than 3 processes, run `/usr/local/nebula/scripts/nebula.service start all` several times or run `/usr/local/nebula/scripts/nebula.service restart all` to restart the service.


# Visualization Service

The NebulaGraph Studio is used as the visualize the NebulaGraph cluster service. Visit `http://n01v:7001` to log into the NebulaGraph Studio. (This is my network environment which is not accessible from the Internet.)

<pic>

- Login: 10.x.x.1 (The IP of any node):9669
- Username/Password: `root`/`nebula`

Refer to [Use nGQL (CRUD)](https://docs.nebula-graph.com.cn/3.0.1/2.quick-start/4.nebula-graph-crud) for common-used nGQL statements.


## Use NebulaGraph

Add the Storage nodes to the cluster.

```nebula
ADD HOSTS 172.x.x.121:9779, 172.16.11.218:9779,172.16.12.12:9779;
```

List all nodes and check whether the status of the cluster is `ONLINE by running `SHOW HOSTS` or `SHOW HOSTS META`.
Create a graph space, equivalent to a database in relational databases.
```nebula
CREATE SPACE mylove (partition_num = 15, replica_factor = 3, vid_type = FIXED_STRING(256));//The number of partitions is recommended to be 5 times the number of nodes; the number of replicas is an odd number, generally set to 3; if the data type of vids is string, make the length as long as a reasonable value, otherwise, it takes up too much disk space.
```

Create a Tag, equivalent to an entity.: 

```nebula
CREATE TAG entity (name string NULL, version string NULL);
```  

Create an edge, equivalent to a relationship.

```nebula
CREATE EDGE relation (name string NULL);  
```

Add `limit` when querying data, otherwise, the query may take a long time.

```nebula
match (v) return v limit 100;
```

# Read and Import CSV Files with Spark Connector 

References:

- [NebulaSparkWriterExample (Scala Json) from the official](https://github.com/vesoft-inc/nebula-spark-utils/blob/master/example/src/main/scala/com/vesoft/nebula/examples/connector/NebulaSparkWriterExample.scala)
- [NebulaSparkWriterExample (Java Json) from an ace](https://www.jianshu.com/p/930e0343a28c)

Code example of NebulaSparkWriterExample

```
import com.facebook.thrift.protocol.TCompactProtocol
import com.vesoft.nebula.connector.{
  NebulaConnectionConfig,
  WriteMode,
  WriteNebulaEdgeConfig,
  WriteNebulaVertexConfig
}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.slf4j.LoggerFactory
object NebulaSparkWriter {
  private val LOG = LoggerFactory.getLogger(this.getClass)
  var ip = ""
  def main(args: Array[String]): Unit = {
    val part = args(0)
    ip = args(1)
    val sparkConf = new SparkConf
    sparkConf
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .registerKryoClasses(Array[Class[_]](classOf[TCompactProtocol]))
    val spark = SparkSession
      .builder()
      .master("local")
      .config(sparkConf)
      .getOrCreate()
    spark.sparkContext.setLogLevel("WARN")
    if("1".equalsIgnoreCase(part)) writeVertex(spark)
    if("2".equalsIgnoreCase(part)) writeEdge(spark)
    spark.close()
  }
  def getNebulaConnectionConfig(): NebulaConnectionConfig = {
    val config =
      NebulaConnectionConfig
        .builder()
        .withMetaAddress(ip + ":9559")
        .withGraphAddress(ip + ":9669")
        .withTimeout(Integer.MAX_VALUE)
        .withConenctionRetry(5)
        .build()
    config
  }
  def writeVertex(spark: SparkSession): Unit = {
    LOG.info("start to write nebula vertices: 1 entity")
    val df = spark.read.option("sep", "\t").csv("/home/2022/project/origin_file/csv/tag/entity/").toDF("id", "name", "version")
    val config = getNebulaConnectionConfig()
    val nebulaWriteVertexConfig: WriteNebulaVertexConfig = WriteNebulaVertexConfig
      .builder()
      .withSpace("mywtt")
      .withTag("entity")
      .withVidField("id")
      .withVidAsProp(false)
      .withUser("root")
      .withPasswd("nebula")
      .withBatch(1800)
      .build()
    df.coalesce(1400).write.nebula(config, nebulaWriteVertexConfig).writeVertices()
  }
  def writeEdge(spark: SparkSession): Unit = {
    LOG.info("start to write nebula edges: 2 entityRel")
    val df = spark.read.option("sep", "\t").csv("/home/2022/project/origin_file/csv/out/rel/relation/").toDF("src", "dst", "name")
    val config = getNebulaConnectionConfig()
    val nebulaWriteEdgeConfig: WriteNebulaEdgeConfig = WriteNebulaEdgeConfig
      .builder()
      .withSpace("mywtt")
      .withEdge("relation")
      .withSrcIdField("src")
      .withDstIdField("dst")
      .withSrcAsProperty(false)
      .withDstAsProperty(false)
      .withUser("root")
      .withPasswd("nebula")
      .withBatch(1800)
      .build()
    df.coalesce(1400).write.nebula(config, nebulaWriteEdgeConfig).writeEdges()
  }
}
```
## Details on NebulaSparkWriterExample 

The following are the details of the functions.

- `spark.sparkContext.setLogLevel("WARN")`: Sets the log print level and prevents the INFO log. 


- `withTimeout(Integer.MAX_VALUE)`: Sets the connection timeout. The timeout can be set as large as possible. The default value is 1 minute. When the number of timeouts is greater than that of retries, the Spark task fails.

- `option("sep", "\t")`: Specifies the separator of CSV files, otherwise, it defaults to 1 column.

- `toDF("src", "dst", "name")`: The schema of the dataset, i.e., converting `Dataset<Row>` to `DataFrame`, otherwise, the `VidField` cannot be specified.

- `withVidField("id")`: This function only works when the column name is set, so a schema must be specified.

- `withVidAsProp(false)`: When the ID is set as the VID by default, data will not be written repeatedly as a property that will not take up disk space.

- `withSrcIdField("src")`: Sets the `IdField` of the source vertex.

- `withDstIdField("dst")`: Sets the `IdField` of the destination vertex.

- `withSrcAsProperty(false)`: Saves disk space.

- `withDstAsProperty(false)`: Saves disk space.

- `withBatch(1000)`: The number of rows of data imported in batch. `WriteMode.UPDATE` is <=512 by default. The value of `WriteMode.INSERT` can be larger (Gigabit NIC/Bandwidth 5Gbps /Local SSD = 1500)

- `coalesce(1500)`: Adjustable according to the number of concurrent tasks. Too much data on a single partition can easily cause executor OOM exceptions.

# Submit Tasks to the Spark Cluster

```spark
nohup spark-submit  --master yarn --deploy-mode client --class com.xxx.nebula.connector.NebulaSparkWriter --conf spark.dynamicAllocation.enabled=false --conf spark.executor.memoryOverhead=10g  --conf spark.blacklist.enabled=false --conf spark.default.parallelism=1000 --driver-memory 10G --executor-memory 12G --executor-cores 4 --num-executors 180 ./example-3.0-SNAPSHOT.jar >  run-csv-nebula.log 2>&1 &
```

## Monitoring with `iotop`

The `iotop` command can be used to monitor the disk IO of the Spark cluster.

```bash
Total DISK READ :      26.61 K/s | Total DISK WRITE :     383.77 M/s
Actual DISK READ:      26.61 K/s | Actual DISK WRITE:     431.75 M/s
```
## Monitoring with `top` 

The `top` command can be used to monitor the CPU and memory of the Spark cluster.

```bash
top - 16:03:01 up 8 days, 28 min,  1 user,  load average: 6.16, 6.53, 4.58
Tasks: 205 total,   1 running, 204 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.3 us, 14.2 sy,  0.0 ni, 56.0 id,  0.6 wa,  0.0 hi,  0.4 si,  0.5 st
KiB Mem : 13186284+total,  1135004 free, 31321240 used, 99406592 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 99641296 avail Mem 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                         
27979 root      20   0 39.071g 0.026t   9936 S 564.6 20.8  83:22.03 nebula-storaged                                                                 
27920 root      20   0 2187476 804036   7672 S 128.2  0.6  17:13.75 nebula-graphd                                                                   
27875 root      20   0 6484644 1.990g   8588 S  58.5  1.6  14:14.22 nebula-metad     
```
## Other Resource Monitoring

<pic>

# Service Optimization

## Configuration Optimization of `nebula-storaged.conf`

Here, I modified the value of parameters in the `nebula-storaged.conf` file.

```bash
# Default reserved bytes for a batch operation.
--rocksdb_batch_size=4096
# Default block cache size used in BlockBasedTable.
# The unit is MB. It is usually set to one third of the memory value. The memory is 128 GB.
--rocksdb_block_cache=44024
############## rocksdb Options ##############
--rocksdb_disable_wal=true
# In JSON, the name and value of each option of rocksdb DBOptions is a string. Such as “option_name”:“option_value”. Use a comma to separate each option.
--rocksdb_db_options={"max_subcompactions":"3","max_background_jobs":"3"}
# In JSON, the name and value of each option of rocksdb ColumnFamilyOptions is a string. Such as “option_name”:“option_value”. Use a comma to separate each option.
--rocksdb_column_family_options={"disable_auto_compactions":"false","write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
# In JSON, the name and value of each option of rocksdb BlockBasedTableOptions is a string. Such as “option_name”:“option_value”. Use a comma to separate each option.
--rocksdb_block_based_table_options={"block_size":"8192"}
# The number of processors per request.
--max_handlers_per_req=10
# The interval time between clusters.
--heartbeat_interval_secs=10
--raft_rpc_timeout_ms=5000
--raft_heartbeat_interval_secs=10
--wal_ttl=14400
# The max value for batch import.
--max_batch_size=1800
# Reduces memory usage when the value is true.
--enable_partitioned_index_filter=true
# The data is indirectly filtered at the bottom storage layer to prevent encountering the super node in the production environment.
--max_edge_returned_per_vertex=10000
```
## Optimizations of the Linux System

```bash
ulimit -c unlimited
ulimit -n 130000
sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sysctl -w net.core.somaxconn=2048
sysctl -w net.ipv4.tcp_max_syn_backlog=2048
sysctl -w net.core.netdev_max_backlog=3000
sysctl -w kernel.core_uses_pid=1
```

# Verify Import Results

```nebula
SUBMIT JOB STATS;
SHOW JOB ${ID}
SHOW STATS;
```
- The entity insertion rate is approximately 27,837 items/s (for this import performance calculation only).

- The relationship insertion rate is about 26,276 items/s (for this import performance calculation only).

- The import performance will be better if the server is better configured. In addition, bandwidth, cross-datacenter or not, disk IO, and even network fluctuations are also factors that affect performance.

# Performance Tests

- Query for the specified entities based on properties.

  ```nebula
  MATCH (v:entity) WHERE v.entity.name == 'Lifespan' RETURN v;
  ```

  It takes 0.002558 (s).

- One-hop queries.

  ```nebula
  MATCH (v1:entity)-[e:propertiesRel]->(v2:attribute) WHERE id(v1) == '70da43c5e46f56c634547c7aded3639aa8a1565975303218e2a92af677a7ee3a' RETURN v2 limit 100;  
  ```
  It takes 0.003571 (s)

- Two-hop queries.

  ```nebula
  MATCH p=(v1:entity)-[e:propertiesRel*1..2]->(v2) WHERE id(v1) == '70da43c5e46f56c634547c7aded3639aa8a1565975303218e2a92af677a7ee3a' RETURN p;
  ```
  It takes 0.005143 (s)

- Query for all the properties of an edge.

  ```nebula
  FETCH PROP ON propertiesRel '70da43c5e46f56c634547c7aded3639aa8a1565975303218e2a92af677a7ee3a' -> '0000002d2e88d7ba6659db83893dedf3b8678f3f80de4ffe3f8683694b63a256' YIELD properties(edge); 
  ```  
  It takes 0.001304 (s)

  ```nebula
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*1]->(v2) return p;
  ```
  å
  It takes 0.02986 (s)

  ```nebula
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*2]->(v2) return p; 
  ```

  It takes 0.07937 (s)

  ```nebula
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*3]->(v2) return p; 
  ```

  It takes 0.269 (s)

  ```nebula
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*4]->(v2) return p;
  ```

  It takes 3.524859 (s)

  ```nebula 
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*1..2]->(v2) return p; 
  ```

  It takes 0.072367 (s)

  ```nebula 
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*1..3]->(v2) return p;
  ```

  It takes 0.279011 (s)

  ```nebula 
  match p=(v:entity{name:"张三"})-[e:entityRel|propertiesRel*1..4]->(v2) return p; 
  It takes 3.728018 (s)
  ```

 - Query for the properties of vertices and the edge in the shortest path in bilateral directions between `A_vid` and `B_vid`.

  ```nebula
  FIND SHORTEST PATH WITH PROP FROM "70da43c5e46f56c634547c7aded3639aa8a1565975303218e2a92af677a7ee3a" TO "0000002d2e88d7ba6659db83893dedf3b8678f3f80de4ffe3f8683694b63a256" OVER * BIDIRECT YIELD path AS p; 
  ```
  It takes 0.003096 (s)

  ```nebula
  FIND ALL PATH FROM "70da43c5e46f56c634547c7aded3639aa8a1565975303218e2a92af677a7ee3a" TO "0000002d2e88d7ba6659db83893dedf3b8678f3f80de4ffe3f8683694b63a256" OVER * WHERE propertiesRel.name is not EMPTY or propertiesRel.name >=0 YIELD path AS p;
  ```

  It takes 0.003656 (s)

# Problems Encountered

## Guava version conflict

```
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;
```

After troubleshooting, I found that one of the dependent modules used Guava version 22.0, while the Spark cluster came with 14.0, resulting in a conflict, and it did not work properly. For tasks running on Spark clusters, Spark loads Guava packages with higher priority than its packages.

The package we depend on uses methods that are newer in Guava version 22.0 and not available in version 14.0. Without being able to modify each other's code, the following options are available:

1. The Spark cluster package can be upgraded, which is risky and can cause unknown problems.

2. Rename your Guava package using the Maven plugin.

The second way is used here, using the Maven plugin shade (link: https://maven.apache.org/plugins/maven-shade-plugin/) to rename the package to solve the problem.

```java
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <relocations>
                    <relocation>
                        <pattern>com.google.common</pattern>
                        <shadedPattern>my_guava.common</shadedPattern>
                    </relocation>
                </relocations>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/maven/**</exclude>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>
```
## Spark Blacklist Mechanism
```
Blacklisting behavior can be configured via spark.blacklist.*.
```
The default value of `spark.blacklist.enabled` is `false`. If the value is set to `true`, Spark will no longer schedule tasks to the blacklisted executors. The blacklisting algorithm can be further configured by setting `spark.blacklist`.
  
*This article was written by Enqiu Jia and first published on the WeChat Official Account *只要5分钟不要996*. This article is translated and published with permission.

This is a best practice of using NebulaGraph Spark Connector by performing import jobs via Spark on Yarn distributed tasks and with CSV files on HDFS as the data source.

Offline CSV Import of Billions of Data | Best Practice Using NebulaGraph Spark Connector

Relational databases are well-suited for storing data that can be easily organized into rows and columns, such as customer addresses or product inventories. Non-relational databases, on the other hand, are better suited for storing data that does not fit neatly into rows and columns, such as unstructured text or images. 

Difference Between Relational and Non Relational Database

The place of data in driving business growth is evident in the way that many organizations are aggressively collecting and utilizing data. As data becomes more accessible and easier to analyze, it is playing an increasingly important role in helping organizations to identify trends, understand customer behavior, optimize products and maximize profits. In many ways, data has become the "new oil" - a valuable commodity that can be used to drive growth and competitive advantage. Prudent data organization, manipulation and management are at the heart of commoditizing this new oil. It would not have been possible to realize the kind of ground-breaking data-driven applications we are witnessing today without database tools.  From social networking sites to recommendation engines across e-Commerce marketplaces, it’s all about creating value out of data and databases are central here. In fact, the [database management systems global market](https://www.researchandmarkets.com/reports/5310748/database-management-systems-dbms-global) is expected to grow at a CAGR of 12.2 percent and reach 142.7 trillion US dollars by 2027.

Depending on what you want to achieve with your data, you might often end up at a crossroad where you’ll have to choose between databases. Graph database vs. relational database is one such crossroad that many may be struggling with. Let’s distinguish these two and help you choose the right one.    
# What is a graph database?

A graph database is a NoSQL database where data is stored as a network graph. A typical graph database contains edges, nodes, and properties that present and store data. Relationships are first-class citizens in a graph database and are identified with unique keys. By making relationships first-class citizens, graph databases can more efficiently store, retrieve, and traverse data. Graph databases excel at handling data with complex relationships and are therefore useful for applications such as social networking, [recommendation engines](https://nebula-graph.io/posts/use-cases-of-graph-databases-in-real-time-recommendation), and fraud detection among many more. 

An [open source graph database](https://nebula-graph.io) is always the best place to start as they come with a supportive community that ultimately creates the perfect ecosystem.

# How does a graph database work?
Graph databases work by using two essential elements: nodes and edges. The nodes in a graph represent entities, and the edges represent relationships between those entities. This allows for more flexible and efficient querying. For example, in a social network, each user would be represented by a node, and the edges would represent the relationships between users. A graph database could then easily answer questions such as "Who are the friends of my friends?" or "What path connects me to another user?"

They perform traversal queries and apply algorithms to determine patterns, influencers, paths, points of failure, and communities. Understanding these aspects allows users to effectively analyze massive amounts of data.  
# Types of graph databases
Graph databases can be categorized into two main groups based on data model and storage. 
## Data model-based graph databases
There are three major types under this category namely the property paragraph, RDF graphs, and hypergraph
1. Property graph

Property graphs organize data in relationships, nodes, and properties and store the data on relationships or nodes. 

Nodes are entities and can hold multiple attributes called properties. You can tag nodes with labels that represent their roles in a domain. In addition, labels can attach metadata, like constraint or index information, to specific nodes. 

Relationships provide relevant connections between different nodes. The relationship is directed, named, and has a start and end node. Furthermore, relationships have quantitative properties like distances, costs, ratings, weights, strengths, and time intervals. Relationships are efficiently stored, and thus nodes can share relationships without compromising performance. The navigation of relationships can be done in either direction, despite the fact that they are stored in one direction. 

2. Hypergraph 

Hypergraphs are data models that allow relationships to connect to multiple nodes by allowing several nodes at either end. This will enable users to analyze and store data with numerous many-to-many relationships. The relationships in a hypergraph are referred to as hyperedges. 

In contrast to property graphs, hypergraphs have multidimensional edges, making them more generalized. However, the two are isomorphic, meaning you can present a hypergraph as a property graph, but you can’t do the opposite. 

3. Resource description framework (RDF)

An RDF, or triple store, stores data in a three-part format of subject-data-object data structure. A separate node is used to present any additional information. RDF graph models are made of Arcs and Nodes, and the graph notation is presented by two nodes, one for the subject and another for the object, and an arc presents the predicate. 

Triple stores are categorized as data model-graph databases because they process logically linked data. However, their storage engines are not optimized for property graphs, nor do they support index-free adjacency; thus, they aren’t native graph databases. 

RDFs can scale horizontally but can’t rapidly transverse relationships because they store triples as individual elements. They must create connections from independent facts and add latency to perform graph queries. Triple stores are mostly applied in offline analytics because of their shortcomings in latency and scale. 
## Storage-based graph databases
This category contains three major types as well, namely the native storage graph, relational storage, and key-value store graphs:
1. Native graph storage
Native graph storage uses edge and vertex units to store and manage graph databases. It is best suited for multiple-hop or deep-link graph analytics. Native graph storage is designed to maximize traversal speed in arbitrary graph algorithms. 
2. Relational storage
This graph database stores the edge and vertex table using a relational model. During runtime, relational “JOIN” concatenates (links)  the two tables. The relational model represents data in tables using an intuitive and easy-to-understand way. Each row is a record with a key, which is the unique ID. Each column holds an attribute of the data, and each record has the attribute’s value. This makes it easy to identify the relationships between data points.   
3. Key-value store
A key-value store is a nonrelational database that uses NoSQL databases to store data. It stores data in key-value pairs where the key is the unique identifier. The keys and values range from complex compound objects to simple objects. 
Perhaps the best benefit of a key-value graph DB vs relational DB is that the key-value graph is highly partitionable. As a result, it allows horizontal scaling that most database types cannot achieve. 
# The graph data model vs the relational data model
Though both graph data and relational models focus on data relationships, they don’t do it similarly.

You decide which entities will be nodes and links and which you will discard in a graph data model. This results in a blueprint that you can use to create visualization models for charts.

A relational data model groups database entries by their relationships using a table of values. Each row in a table represents corresponding data values and denotes a real-world relationship. Table and column names help in interpreting the values in each row. 

A graph data model stores relationships at an individual record level. On the other hand, a relational data model uses predefined structures called table definitions. However, you can convert a relational database to a graph database. 
# Key differences between graph databases and relational databases
The fundamental difference between graph databases and relational databases lies in how relationships are handled. In a graph database, relationships are driven by data points. In a relational database, relationships are driven by columns in data tables.

Here are the differences in detail:

- **Storage format:** A graph database stores entities as nodes and relationships as edges. A relational database stores data in tables with rows and columns and uses “JOIN” for fast querying. 

- **Dataset size:** Graph databases are quick even for large datasets, while relational databases are slower.
- **Index:** Graph databases typically use index-free adjacency, meaning that every node is connected to every other node in the database, while relational databases use indexed pointers to connect related data.
- **Transactions:** Graph databases typically don't support transactions very well, while relational databases can perfectly support transactions and even accounting.
- **Processing power:** Graph databases require more processing power and storage space.

# Advantages of graph database over relational databases
There are reasons you would choose a graph database over a relational database, and this is informed by the advantages. Here are the key advantages of a relational database vs a graph database:
1. Graph databases are better at handling relationships

When it comes to handling relationships, graph databases are king because of the way they're built. They can easily represent complex relationships between data points.

This is a huge advantage when you're trying to model data that's naturally hierarchical, like social networks or business relationships. With a graph database, you can easily create entities and track the relationships between them. This makes finding and understanding data a lot easier, which is essential when you're trying to make quick decisions based on insights from your data.

In a relational database, you have to define the relationships between tables ahead of time. With a graph database, you can define relationships on the fly, which makes it much easier to handle complex data models.

2. Graph databases are more scalable

This means that graph databases can handle large amounts of data without running into issues. And as data volumes continue to grow, this is an increasingly important factor to consider.

You can easily divide the database across multiple servers while keeping intact vital aspects of compliance such as privacy requirements. 

3. Graph databases can be more performant

When it comes to graph database vs relational database performance, data is stored in a graph format in graph databases, which makes it easier to navigate and find connections between data points. Second, the way that data is structured in a graph database makes it possible to index data more effectively. This means that queries can be executed more quickly, which leads to better performance and user experience.

You can actually  increase the relationships without compromising the performance.

4. Graph databases offer more flexible data modeling

With a graph database, you can model your data however you want, which means you're not limited to the rigid structures of a relational database. By representing data as a series of interconnected nodes, graph databases can more accurately capture the intricate web of relationships. 

5. Graph databases are easier to understand

Things are a lot simpler to understand in a graph database. In fact, you can think of a graph database as a collection of nodes and relationships between nodes. And that's it!

There's no need to worry about tables, columns, or foreign keys. Just create your nodes and relationships and you're good to go. This makes data management a breeze and makes it easy for other people to understand what you're trying to do.

6. Graph databases can be more trusting

Graph databases can be more trusting because they are based on a one-table model. In a graph database, the table is called a graph, and it contains all the information about entities and the relationships between them.

This makes it much easier to find information because there is no need to join tables or run complex queries. All you need to do is find the node that you're interested in and look at the relationships associated with it.

7. Graph databases can better handle data consistency

Let's say you're a retailer and you have a customer database. A relational database would be fine for storing information about your customers, but it wouldn't be the best tool for handling customer-product interactions.

Why? Because a relational database is based on a normalized model, which means that data is divided into tables and columns. And this can lead to inconsistencies when you're trying to query data. For example, if you want to get a list of all your customers for a particular product, you might have to query two or three different tables.

A graph database, on the other hand, is based on a non-normalized model, which means that data is stored in one place. This makes it much easier to query data, because everything is in one place.

8. Graph databases can be more extensible

What this simply means is that they can handle more data combinations. For example, let's say you want to track the relationships between people, organizations, and things. A graph database would be perfect for this task, whereas a relational database would quickly become overloaded.

Graph databases are also great for managing data that changes rapidly, such as social media data or sensor data. This is because they can quickly adapt to changes in the data model, without having to perform a full database rebuild.

# When should you use the graph database?
Graph databases are well suited for applications that require the storage and retrieval of data that can be represented as a graph, such as social networks, maps, and networks. They are also well suited for applications that require the analysis of data that is connected in complex ways, such as [fraud detection](https://nebula-graph.io/posts/fraud-detection-using-knowledge-and-graph-database) and recommendation engines.

For example, in a social network, a user's friends are also friends with each other. A graph database can quickly find all the friends of a user's friends. In contrast, a relational database would need to perform multiple joins to find the same information. By prioritizing relationships, graph databases can provide greater insight into data. In general, any application that would benefit from being able to represent data as a network of interconnected nodes would be a good candidate for a graph database.

# Conclusion
Understanding graph database vs relational database is the first step to building effective data models that will provide valuable insights into connected data. It is also important to note that the two are not alternatives, but each serves a different purpose. 

The most important point that you’ll always need to bear in mind in graph vs relational databases is that graph databases are better suited for applications that require multiple relationships between data points, while relational databases are better for applications with less complex data structures.

# FAQ Section
## How does a graph database differ from a relational database?
A graph database uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. A relational database, on the other hand, uses tables and relations between them to store data. Graph databases are used to query complex relationships, whereas relational databases are used for more simple relationship structures. In addition, graph databases typically require fewer joins than relational databases. As a result, graph databases can be faster and more efficient when it comes to handling complex data.
## Are graph databases better than relational databases?
Yes to a large extent. Graph databases are better than relational databases because they are more flexible and can handle more complex data relationships. Relational databases are based on the table structure of data, which is difficult to change once the data is in the database. Graph databases, on the other hand, are based on the graph structure of data, which is easy to change.  Considering that data is becoming increasingly complex, graph databases are far much better for most use cases where complex data manipulation is a priority. 
## What is a graph database not good for?
Graph databases are well suited for storing data with complex relationships, such as social networks or financial data. However, graph databases are not well suited for storing data that can be easily represented in a tabular format, such as product catalogs or customer orders. 

# You May Also Like:
Has this article aroused your interest in **graph databases**? If yes, then don't hesitate and dive in. Explore more in-depth details of the concepts and practical applications of graph databases in this blog: 
[What is a graph database and what are its use cases - Definition, examples & trends](https://www.nebula-graph.io/posts/what-is-a-graph-database)

Graph databases are better than relational databases because they are more flexible and can handle more complex data relationships. 

Graph Database vs Relational Database: What to Choose? 

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

# Graph learning and storage technology have attracted much attention

‍

**There are two hotspots that most participants pay attention to at this conference, which reflects the new trend in the industry.&nbsp;**

**First, the importance of machine learning is further increasing, and graph learning is becoming a new hotspot**. In recent years, more and more conferences that focus on specific systems set sections about machine learning to study the use of machine learning in optimizing systems or algorithms. Among the sub-fields of machine learning, a new force, i.e., graph learning, which is machine learning on graphs, suddenly rises. Just like other sub-fields of machine learning, graph learning also aims at solving problems in the real world. **Through the method of embedding, graph learning extracts feature vectors from a graph, and then uses machine learning methods to solve complex problems in the graph, including classification, subgraph matching, an  
d link classification.**		

Judging by the tendency of conferences and guidance from the National Science Foundation (NSF), this trend is likely to continue. We have also received a lot of inquiries from students and professors about whether NebulaGraph supports graph learning. From a graph database provider's perspective, graph learning is one of the NebulaGraph application scenarios. We should provide not only the graph database core, but also libraries, interfaces, and even optimizations for upper-layer applications to ensure the smooth and efficient operation of upper-layer applications. On the other hand, if NebulaGraph can help academia make some breakthroughs in areas like graph learning, it'll be great for our company, and the industry as a whole.
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure1.jpeg)

**In addition to machine learning, I think another hotspot is memory, including in-memory databases, persistent memory, etc**. Non-volatile memory, or persistent memory, has become more and more mature through the joint efforts of academia and industry for more than ten years. Research on how to use persistent memory is also very hot, and there are numerous articles on it at top events of almost all systems.

**In the database field specifically, research topics include how to design an in-memory database, how to mix durable memory with SSD, DRAM, and other types of storage, and how to solve the problem of memory segregation.** NebulaGraph, as a graph database provider, should be embracing the changes that new hardware brings, and exploring persistent memory-based storage systems as well.

# The advantages and development direction of the graph

**In addition to the new trends, my biggest takeaway and thought from the conference was the comparison between graph databases (GDBMS) and relational databases (RDMBS). **There were quite a few people at the conference who were wondering why the graph database was not implemented based on a relational database, because relational databases have been optimized over the years, and they are really armed to the teeth. For example, Professor Peter Boncz (one of the founders of LDBC) gave a keynote address at the EDBT/ICDT conference earlier this year on The (Sorry) State of Graph Database Systems [1]. He stated that compared with RDBMS, GDBMS still had a lot to improve in terms of computation and storage, especially regarding AP queries like subgraph matching. He proposed a performance benchmark for subgraph matching scenarios. In his experimental environment, none of the graph database software came close to the performance of relational databases Hyper and Umbra in the subgraph matching scenario (see the following figure).
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure2.png)

Certainly, the setup of this experiment is worthy of discussion. Still, Peter, as a founder of LDBC, did not mean to disparage graph databases but to encourage graph database providers to improve their products by using subgraph matching, a query scenario in which graph databases are not well optimized.

From the broader picture, NebulaGraph, as a graph database provider,** must know the differences between graph databases and relational databases and grasp the advantages of graph databases**. For example,** GQL is easier to use and more efficient when executing graph queries than SQL**. Emil Eifrem, CEO of Neo4J, has a perfect example [3]: For a very common query in an AP system, if using SQL, it needs a very long statement consisting of 23 SELECTs, 21 WHEREs, 11 JOINs, and 9 UNIONs. But if you use a graph query language, you only need a MATCH and a WHERE. For SQL, the query task may be impossible to complete or easy to go wrong, but for a graph query language, the task is easy to complete, saving countless manpower and resources.

**In terms of performance**, in addition to the natural advantages of graph databases over relational databases in association relation queries, i**t is also necessary to gain an advantage over relational databases in a variety of other graph-related queries (not limited to subgraph matching mentioned above). **This is actually the core enlightenment of Peter's proposal of subgraph query as a benchmark for us.		

‍

# Thoughts about the underlying storage structure

![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/thoughts-after-attending-the-sigmod22-conference/figure3.png)


At last, we also gained a number of suggestions for the underlying storage. NebulaGraph currently applies RocksDB, which is based on LSM-Tree, as the underlying storage engine. However, it has been controversial whether LSM-Tree is suitable for the workload of the graph database, especially, whether it is the best choice with the lowest cost on the cloud. Possible alternatives are:

- Bε-tree File System, or betrFS.[ https://www.betrfs.org/](https://www.betrfs.org/)

- LiveGraph. [https://marcoserafini.github.io/papers/LiveGraph.pdf](https://marcoserafini.github.io/papers/LiveGraph.pdf)

- B+ Tree. We will follow up the research on this aspect.	

All in all, the SIGMOD'22 trip was fruitful. Look forward to SIGMOD next year. See you in Seattle!

[1] [https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html](https://conferences.inf.ed.ac.uk/edbticdt2022/?contents=invited_talks.html)

[2] [https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47](https://docs.google.com/presentation/d/1xUbooiL8rIMzkp9G9EXmN4tIMMWt2mC53Q0u4mslq5g/edit#slide=id.ge0f13c0edc_0_47)

[3] [https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf](https://people.eecs.berkeley.edu/~istoica/classes/cs294/15/notes/21-neo4j.pdf)

‍

This year's SIGMOD'22 was held in Philadelphia, and I was honored to attend as a speaker on behalf of the company. During this conference, I communicated a lot with students, professors, and companies. On the one hand, I publicized our products; on the other hand, I got many valuable suggestions and opinions. Next, I'll share some of these suggestions, as well as my overall thinking.

Thoughts after attending the SIGMOD'22 conference

# Financial Fraud Detection: One of the Best Practices of Knowledge Graph

Hello, I'm Qingxin Kong from Bangsun Technology. I'm glad that I have this opportunity to share the topic **Financial Fraud Detection: Best Practices of Knowledge Graph **with NebulaGraph community members.

### Background: Problems with traditional fraud detection solutions

Financial frauds remain a hot topic in recent years. There's a new trend emerging that fraudsters are organized and ganged up on. It is difficult to verify whether the customer's information provided is fake or not. Such inconsistent information makes it difficult to identify the real frauds and leads to a lot of problems. The bank may control the passing rate by raising the requirements to handle loan applications due to the uncertainty of the customer's information, which is effective but brings losses.

The 2022 government work report said that financial inclusion will be further expanded and the customer base will include rural customers, which means there will be "white accounts" having no credit reports. Traditional fraudulent behaviors include fake credit reports, commission agencies, and collusion. 

Some agencies focus on a few banks for credit card applications or loans. Those agencies know the bottom line and red line of the banks. As long as the bottom line is not touched, they can successfully apply for loans. Those agencies may even know how to make the loan amount higher.

Traditional fraud detection solutions rely heavily on expert rules, that is, risk control experts set a batch of rules through experience and then adjust the trigger conditions to find a business balance between the pass rate and rejection rate. It costs little to forge things like ID numbers and bank card flows. There are no good means of control for mass and high-volume frauds.

### Empowering graphs: Paths to intelligent fraud detection solutions
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure1.png)

Let's take a look at the development path of intelligent fraud detection solutions. 

The first step is to build expert rules, but it is difficult to cover all risks through expert experience. The second step is to build a machine learning model to prevent and control the overall risk and then use expert rules to help the model. Expert rules and machine learning models well complement each other. However, based on the current experience, we may face lots of problems when building the machine learning model. The first reason is insufficient data and the second is the limited growth of the sample data. Because there are probably be 6~7 ways to apply for loans or credit cards, manual inspection is required in the end, so the data growth is very low. There will be a case below to explain the problem.

Therefore, we need to use relational graphs to crack the problems of information inconsistency. Both group and agency frauds have a batch of application behaviors. We can use the visualized graphs to recognize fraud samples. In this way, we may capture dozens to hundreds of fraudulent applications. It is also a very effective way for sample replenishment. Graphs can also be used for consistency checks. The multiple sources of information prove the information inconsistency. Graphs are displayed from the dimension of relations which helps us to do secondary verification. For example, multiple applications sharing the same information like the same phone number and IP address are considered group fraud.

To detect these group frauds, we use graphs that help us find rules of relations. These rules are complementary to the original expert rules. On the other hand, we can extract graph information to optimize machine learning patterns which increases the pattern's ability to detect group frauds and label more suspicious objects. Therefore, the expert rules, machine learning models, and graphs construct an effective ecological closed loop and complement each other to detect frauds. 

In terms of the specific applications of relational graphs, I've listed the following applications:

**_Application Graph_**: An example is using graphs to detect fraudulent applicants for loans and credit cards. One characteristic of a group is that information is shared by the group members. From the visual queries of graph data, a lot of correlations will be found. The more common information includes identity card, cell phone number, device fingerprint, e-mail address, etc. We use this information to explore relationships among these data, and then visually analyze the specific fraudulent techniques.**_‍_**

**_Transaction Graph_**: For multi-level transfer relationships, we mainly focus on the final flow of funds and who are the beneficiaries. The recent gambling fraud issue is the main focus of concern for financial supervisors, which is also one of the main directions we focus on. We have many customer cases. Through the list of suspects provided by the supervision of our internal rule models, suspicious transactions can be detected. In this process, we use historical flow data to explore more layers of data starting with the collected accounts. We may also use the graph exploration feature or a graph platform to analyze existing data for incremental prevention and control. 

**_Enterprise Graph/Internal Control Graph_**: Nowadays, there are interconnections between two or more enterprise businesses. The impact of external risks on the enterprise is increasing. Using graphs can outline the whole picture of risks for an enterprise. Combined with the introduction of external risks, the impact of external risks on the enterprise can be predicted as early as possible. Internal control graphs are more often used for detecting ethical or operational risks, such as misuses or failure to do certain operations in accordance with rules and regulations. Moral risks occur more often in unusual financial transactions between employees and enterprises, such as illegal fundraising or misappropriation of funds. The prevention core is to use graphs to check the actual control accounts of employees. Because bank employees are well acquainted with the bank's business. Few employees take their own accounts for some crimes. The identification of these control accounts can be explored by the means of graphs. For example, the accounts of relatives. We have a case of a joint-stock bank. In this case, the target was located in the account of the employee's girlfriend, because from visual graphs we found that many funds actually flowed to the girlfriend's account. It is very difficult to detect a case like this by traditional means.

**_Money Laundering Graph_**: The money laundering problem has existed for a long time. There are many derivative money laundering ways in different periods. Popular ways include the underground money bank and the "order sneaking" platform. Criminals hide their money on the platform where order-sneaking customers' daily consumption transactions are normal, however, there is actually a fixed pattern on visual graphs to detect cases like this. So how creating business relations with graphs and making graph rules to perform real-time monitoring and detections have become an important task for Bangsun, taken as a graph technology platform, to empower business developers. We will talk about it later. 

**_Vehicle Insurance/Ops Graph_**: For the vehicle insurance/ops field, we combine the initialization network with the design of vertices and edges, then query for a subgraph where the data is relatively appropriate and related, and then combine the experience of business experts to perform some exploratory and analytical work.

### **Use case 1: Flow of funds**
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure2.png)

Here, some specific application scenarios are also briefly listed. The first one is the flow of funds after the loan is approved.

We can focus on business and consumer loans through graphs, and track whether the approved loans flow into the housing market or the stock market. In this process, we will use the penetration ability of graphs to calculate the amount and proportion of transferred funds. Thereafter, we can easily find risks and violations and provide a basis for investigation. The regulators currently follow the principle of "three approaches and one guideline" for internet credits, so we establish real-time risk control through graphs and focus on the fund flows.

Next, we can show the specific methods. For example, loans flow directly into the blacklisted accounts or their own accounts are transferred to an account not under their name, but the account may be a controlled account. The account is used for investment and financial management. These can be analyzed with graphs and the specific pattern of the funds because the fund flows will form a very obvious distribution or pattern on visual graphs.

This pattern is difficult to portray with the previous expert rules, because the expert rules can only find a one-dimensional relationship, up to two-dimensional relationships, while the graphs are more adept at multi-dimensional relationship detecting. In massive transaction structures, we can also find some patterned structures. For example, the above figure shows 4-5 vertices, which involve some centralized transfer in, decentralized transfer out, and chain transaction structures like tree branches and ant nests, which are all abnormal funding patterns. Through visual graphs and combination with business knowledge, in many cases, we can see that it is problematic at a glance. As for where the specific problem is, we can check it through some functions provided by the graph, such as the k-degree query or the discovery algorithm of important nodes.

### **Use case 2: Insurance frauds** 
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure3.png)

The second use case is frauds in insurance, such as vehicle insurance or health insurance. The primary vehicle insurance issues are collusive claim frauds. By monitoring and associating shared identity information, it is easier to find group members and excludes irrelevant data. As for health insurance, we generally focus more on the improper relationship between doctors and patients or providers. Once there is a large number of discrete values on a drug or disease, it represents fraud. This is the common way to detect individual fraud in the past. Now the group fraud problem is severe. Group fraudsters have a shift in thinking that dozens of accounts buy the same drug through individual payment accounts. There are also aggregation accounts and many other commonalities. Combined with graph analysis and algorithms, those group frauds can be detected.

### **Summary: Advantages of graph fraud detection**
![](http://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/financial-fraud-detection-one-of-the-best-practices-of-knowledge-graph/figure4.png)

The advantages of graph fraud detection mainly include the following four aspects.

Association analysis: This means that people take the initiative to explore and discover suspicious features, and then use graphs to make some visual associations. Business developers take the initiative to use graph tools for visual exploration and then combine the features of different business areas, such as money laundering, credit card applications, and loan applications to find some suspicious points. The suspicious points are certainly different in different areas, but the business developers who often use graph tools have a certain sensitivity to such suspicious points.

Graph rules: This aspect is more for incremental monitoring. Unlike the previous expert rules, graph rules use the means of graph exploration associated with knowledge comparison to draw some conclusions. We conclude graph rules from the dimension of relationships, and then the system uses the rules to detect risks.

Pattern analysis: We recently called the same model, because many fraud patterns have a solidified pattern, such as cash-out from credit cards. After analyzing these solidified patterns, we convert the pattern in a graph query language into a graph library for a traversal query. Compared to the graph rules introduced above, it is a method for detecting existing data.

Community analysis: As mentioned above, insufficient data is lacking for modeling graph data, the community analysis is actually a very good feature to discover fraudulent groups by algorithms and combination with data labels.

The above four graph fraud detection advantages can help us detect frauds in real-time, on time, and afterward. They are both the main graph functions for anti-frauds and are considered our strengths. These advantages are currently applied by about dozens of our customers.

This is all we have for today's article! Please look forward to the next topic Introduction of Bangsun Knowledge Graph Platform and Practical Cases of State-owned Banks.

‍

A knowledge Graph specialist from Bungsun shares his best Practices of Knowledge graph in the Financial Fraud detection field.

Financial Fraud Detection: One of the Best Practices of Knowledge Graph

Real-time recommendation is a very popular topic in the retail industry. The goal of real-time recommendation is to provide personalized recommendations for users immediately after they search for products on e-commerce websites. It is a challenging task since the user has not yet decided which product they want to buy, so it requires a deep understanding of their preferences and behaviors.

In order to achieve this goal, real-time recommender systems need to be able to process big data at near real-time speed. Real-time systems are often used as part of e-commerce websites or mobile apps, where they can recommend products based on what the user is currently doing. For example, if an e-commerce site is selling shoes, it might want to show the user different styles of shoes when they are looking at the "men's sneakers" page.

Graph technology is a good choice for real-time recommendation. It has the ability to predict user behavior and make recommendations based on it. Graph databases like NebulaGraph provide a flexible data model that allows you to represent any kind of relationship between entities. This includes not only the typical "product" and "user" relationships, but also any other relationships that are important to your application. For example, you can use a graph database to represent complex groupings of users (such as "friends") or categories of products (such as "books"). As long as there's an edge between two entities, you can use graph technology.

The ability to represent any kind of relationship also makes it possible for graph databases to be more expressive than relational databases. For example, if you have a large number of users who share similar interests but belong to different groups, then in relational databases this might mean having a separate table for each group and then joining them together into one big table at query time. In contrast, in a graph database, it would be possible for each user to have multiple edges representing their interests in different groups.

# **What is a graph database?**

A graph is known as a diagram that illustrates a relationship between two things. Thus, a graph database can be assumed to be a database to understand relationships between data.

Graph databases are unlike traditional Relational Database Management Systems (RDBMSs) most people are familiar with. RDBMSs house data in a table of columns and rows with little or no relation to each other.

Graph databases are fundamentally designed with a focus on the relationship between data sets. So, they require more intense and specialized processing capabilities. As a result, graph databases like NebulaGraph use sophisticated design and architecture.
 ![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure1.png)


Graph databases are designed to uncover important relationships between many big data sets. In other words, it can connect the dots between multiple datasets that might otherwise just be sitting useless in silos. And graph databases are already widely used for such purposes.

# **What is a real-time recommender system?**

recommender systems are used to predict customer preferences and provide them with products or services that they might like. It helps companies reduce costs and increase revenues by providing personalized recommendations for their customers.

Recommender systems are used in a variety of industries including retail, media and entertainment, travel, and the public sector.

For example, Netflix uses an algorithm to recommend movies based on what you’ve watched previously. Amazon uses recommendations to help shoppers find new products they might be interested in purchasing. And Facebook uses its own algorithm to show you relevant ads based on your interests and likes.

# **Why should you use NebulaGraph for real-time recommendation?**

NebulaGraph is a highly performant linearly scalable graph database available for use via a shared-nothing distributed model. In several real production environments, data mining and technical performance has been proven to beat the performance of competing graph databases by multiple times over.

The goal behind NebulaGraph is to unleash the power of exponentially growing connected data.

NebulaGraph securely processes data sets of at least twice the size and twice as fast as any competing graph databases

*   It is the only database that can store and process billions of data points with trillions of relational connections
*   NebulaGraph is designed for scalability and recovery without disruption, ensuring the best business continuity available
   ![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure2.png)

# **How graph technology is used in recommendation**

Recommender systems aim to provide users with personalized recommendations based on their tastes and preferences. The characteristics of a user can be represented by their preferences for movies and books, for example, or their shopping habits on an e-commerce website. There are two main types of recommender systems: 1) collaborative filtering methods and 2) model-based methods.

A collaborative filtering method, such as KNN, can predict the movie rating without knowing the attributes of the movies and users. To address this challenge, the graph factorization approach combines the model-based method with the collaborative filtering method to improve prediction accuracy when the rating record is sparse.

Graph factorization methods have been widely used in many online recommender systems. Graph factorization is a graph-based model that can be used to represent user preferences as well as the relationship between users, items, and attributes. The goal of graph factorization is to extract the latent features from user ratings and recommendations so that these features can be used to predict users' preferences in an unsupervised manner.

Graph factorization is done by breaking down the original dataset into smaller datasets or clusters. This process can be done using graph databases because they are designed to support highly connected data structures and relationships between data points.

# **How to build a recommender system using NebulaGraph?**

If you want to build your own recommender system, there are many different approaches you can take. One of the most popular is using a graph database like NebulaGraph. This section will walk through how to build a recommender system using a graph database.

**Define the data model**

In this step, we define the data model for our recommender system. The first thing we need to define is what type of data can be recommended. For example, we want to recommend movies so we will use movies as our entity. The second thing is what kind of information do we need about a movie? For example, if you are building a recommender system for books, maybe you need to know what genre it belongs to or what language it was written in. But in our case, all we need is the title, the year of release, the genre, and the country. So our movie entity is defined like this:

‍
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure3.png)

‍

**Define relationships**

The next thing we need to do is define what type of relationship exists between movies and users (or people). There are two types of relationships that exist between movies and users: user likes movie and user has watched movie (or user has not watched movie). In this example, we will simplify it to just user likes movie.

Now imagine two users James and Kelsey, who are both fans of Sci-Fi Movie A, which has a genre of Science fiction. James also likes Sci-Fi Movie B, but Kelsey hasn’t watched it, so we don’t know whether Kelsey likes it or not.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure4.png)

But using this graph modeling of data, we may easily find out that Kelsey may like Sci-Fi Movie B. The recommender system would recommend Sci-Fi Movie B to Kelsey because James — who likes the same things as Kelsey — likes Sci-Fi Movie B.
![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/use-cases-of-graph-databases-in-real-time-recommendation/figure5.png)

**Making recommendations**

There are a few graph algorithms that you can use to make recommendations within a graph database.

**The PageRank algorithm**: The [PageRank algorithm](http://pi.math.cornell.edu/~mec/Winter2009/RalucaRemus/Lecture3/lecture3.html) is used to rank web pages in search results. The purpose of this algorithm is to determine which web pages should be displayed first when someone searches Google or any other major search engine.

The basic idea behind PageRank is that if you link to another page, then that page is more important than your own page. So if you are linking to a page about shopping for shoes, and someone else links to your shoe shopping site, then that person thinks your site is more important than it really is.

If everyone is linking to the same sites, then the people who link most get more weight.&nbsp;

Continuing with the movie analogy, while watching movies one can either carry on watching movies of a similar genre (and hence follow their most expected journey), or skip to a random movie in a totally different genre. It turns out that this is exactly how Google ranks websites by popularity using the _PageRank_**_ _**algorithm.

The popularity of a website is measured by the number of links it points to (and is referred from). In our movie use case, the popularity is built as the number hashes a given movie shares with all its neighbors.

**Collaborative filtering:** Collaborative Filtering (CF) is a method for recommender systems based on information regarding users, items, and their connections. Recommendations are made by looking at the neighbors of the user at hand and their interests. Since they are similar, the assumption is made that they share the same interests.

Read more: [Build a Recommendation Engine With Collaborative Filtering](https://realpython.com/build-recommendation-engine-collaborative-filtering/)

# Conclusion

In conclusion, it's worthwhile to use graph databases to build real-time recommender systems.&nbsp; Since graph databases are well designed for representing the relationship among users and products in the recommender system, they can be used to build real-time recommendation systems. Graph databases' support of powerful data analysis makes it useful in constructing recommender systems that need to take into account users and products with different preferences.

Why graph database is a good choice for recommendation engines? How to implement a recommender system using graph technology? How graph algorithms like PageRank and Collaborative filtering are used in real-time recommendation?

Use cases of graph databases in real-time recommendation

NebulaGraph, a leading graph database technology provider, today announced that it has joined the Linked Data Benchmark Council (LDBC) as an organizational member. LDBC is the official platform for international benchmarking of graph and RDF data.

NebulaGraph’s move to join the LDBC means it is taking another step toward participating in developing graph industry standards and specifications such as Graph Query Language (GQL), an upcoming International Standard language for property graph querying that is currently being created, as well as a series of linked data benchmarks. NebulaGraph is also to play a constructive role in data evaluation guidelines across multiple industries including finance, manufacturing, and internet of things.

“With the ongoing digital transformation, more and more enterprises around the world have large linked data that consists of hundreds of billions of vertices and edges. The LDBC sets global standards for evaluating graph database vendors’ ability to handle those data. NebulaGraph is proud to be part of the LDBC’s organizational members and we are excited about the prospect of working on global graph database standards and promoting the establishment of graph query language standards with the LDBC,” said NebulaGraph founder and CEO Sherman Ye.

The LDBC is a consortium comprised of industry-leading companies and organizations in the database management system industry. Their objective is to “establish an industry-neutral entity for developing graph and RDF benchmarks, auditing benchmark results, and publishing audited results.” 

The LDBC’s benchmark tests help graph database vendors identify weaknesses in their current architectures, identify problems in how they implement queries, and scale to solve common business problems. They can also help enterprises vet the performance of databases in a way that is relevant to common business problems they want to address.

Graph databases are involved in mapping the relationships between entities such as organizations, people, and transactions. Such technology could facilitate rapid contextualization for decision-making in 30% of organizations globally, according to market research firm Gartner.

NebulaGraph is an open-source graph database developed by Vesoft Inc. It is the world’s only large-scale graph database solution with a latency of milliseconds. Hundreds of enterprises around the world including Tencent, Meituan, JD Digits, and Kuaishou are already leveraging NebulaGraph technologies to boost their graph data processing capabilities.

# About Vesoft Inc.

Vesoft Inc. is the creator of NebulaGraph, the world’s most capable database for big data analytics discovery. NebulaGraph provides an industry-leading capability of storing and handling billions of vertices and trillions of edges, with just milliseconds of latency. Its shared-nothing deployment architecture removes any single point of failure and allows fast recoveries, enabling the industry’s best business continuity.

In June 2020, Vesoft secured $8 million Series Pre-A funding from investors led by RedPoint China Ventures and Matrix Partners China, which previously contributed as an angel investor. For more information, please visit [https://nebula-graph.io/](https://nebula-graph.io/).

NebulaGraph, a leading graph database technology provider, today announced that it has joined the Linked Data Benchmark Council (LDBC) as an organizational member. 

Posts

Tag