
> The author,Xin Hao, is a risk management  engineer at Airwallex and a maintainer of nebula-go. This article will introduce the applications of NebulaGraph at Airwallex.


#  Airwallex  

As a renowned global payment and financial platform, Airwallex provides one-stop cross-border payment solutions for enterprises of all sizes and stages of development.  

When shopping, gaming, subscribing to video platforms, or using paid software, you encounter various cross-border payment scenarios, including card payments or recurring subscriptions. These payment services are facilitated by game companies, e-commerce platforms, or merchants on other platforms.  Behind the scenes, Airwallex supports these enterprises by offering services such as payment acceptance, fund collection, and treasury management.  

![](https://nebula-website-cn.oss-cn-hangzhou.aliyuncs.com/nebula-website-5.0/share/Airwallex/1744943392895.jpg)

In cross-border scenarios, our clients are highly diverse.  On one hand, some clients are gaming companies providing in-game purchase products to users across different countries. Others are e-commerce platforms, including self-operated platforms and those like Taobao that serve multiple merchants.  On the other hand, our clients extend beyond the platforms themselves to include the merchants on these platforms. These merchants are legally independent entities, necessitating clear and precise identification.  


#  Application Scenarios of NebulaGraph  

We categorize NebulaGraph’s use application scenarios into four main areas:  

## Know Your Customer (KYC)  

It is well known that cross-border operations involve unique legal frameworks across jurisdictions. Commercial entities are the fundamental legal units, and we must classify and link these entities to ensure compliance with local regulations and business scopes.  

1. **IPersonal Information Association:** 
    - Authorized Representatives and Legal Persons of Commercial Entities: Their information must be linked to the entities to identify who actually controls and manages them.  
    - Operators of Registered Accounts: When individuals register and use our services, their email addresses, phone numbers, and other details are associated with their accounts, aiding identity verification and risk assessment when needed.  

2. **Device Information Association:** Device information association involves client devices, such as IP addresses and device IDs. This helps detect abnormal login behaviors, such as access from unusual geographic locations or devices.  

3. **Network Information Association:** Network information association covers clients’ online behaviors and transaction patterns. By analyzing network activities, we identify anomalous transaction patterns or potential risks—for instance, frequent large transactions or multiple transactions from different locations within a short timeframe.  

4. **Facial Recognition Association:** Facial recognition association leverages biometric technology to verify client identities.  

## Real-Time Transaction Correlation  

During real-time transactions, NebulaGraph is used to correlate transaction data.  

Device information association here differs from the KYC context. Here, device data originates from payment components integrated with Airwallex SDK. When users initiate payments via Airwallex SDK (mobile, web, etc.), we track device IDs, models, IP addresses, and other details.  

Payment and collection information association ensures transaction legitimacy and security. In e-commerce, billing information association is key to detecting abnormal purchase patterns and potential fraud, while logistics information association tracks product flows and ensures transaction integrity.  

## Model Feature Extraction  

For model feature extraction, raw data is preprocessed into features used for model training and real-time inference. During online graph service operations, periodic data snapshots are generated for training, while real-time interfaces compute features for inference.  

## Visualization and Interaction  

Our current implementation is simplified, primarily serving internal operations teams. We provide web interfaces for operators to input queries and interact with the system on demand.  

# Why NebulaGraph?  

## Graph Relationships and KYC  

Know Your Customer (KYC) is a critical risk control process to verify client identities and assess risks throughout their lifecycle. It ensures compliance and prevents fraud by validating identities, understanding financial behaviors, and detecting anomalies.  

Graph databases naturally model KYC relationships. By representing clients, accounts, devices, and transactions as nodes and edges, NebulaGraph visualizes complex connections—such as legal-entity associations or account-device links. This structured approach enhances data readability and enables efficient anomaly detection, such as identifying fraudulent chains through relationship analysis.  

Thus, adopting NebulaGraph was the best choice.  

## Challenges  

Before NebulaGraph, we used a simple Python-based graph service for basic applications. During our early stages with smaller datasets, we extracted nodes and edges from offline data warehouses and stored them in memory.  


This architecture worked initially but faced issues as data grew:  
1. **Instability:** Periodic data updates every 10 minutes caused frequent failures.  
2. **Memory Inefficiency:** Increasing data volumes strained memory resources.  
3. **Limited Query Flexibility:** Basic query languages hindered complex scenario support, reducing development efficiency.  

These challenges drove us to adopt NebulaGraph.  

## NebulaGraph Selection  

During evaluation, we benchmarked multiple graph databases. Despite some competitors’ comparable performance, NebulaGraph excelled in functional completeness and enterprise adoption, making it the optimal choice.  

# System Architecture Design  

## Customer Relationship System Architecture  

### Graph Construction for Customer Relationships  


We built a heterogeneous graph, offering three key advantages:  
1. **Adaptability:** Supports diverse query scenarios, such as analyzing account linkages or transaction patterns.  
2. **Scalability:** New relationship types can be added effortlessly as business evolves.  
3. **TTL-Based Data Cleanup:** Automatic expiration of outdated data ensures relevance.  

Over time, we enriched the graph with additional relationship types, allowing granular analysis via edge attributes and weights.  

### System Architecture  


1. **Data Ingestion**  
   - Batch Processing (Full Data): Hourly bulk updates.  
   - Real-Time Data Supplement (Incremental): APIs fill missing data during new client registration.  

2. **Schema-First Approach**  
   - **Schema as Code:** Field definitions and changes are codified, ensuring consistency.  
   - **Unified Schema:** Shared across services and ETL pipelines for efficiency.  

Key applications of the graph service include:  
1. Model feature extraction for real-time client and transaction analysis.  
2. A rules engine for risk control decisions based on graph features.  
3. Internal business services leveraging graph data.  
4. Visualization interfaces for operational queries.  

## Business Complexities  

1. **Dynamic and Complex Schemas**  
   - Multiple node types (10+ variants) with diverse attributes.  
   - Frequent schema adjustments (monthly).  

2. **Data Logic Changes and Cleansing**  
   - Requires flexible workflows to accommodate rapid business shifts.  

### Mitigation Strategies  

1. **Centralized Schema Definition**  
   - Define schemas in code, auto-generating changes upon commits.  
   - Field modifications (renaming, additions) are code-driven and automated.  

2. **TTL + Full Data Writes**  
   - Combines TTL for data freshness with full writes for completeness.  

We adopted a declarative schema management approach. Field modifications trigger auto-generated schema change statements.  


For query result mapping, we use a "declare-and-get" method. Structs defined in code with annotations automatically map query results to fields, eliminating manual handling. SDKs or APIs dynamically fetch the latest schema, ensuring stability despite changes.  

## Transaction Correlation System Architecture  


This system prioritizes real-time performance, adopting a "compute-separate, merge-on-read" architecture.  

- **Historical Data (>4 hours):** Pre-processed via hourly batch jobs into subgraphs stored in the graph database.  
- **Real-Time Data (≤4 hours):** Ingested via Kafka, split, and written to the graph database after processing.  

# Future Outlook  

- **Resource Isolation:** Single Nebula clusters lack isolation, causing interference between applications during query peaks.  
- **TP/AP Compatibility:** A single cluster struggles to balance transactional processing (TP) and analytical processing (AP).  
- **Algorithm Ecosystem:** Limited openness and ecosystem for graph algorithms.  

Future improvements aim to enhance resource isolation, enrich graph algorithm support, and foster community collaboration to advance the technology and ecosystem.  

This article details Airwallex's use of NebulaGraph in cross-border payment scenarios. It covers application areas like KYC, real-time transaction correlation, model feature extraction, and visualization. 

How Airwallex Leverages NebulaGraph for Cross-Border Payment Solutions

In the digital age, recommendation systems have become an integral part of online experience, driving better user experience and engagement. A recommendation system, often integrated with ad serving systems, is designed to suggest highly relevant content or advertisements to users based on users’ profile and engagement history. However, the complexity of user interactions and preferences demands robust data management solutions. Enter graph databases—an innovative approach that enhances the efficiency and accuracy of recommendation systems. This blog explores how recommendation systems work and how graph databases empower recommendation systems, providing real-life examples and architectural insights.

# What are Recommendation Systems and How They Work
## Key Components of Recommendation Systems:
- User Data: Information about users, including their demographics, behavior, preferences, and past interactions. This data is crucial for understanding user interests and making personalized recommendations.
- Content Data: Information about the content or ads available for recommendation. This includes metadata, categories, keywords, and performance metrics.
- Recommendation Algorithms: The core of the system, these algorithms analyze user and content data to generate personalized recommendations. Common types include collaborative filtering, content-based filtering, and hybrid methods.
- Real-Time Processing: The ability to process data and generate recommendations in real-time, ensuring that suggestions are timely and relevant.
- Feedback Loop: Mechanisms to collect user feedback on recommendations (e.g., clicks, likes, shares) to continuously improve the accuracy and relevance of the system.

## Types of Recommendation Systems
Recommendation systems are powerful tools designed to analyze user behavior and preferences to deliver personalized content and suggestions. These systems operate by collecting extensive data on user interactions, such as browsing history, ratings, and purchase activities. Using this data, recommendation systems employ sophisticated algorithms to identify patterns and relationships within the data. 

Two common approaches are collaborative filtering and content-based filtering. Collaborative filtering analyzes the behavior and preferences of similar users to make recommendations; for example, if two users have a history of liking similar movies, the system might suggest a movie liked by one user to the other. Content-based filtering, on the other hand, focuses on the attributes of items and recommends items that are similar to those a user has previously enjoyed. By combining these methods, recommendation systems can generate highly personalized suggestions that cater to individual tastes and preferences. This tailored approach not only enhances the user experience by presenting relevant content but also drives higher engagement and satisfaction, ultimately benefiting both users and businesses.

# What are Graph Databases
## Definition and Structure
Graph databases are a type of NoSQL database designed to handle and represent complex relationships between data entities. Unlike traditional relational databases that use tables to store data, graph databases use nodes, edges, and properties to create a flexible and highly interconnected data structure. Nodes represent data entities, edges denote the relationships between these entities, and properties provide additional information about the nodes and edges. This structure allows graph databases to efficiently model, store, and query intricate relationships and dependencies within data. Graph databases are quite young compared with relational databases and well-known graph databases include Neo4j, NebulaGraph, Amazon Neptune, and TigerGraph.

They excel at managing complex relationships because their inherent design closely mirrors real-world networks, such as social connections, organizational hierarchies, and financial transactions. This enables rapid traversal and querying of relationships, providing powerful insights and enabling applications like fraud detection, recommendation systems, and network analysis to operate with high performance and scalability.

## Architectural Insights

1. Data Modeling:
    * Nodes: Represent users, items, categories, etc.
    * Edges: Capture interactions such as likes, views, purchases, etc.
    * Example: In a movie recommendation system, nodes could represent users and movies, while edges represent viewing history and ratings.
2. Querying:
    * Graph Traversal Queries: Efficiently find relevant items by traversing the graph.
    * Example: To recommend books, a query might traverse from a user node to books they’ve rated highly, then to other users who rated those books highly, and finally to other books those users have rated highly.
3. Performance Optimization:
    * Indexing and Caching: Improve query performance by indexing frequently accessed nodes and caching results.
    * Example: Caching the results of common recommendation queries can reduce latency and improve user experience.

# How Graph Databases Support Better Recommendation Efficiency and Accuracy

Graph databases, such as [NebulaGraph](https://www.nebula-graph.io/), are particularly well-suited for handling the complex relationships and interconnected data involved in both ad serving and recommendation systems. Here’s how they support efficiency:

1. Complex Relationships and Connections:
    * Graph Structure: Graph databases store data in nodes (entities) and edges (relationships), making it easy to model and query complex relationships.
    * Example: Users, ads, and their interactions can be represented as nodes, with edges capturing relationships like views, clicks, and purchases. [LinkedIn](https://www.linkedin.com/) uses graph databases to model professional connections, endorsements, and job recommendations. By understanding the intricate relationships between users, companies, and skills, LinkedIn can provide highly relevant job suggestions and networking opportunities.
2. Real-Time Recommendations:
    * Fast Traversal: Graph databases excel at traversing relationships quickly, enabling real-time generation of recommendations based on user behavior and preferences.
    * Example: Quickly finding similar users or content based on shared attributes or interactions. [Pinterest](https://www.pinterest.com/) leverages graph databases to recommend pins based on user interactions and the connections between pins, boards, and users. This allows Pinterest to deliver personalized content in real-time, enhancing user engagement.
3. Personalization and Targeting:
    * Rich User Profiles: Graph databases can store and query detailed user profiles, including their connections, behaviors, and preferences, allowing for highly personalized ad targeting.
    * Example: Identifying users with similar browsing patterns and recommending ads that performed well with similar users. [eBay](https://www.ebay.com/) uses graph databases to improve its recommendation engine by analyzing user behavior, purchase history, and item relationships. This enables eBay to provide personalized product suggestions, increasing the likelihood of purchases.
4. Scalability:
    * Handling Large Datasets: Graph databases are designed to handle large volumes of interconnected data, making them suitable for high-traffic environments.
    * Example: Managing millions of users and ads, and their interactions, without performance degradation. [Facebook](https://www.facebook.com/)’s social graph, which models the relationships between billions of users, posts, and interactions, relies on graph databases to provide scalable and efficient recommendations. This ensures that users receive timely and relevant content, even as the platform grows.
5. Data Integration and Enrichment:
    * Combining Data Sources: Graph databases can integrate data from various sources, enriching user profiles and improving recommendation accuracy.
    * Example: Combining browsing history, social media activity, and purchase history to create a comprehensive user profile.
6. Feedback Loop and Continuous Learning:
    * Adaptive Algorithms: Graph databases facilitate the continuous updating of user profiles and recommendation algorithms based on real-time feedback.
    * Example: Adjusting recommendations based on recent user interactions to improve relevance.

# Conclusion
In summary, graph databases enhance the efficiency of ad serving and recommendation systems by providing a powerful way to model, store, and query complex relationships. They enable real-time personalization, effective targeting, and scalability, ultimately leading to better user engagement and optimized ad performance. As demonstrated by industry leaders like LinkedIn, Pinterest, eBay, and Facebook, leveraging graph databases can significantly improve user engagement and satisfaction.


This blog explores how recommendation systems work and how graph databases empower recommendation systems, providing real-life examples and architectural insights.

From Data to Insight: Leveraging Graph Databases for Superior Recommendations

Imagine navigating through a complex web of interrelated parts in a large manufacturing factory's supply chain. From procurement to customer service, each element is interconnected, and a delay in one can create a ripple effect across the entire chain. This blog post explores how graph databases, with their unique structure, can simplify this process and bring significant efficiency improvements, saving both time and costs in supply chain management.

# Understanding Supply Chain Management and Key Factors to Its Efficiency
The supply chain is the network of people, organizations, resources, activities and technology involved in the production and sale of a product from the supplier delivering raw materials to the manufacturer to the end user. It’s all the steps to get a product or service from supplier to customer.

Supply Chain Management (SCM) is the active management of the supply chain to maximize customer value and gain a sustainable competitive advantage. It involves the coordination and collaboration with channel partners, which can be suppliers, intermediaries, third-party service providers, and customers.

![image](https://media.licdn.com/dms/image/C4E12AQFs6QMs0h6WFg/article-cover_image-shrink_720_1280/0/1647752062480?e=1726099200&v=beta&t=N2VbMBySvyWfLg28ATvNgXTqgjJEiYJ9NDie9S99JEE)

*Image source: https://www.linkedin.com/pulse/what-supply-chain-management-its-importance-dhruman-gohil/*


Some of the key factors to a well-managed supply chain are:
- Demand Forecasting: Accurate forecasting of demand is crucial as it impacts all other supply chain planning activities including production, inventory management and shipping.

- Inventory Management: Keeping the optimal level of inventory is a delicate balance; too much will lead to high holding costs while too little leading to stock-outs and lost sales.

- Technology Integration: SCM requires robust technology systems that can automate and streamline the processes involved from order processing and production scheduling to inventory control and customer service.

- Flexibility: The supply chain should be able to respond quickly and efficiently to changes in demand or supply.

- Visibility: A transparent supply chain helps companies identify inefficiencies or problems quicker and take action sooner.
Risk Management: Identifying and managing risks such as supply disruptions, demand volatility and currency fluctuations is also key to SCM.

- Sustainability: Increasingly, companies are looking at the sustainability of their supply chains in terms of environmental impact, social responsibility and economic performance.

# Understanding Graph Databases and Their Distinction from Traditional Databases
A graph database is a unique type of database designed to handle complex data relationships. It’s a NoSQL database that uses graph theory to store, map and query relationships. A graph database is fundamentally different from traditional databases, for example relational databases, because of the way it treats data relationships. In traditional databases, relationships between data are often secondary. They are inferred through JOIN operations that link together separate tables based on common attributes. This can lead to performance issues when dealing with large amounts of complex and interrelated data.

But graph databases, such as NebulaGraph, take a different approach. They put relationships first, storing data in a flexible graph structure of nodes and edges. Nodes are entities like a person or a product, edges are the relationships between those entities. This structure allows for high performance when querying relationships making it great for managing complex connected data.

![image](https://user-images.githubusercontent.com/57335825/100973399-57cfdb00-34ef-11eb-9c12-262e4922cac1.png)

# How Graph Databases Drive Efficiency in Supply Chain Management

In a large manufacturing factory, the supply chain is a spider’s web of interconnected parts. From procurement and production to distribution and customer service, each step is connected. A delay in procurement can impact production schedules which can then impact distribution and delivery times.

Think of a car manufacturing plant. A car is a complex machine with thousands of parts, each with its own supply chain. If one part fails, it could be a bigger issue in the supply chain that needs to be fixed.
With a traditional database it’s hard to understand these ripple effects. The tabular structure of traditional databases makes it difficult and time-consuming to follow the impact of a change through the entire supply chain. But with a graph database it’s much simpler. By clearly defining the relationships between different aspects of the supply chain, a graph database allows for quick and easy navigation through the data. This can lead to significant efficiency improvements, saving both time and costs.

Here are some ways graph databases can support enterprise supply chain management:
- Revealing Complex Relationships: Graph databases can show complex relationships between data, like materials, suppliers, production equipment and personnel in the supply chain. That’s very useful for optimizing production, reducing waste and improving efficiency.
- Tracking and Forecasting: Graph databases can help companies predict potential issues, like supply chain disruptions and equipment failures, by tracking every link in the production chain in real time. So companies can take action before they lose.
- Decision Support: By analyzing supply chain data, graph databases can support company decision making. For example, they can predict future demand based on historical data so companies can make better production plans.

To elaborate, graph databases, with their ability to show relationships between different data points, can give you a complete view of the entire supply chain. This enables companies to identify bottlenecks, inefficiencies, or risks, and take corrective action. Also they can help companies respond faster to changes or disruptions in the supply chain. Furthermore, by analyzing and forecasting based on historical data you can improve demand planning and inventory management. Lastly, the graph structure itself ensures data access is controlled and secure so your supply chain data is protected.

# Graph Databases Benefiting Different Departments in the Manufacturing Sector
The benefits of graph databases are not limited to a single department within the manufacturing industry. In fact, they can be leveraged by various departments to improve efficiency and decision-making.

Production planning can optimize production schedules by using graph databases, for example. They can create schedules that are more efficient in minimizing downtime and increasing productivity by understanding the relationships between different parts of the process. 

Procurement department can use graph databases to find the best way to get materials. They can make better procurement decisions by looking at the relationships among suppliers, costs, delivery times, and other factors. 

The quality control department can also use graph databases. By tracing the origins of quality issues, they can identify the responsible parties and take corrective action. This can lead to improved product quality and customer satisfaction.

# Conclusion

Graph databases are great for managing complex connected data in big factory supply chain management. They can help you detect and predict failures, optimize and decide. By detecting and predicting failures, optimizing and deciding they can deliver big efficiency gains.

Graph databases are great for managing complex connected data in big factory supply chain management. This blog post introduces how graph databases empowers supply chain management to be efficient.

How Graph Databases Drive Efficiency in Supply Chain Management

According to a report published on [Statista](https://www.statista.com/topics/5157/database-software/), the Database Management Systems (DBMS) market is worth $80 billion in 2023. This figure is expected to grow much faster as more businesses begin to do more with their data. 

Social networking apps are some of the greatest consumers of data, and so the need for robust databases is ever important in this industry. Social media apps need databases to store user information, distribute content, and enable interactions between online communities. 

This guide explores choosing the right database for a social networking app and the best practices for creating the ideal database design for social media. 

# Databases for a social media

In most cases, you will most likely end up using a combination of databases to suit specific features and requirements. 

Having said that, these are the most common types of databases that programmers use when working on social apps:  

## 1. Relational databases

A **relational database** uses tables and rows to store data. It is ideal for information that has a standardized, consistent structure. 

Examples of such information could include customer contacts (first name, last name, phone number, address, etc). Examples of relational databases include MySQL, PostgreSQL, and Microsoft SQL Server. 

Also Read: [Relational vs Non-Relational Databases](https://www.nebula-graph.io/posts/difference-between-relational-and-non-relational-database)

## 2. Document databases 

A document database stores unstructured data that does not fit into traditional relational databases. It is a type of NoSQL database, meaning it does not use Structured Query Language to retrieve data. 

Document databases are ideal for user-generated social data like user chat messages, comments, photos, and videos. Examples of document databases are MongoDB, Firebase Realtime Database, and CouchDB. 

## 3. Graph databases

A [graph database](https://nebula-graph.io/) stores data as connections rather than rows and tables. It is ideal for data that has complex relationships, e.g., friends, followers, likes, and shares on a social network. These databases enable social media apps to recommend new friends, pages, or products. Examples of graph databases include NebulaGraph, Neo4j, ArangoDB, and OrientDB. 

Learn more about the kind of result to expect from a graph database in action for a social network - the case of [NebulaGraph and WeChat](https://www.nebula-graph.io/posts/nebula-graph-for-large-social-network). There are more [use cases](https://www.nebula-graph.io/posts/social-networks-with-graph-database-2) to consider. 

## 4. Key value databases

A key value database stores data that has unique identifiers, e.g., time stamps. In a social networking app, these identifiers or keys can be login sessions or notification preferences. Examples of key-value databases include Redis, DynamoDB, and Memcached. 

**Note:** Top social networks like Facebook, Instagram, LinkedIn, and YouTube use all the above databases in some capacity. For example, [Facebook ](https://research.facebook.com/publications/research-area/databases/?s)uses the following types of databases: 
* [RocksDB](https://rocksdb.org/) for data storage
* [Velox](https://engineering.fb.com/2023/03/09/open-source/velox-open-source-execution-engine/) for data management
* [MySQL database](https://www.mysql.com/) for storing all social data
* [Apache Hadoop](https://hadoop.apache.org/) for big data, and more.

This approach is essential for social networks because their databases receive billions of requests per second (RPS). These platforms also generate terabytes of data every day, meaning that they need a variety of databases to sort, index, and secure user data. 

# Choosing a database for a social media app: factors to consider 

Since social media apps have numerous features, they require a mix of databases to support their functions. 

For example, one database can store user authentication data while another stores their posts and preferences. 

To make sure you are using the right database for your social app, please consider these items: 

## 1. Data volume

A social app generates huge volumes of data, e.g., profiles, comments, images, videos, shares, etc. The database you choose for your app should support this data securely and efficiently. 

## 2. Update frequency 

Social apps require real-time notifications and updates, which means that they must be fast and reliable on desktops and mobile devices. 

Key-value and document databases are best suited for frequent updates because they allow users to easily add, delete, or edit their data. Relational databases may prove less flexible and difficult to update quickly. 

## 3. Privacy and security

When choosing databases for a social network system, it is important to balance between data security and app performance. 

Pick a combination that will provide data protection and compliance while maintaining a good user experience. This enables the social media app to encrypt, authenticate, and back up user data without compromising on speed. 

## 4. Scalability 

Social networking apps scale in four main ways: 
* New features and functionalities
* Increased number of sign-ups
* Increased traffic to the app or website
* Increased user generated content (user chat messages, photos, videos, links, etc.)

The social network system should therefore support horizontal scaling, which means adding more resources like servers to manage growth. The social app databases should be able to manage increasing capacity without slowing down or compromising data security.
A graph database with horizontal scaling capabilities is best suited for this. NebulaGraph is great in this aspect. 

## 5. Supportive community

Of course, you will always need a strong community around the database option you are going to choose. A strong community means you will always be able to get the help you need whenever you get stuck somewhere, which is normal in any development set up. 

Please note that a community is both the people running the database as well as the developers who use it. When these two sides are strong and committed, you will have a smooth time working on the app. This is particularly critical for open source solutions. 

# Best practices for designing social media databases

We define database design as the process of modeling data to work with a database management system. 
This can be a complex task that needs careful planning because of the various types of data that a social app generates. It also affects the budget and resources that developers can allocate to creating social networking apps. 
Since each type of database presents unique pros and cons, please apply the following best practices. 

## 1. Prioritize simplicity

Whatever database system you choose to go with, the frontend should be as user-friendly as possible. This means using easy naming conventions for all pieces of data.

In addition, a social network system is an ever-changing application. This means that end users and backend development teams should be able to modify it easily. 

An example of database simplicity is creating a simple user message table for storing messaging data. It should contain basic attributes like the sender’s first and last name, time stamps, and message body. These are enough to create a user-friendly messaging system for a social app. 

## 2. Normalize the data

Data normalization means that all data entries in a database are similar across the social network system. 

This creates a standardized data format that links tables or graphs together, making it easy to query and update the database. When data is normalized, it addresses the following challenges of database design: 

* Redundancy: This means storing the same piece of data in multiple separate locations, which leads to duplications and errors. 
* Dependencies: This means the relationships between data attributes in a database, e.g., linking a username to its owner’s location or transaction history. Dependencies must be verified to maintain data integrity. 
* Anomalies: This means errors that occur when adding, deleting, or updating a database. Normalized data helps to avoid these errors.

 

## 3. Consider long-term usage

Database design needs to consider the lifespan of the social network. Once past the beta phase, the social app database must adapt to changes in internet connection speeds and new use cases going forward. 

Cultivate the habit of documenting each stage of the database design process. You should also create diagrams of the specialized databases for future reference. 

This may seem tedious, but it enables the social app to grow and adapt consistently over time. It also gives insights into how the app will use computing resources at scale as more users sign up. 

## 4. Prioritize collaboration

While automation is an essential part of database design, database maintenance requires a team of experts that collaborate at every step. The ideal database backend development team comprises of: 

* A database engineer
* A business analyst
* A solutions architect
* A database administrator

Most programmers often make the mistake of bringing in Database Administrators towards the end of the design. Please avoid this mistake. The DBAs are critical for aspects such as naming conventions, which developers need in the early phases.

Bear in mind  that all these experts can be a significant human resource investment which can also involve intense training and outsourcing when the need arises. It also includes the licenses for database software and tools that the team will need. 

## 5. Use models to test the database

The database design phase determines the success of the social network app. Therefore, it is critical to take time and test it using different database models. 

All stakeholders should be involved during the testing process to encourage constructive feedback. Database modeling has several advantages for social networking apps, including: 

* Reducing maintenance requirements
* Managing costs
* Visualizing databases that best fit the desired use case
* Optimizing databases before deployment
* Discovering bottlenecks or other performance challenges, etc.

It is important to dedicate time and resources to modeling and testing the databases upfront, rather than deploying and fixing issues later. This ensures that the database for social media app works as desired and requires minimal maintenance.  

# Conclusion 

Before choosing any database for your social media app development, it is important to begin with the clear goal of creating the app in the first place.

If creating a photo and video sharing app, for example, document databases will work well because they are good at storing unstructured data like media files. If developing a music streaming app, key-value databases are ideal for creating playlists, updating track details, and recommending new bands and genres. 

Similarly, a social blogging app is best served by graph databases. This is because these databases can connect users’ posts to their followers, organize timelines, and follow topics and trends.

Social networking apps are some of the greatest consumers of data, and so the need for robust databases is ever important in this industry.

How to choose a database for a social media app

The [2023 State of Fraud report](https://www.signifyd.com/ecommerce-fraud-prevention-best-practices-statistics-trends-report/%C2%A0) by Signifyd indicates that cases of online fraud are increasing at an alarming rate. E-commerce sites, for example, experienced a 71% increase in bot attacks, and fraudulent orders grew by 34%. 

Even worse, scammers are now using new technology like artificial intelligence (AI) to automate fraud. This is bad news for any business, regardless of size or industry. Our recommendation for decision makers like yourself, is to start with the HOW. 

How do you safeguard your assets when fraudsters are adopting new technologies to attack traditional security methods?

Not to worry, graph analytics is filling the void that has overwhelmed the traditional approaches. This superior method of identifying and preventing online fraud is already working for the organizations that have embraced it. You too should make the switch. 

Let’s understand why this is important. 

# What is graph analytics?

To understand graph analytics, it is important to briefly visit how graph databases work. A typical graph database, whether a fully commercial or [open source graph database](https://nebula-graph.io/), stores data as connections rather than tables and indexes. Each database has nodes which represent different data points. Examples of data points can be details like phone numbers, addresses, or devices. Each data point is then linked to one another by a line, also called an edge. These lines can represent data transfers, phone calls, email messages, or other information linked to the nodes. Therefore, a graph database looks like a connected network. 

Now, graph analytics is the technique of studying these connected data points to find relationships and patterns. Through graph analytics, analysts can see how different data points interact within a network. It reveals the strength of each interaction and how information flows throughout the network. 

Graph analytics also prioritizes the context around data points. It assumes that no data point acts in isolation, and that certain data points can influence the rest to varying degrees. This is different from traditional analytics that calculates averages and frequencies rather than how data points affect each other. 

From what we have been seeing, fraudsters share information with each other through platforms in places like the dark web. An example of such information could be how-to tutorials for new hacking and scamming techniques. They also tend to cooperate with each other to maximize on lucrative or high-stakes targets. 

While the fraudsters’ activities may be difficult to trace, their behaviors in the real world are not. They show patterns that graph analysis can detect. This is why graph analytics have become crucial in fraud detection. 

# Strengths of graph analytics in fraud detection
Graph analytics is well suited to fraud detection because of the nature of fraudulent activities. Some online scammers act alone, but most frauds are committed by a group of people, or a fraud ring. 

A typical fraud ring may comprise a handful of individuals or thousands of members across the world. Fraud rings can target a wide range of businesses, e.g., banks, retailers, insurance companies, currency exchanges, etc. The scammers typically use different techniques to hide their identities and locations, making them difficult to track. But they cannot hide from graph analytics. 

Using machine learning algorithms, graph analytics can identify a fraud ring and reveal its connections. For example, it can spot fraudulent activities in credit card transactions or wire transfer records. Graph analytics empowers investigators to see how fraud rings operate. 

Here is a brief breakdown of the specific strengths that make graph analytics superior:

* The visual representation of graph data is more intuitive to interpret than charts or spreadsheets
* The hidden relationships between fraudulent profiles and activities become clear
* Machine learning algorithms reduce the manual tasks of scanning and analyzing data
* Fraudulent actors are identified faster and more accurately based on real-time data.

# How a typical graph data model in  graph analytics works (for fraud detection)
When used in fraud detection, a graph data model acts as a virtual detective that finds evidence of fraud. It links individuals, transactions, and financial institutions involved in crime. Then, it compares these connections with previously identified fraud patterns to find similarities or anomalies. 

Let's consider a money laundering pattern as an example. One person wants to transfer large amounts of money without being detected by banking systems. The recipient also wants to get the funds without detection. The sender usually divides the funds into smaller portions. They also open several accounts with different banks to set up the transfers. Similarly, the recipient prepares several accounts to appear as though they are receiving payments from different clients. 

At a glance, these transactions may appear normal and continue to flow without problems. But with graph analytics, the connections between the sender and recipient become clear. The graphs will show how the money flows from a single sender to a single recipient through multiple channels. This is a common pattern of money laundering, where money generated by criminal activities appears legitimate. 

Graph analytics can also determine if such fraudulent activities are expanding rapidly, e.g., new criminal rings emerging in unexpected locations. It reveals shared data points, e.g., the same devices or email addresses accessing multiple banks or websites. 

In terms of e-commerce fraud, graph analytics can alert businesses about frequent chargebacks or transaction reversals from specific customers. These patterns indicate that the customers intend to keep purchased goods without paying for them, which is a criminal offense. 
# The key approaches that graph analytics employ to identify fraud
There are four approaches that data scientists use to identify fraud using graph analytics, as we’ll explain below: 

## 1. Querying 

Graph databases require a graph query language to extract and modify data. An example of a query language is NebulaGraph’s [nGQL](https://docs.nebula-graph.io/3.0.0/3.ngql-guide/1.nGQL-overview/1.overview/). These are specialized programming languages that enable graph analysts to ask questions and derive valuable answers from their databases, such as: 

* How many individuals are transacting with a known fraudster?
* How far apart (in miles) are transactions of $10,000? 
* How many of those transactions were conducted at 3AM in the morning?

Using such questions, graph analysts can identify any data points of interest and determine how they are related. The queries can be as detailed as possible, e.g., differences between a shipping address and a billing address on specific orders. They can also analyze the contents of a checkout basket to find anomalies in purchasing patterns. These patterns may be difficult to see without graph technology. 
## 2. Machine learning
The specific form of machine learning that is utilized in this context is what we call supervised machine learning. So what is supervised machine learning? This is a process that classifies the data used to train a graph analytics model. This means assigning labels to the training datasets and allowing the machine learning model to learn and adapt. 

A common example of supervised machine learning is image recognition. This is where data scientists label the training images, e.g., photos of a bus. Then, the model is shown photos of all kinds of vehicles (planes, cars, trains, etc.) and learns how to identify a bus. 

In fraud detection, graph analysts can label training data as fraud or non-fraud. For example, a fraudulent transaction can be tagged and fed into the ML model. A tag can be a particular tax haven location which signals fraud. Using this tag, the model can analyze all transaction data and pick out all matching incidents. 

This approach is particularly helpful to track money laundering activities. Data scientists can tag the training datasets with parameters such as: 

* User identities
* Frequency of purchases
* Transfer amounts
* Internet Protocol (IP) addresses, etc.

## 3. Graph Algorithms
A graph algorithm is a mathematical tool used to detect patterns in graph databases automatically. It can identify specific qualities and connections between data points in ways that graph queries might miss. Some examples of graph algorithms include: 

* Centrality algorithms: These detect the most influential individuals or data points in a network, e.g., accounts with the most frequent chargeback activities on a payment platform.
* Community detection algorithms: These identify data points that have the most connections, e.g., transaction patterns that show signs of a fraud ring. 
* Similarity algorithms: These identify data points that have matching qualities, e.g., clusters of accounts that are linked to the same email addresses. 
* Link prediction algorithms: These predict connections between two data points, e.g.,how likely a suspicious account activity will turn out to be fraud, based on established patterns.

There are numerous other algorithms that apply to graph analytics. Data scientists can customize these algorithms to match specific use cases and derive detailed insights. This means that they can keep up with changing fraud tactics before they impact an organization.
## 4. Real-time analytics
Some types of fraud are extremely time-sensitive, such as phone-based scams targeting senior citizens. With graph analytics, such types of fraud can be detected in real time. For example, phone communications can be scanned in real-time based on the scam callers' patterns. 

The calls are automatically redirected to a fraud detection team. The call recipient can also see an alert on their devices saying that the caller is a potential scammer. 

# Conclusion

According to the [Federal Bureau of Investigations (FBI)](https://www.fbi.gov/contact-us/field-offices/springfield/news/internet-crime-complaint-center-releases-2022-statistics), the Internet Crime Complaint Center (IC3) received fewer scam reports in 2022. However, the value of the money lost in these reports increased almost by 50%. 

This increase is attributed to an emerging trend where scammers are targeting citizens who are over 60 years old, who have retirement savings and investments in large amounts. What we learn here is that scammers are constantly evolving their techniques to increase their fraudulent returns. 

Because of the increasing sophistication by frauds, it’s important that your organization’s fraud detection approaches are always a step ahead. As we have seen, graph analytics will get the job done. 

Remember that traditional fraud detection methods rely on stored data, which requires intensive human resources to categorize and analyze it for patterns. Graph analysis, however, automates databases through machine learning. It reveals patterns from the data that traditional analysis methods may fail to detect. This is why the switch to graph analytics is essential for a robust organizational fraud detection strategy. 

[Get Started](https://docs.nebula-graph.io/3.6.0/#getting_started).


Explore graph analytics: a key tool in detecting fraud by analyzing connections and patterns in data. Understand how it differs from traditional analytics and its role in identifying complex fraud schemes in today's interconnected digital world.

Fraud Detection with Graph Analytics

This article attempts to explore the traces of these contributors using graph visualization, "peeping" into the relationship between contributors and projects in the Open Source NebulaGraph Community and the PR traces they leave behind.

Exploring Contributor Relationships in the Open Source NebulaGraph Community: A Graph Visualization Approach

We will now explore new techniques for pinpointing important content using common neighbors, pushing information flow based on friend relationships and geographic location, and using spatio-temporal relationship mapping to query the relationship between people. 

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 3

We have had a thorough introduction to graph data modeling and data import in the application of social network with graph technology. 

[**7 Fundamental Use Cases of Social Networks with NebulaGraph Database | Episode 1**](https://www.nebula-graph.io/posts/social-networks-with-graph-database-1)

Now, let me introduce more use cases in this episode.

# Identifying key people

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

There are many ways to identify them, and the specific methods and information, relationships, and perspectives considered also depend on the type of these key individuals, and the purpose of acquiring them.

Some common methods include looking at the number of followers a person has or the amount of content consumed, their reader engagement on their posts, videos, and the reach of their content (retweets, citations). These methods are also doable on the graph but are rather mundane, so I won't give examples. Here, we can try to derive these key people on the graph using graph algorithms that evaluate and calculate the importance of nodes.

## PageRank

PageRank is a very classic graph algorithm that iterates through the number of relationships between points on a graph to get a score (Rank) for each point. It was originally proposed by Google founders Larry Page and Sergey Brin and used in the early Google search engine to sort search results, where Page can be a pun on Larry Page's last name and Web Page.

PageRank has long been abandoned as too simple in modern, complex search engines, but it still shines in other graph-structured web scenarios, where we can roughly assume that all links are of similar importance and run the algorithm to find those key users in social networks.

In NebulaGraph, we can use NebulaGraph Algorithm, NebulaGraph Analytics to run PageRank on large full graphs, while in the daily analysis, validation, and design phases, we don't need to run results on full data, but on very small subgraphs (up to tens of thousands), we can easily run various graph algorithms in the browser to derive methods that can be used for production.

Today, we will use the built-in in-browser graph algorithm function of NebulaGraph Explorer to execute PageRank (the specific method is omitted here, you can refer to the [documentation](https://docs.nebula-graph.io/3.3.0/nebula-explorer/graph-explorer/graph-algorithm/ ), but it's really just a matter of mouse clicks).

![PageRank](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/PageRank.png)

We can see from the above that among all the green players (people) after PageRank calculation, "player.name: Tim Duncan" is the largest one, and the relationship associated with it does seem to be quite a lot, so we select him on the graph, then right-click to invert, select all the points except Tim Duncan, use the backspace key to delete all the other points, and then explore 1 to 5 steps in both directions with him as the starting point. In one step, we get Tim Duncan's subgraph.

![TimDuncan](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/TimDuncan.png)

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

## Betweenness Centrality

As you can see from the subgraphs, Tim Duncan is associated with a very large number of other players, while some other very popular players have served with him in the very popular Spurs team, which confirms the way PageRank is evaluated.

Now let's see if the algorithm will come to the same conclusion for the other dimensions of determination.

![Betweeness_centrality](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Betweeness_centrality.png)

From its five-hop subgraph, it can be seen that unlike the star-shape of the key figure Tim Duncan obtained from PageRank before, Dejounte Murray's subgraph shows clusters, where it is sensory, intuitive to imagine that Dejounte Murray is really on the necessary path of the minimal path between many nodes, while Tim Duncan seems to be associated with more important connecters.

![DejounteMurray](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/DejounteMurray.png)

In practical application scenarios, we usually have to understand the definitions in different ways, experiment with different execution results, and analyze to find the structural features that affect the key people we care about, and use them to choose different algorithms for different needs.

# Detect communities and clusters

Community detection in social networks is a technique to discover community structure by analyzing social relationships. A community structure is a set of nodes that are closely connected to each other in a social network, graph, and these nodes usually have similar characteristics or interests. For example, a community structure may manifest itself as a group of users who are clustered together based on common topics or interests.

The purpose of community detection is to identify the boundaries of different communities and determine the nodes in each community by analyzing the social network. This process can be done by using various algorithms such as label propagation algorithm, weakly connected component algorithm and Louvain algorithm. By discovering the community structure, we can better understand the structure and characteristics of social networks, and help social network service providers to better infer and predict behaviors in social networks, and help in good social network governance, advertisement placement, marketing, etc.

Since our dataset is fake-generated, the results I get under different algorithms do not show the real meaning, so this chapter just shows the results after community identification using several graph algorithms, in real-world cases, we should also use domain knowledge or other technical means on top of that to collaboratively give the portraits and labels of different groups and communities.

Effect of label propagation algorithm.

![LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/LPA.png)



Louvain algorithm:

![Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

WCC algorithm:

![WCC](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Louvain.png)

In later sections, we could in better chance verify these algorithms again on smaller and simpler subgraphs, with somewhat more interpretable results.

# Friend Closeness

With the community detection algorithm, it is actually possible to obtain friends with similar interests and close associations to some extent, in a global calculation. So how do we get the other close friends of a given user? We can get this information by counting the number of friends this user has in common with him in order to get this information!

Let's take "Tim Duncan" for example, we know that his two-degree friends (friends of friends: `(:player{name: "Tim Duncan"})-[:follow]-(f:player)-[:follow]-(fof:player)`) are also his friends: Mutual Friend, then it is reasonable to believe that those who have more friends in common with Tim Duncan may have a higher closeness to him.

```cypher
MATCH (start:`player`{name: "Tim Duncan"})-[:`follow`]-(f:`player`)-[:`follow`]-(fof:`player`),
(start:`player`)-[:`follow`]-(fof:`player`)
RETURN fof.`player`.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

This query shows that "Tony Parker" and Tim have 5 friends in common and are the closest.

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| Tony Parker       | 5           |
| Dejounte Murray   | 4           |
| Manu Ginobili     | 3           |
| Marco Belinelli   | 3           |
| Danny Green       | 2           |
| Boris Diaw        | 1           |
| LaMarcus Aldridge | 1           |
| Tiago Splitter    | 1           |

Here, let's verify this result through visualization!

First, let's see who the common friends (f:) are for each of the friends.

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player),
(start:player)-[:`follow`]-(fof:player)
RETURN fof.player.name, collect(DISTINCT f.player.name);
```

The result:

| fof.player.name   | collect(distinct f.player.name)                              |
| :---------------- | :----------------------------------------------------------- |
| Boris Diaw        | ["Tony Parker"]                                              |
| Manu Ginobili     | ["Dejounte Murray", "Tiago Splitter", "Tony Parker"]         |
| LaMarcus Aldridge | ["Tony Parker"]                                              |
| Tiago Splitter    | ["Manu Ginobili"]                                            |
| Tony Parker       | ["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"] |
| Dejounte Murray   | ["Danny Green", "Tony Parker", "Manu Ginobili", "Marco Belinelli"] |
| Danny Green       | ["Dejounte Murray", "Marco Belinelli"]                       |
| Marco Belinelli   | ["Dejounte Murray", "Danny Green", "Tony Parker"]            |

 Then we visualize the result on Explorer.

- First, let's find out all of Tim's 2-degree friend paths

```cypher
MATCH p=(start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:follow]-(fof:player)
RETURN p
```

- Then we render the node size by degree in which we select Tim and Tony and find all paths between them for follow type edge, bidirectional, up to 2 hops:

We can see that they are the closest of friends to each other and that their mutual friends are also in the paths.

`["Dejounte Murray", "Boris Diaw", "Manu Ginobili", "Marco Belinelli", "LaMarcus Aldridge"]`

![closest_friend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/closest_friend.png)

## Small groups in your friends

At this point, as mentioned earlier, the non-authenticity of this dataset itself makes the results of the community discovery algorithm unable to get the insightful connotation of it. Now we can follow this small subgraph to see how groups and communities can be distinguished among Tim's friends.

- Weakly connected components can split Tim's friends into two or three parts that are not connected to each other, which is very much in line with the intuitive understanding and definition of connected components.

![Tim_wcc](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_wcc.png)

- Label propagation, we can control the number of iterations on-demand to delineate different degrees of division by random propagation, which results in a certain degree of differentiation.

  20 iterations

  ![Tim_LPA](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

  1000 iterations

  ![Tim_LPA_1000](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_LPA_1000.png)

- Louvain, a more efficient and stable algorithm, basically under this subgraph we can get a very intuitive division with a very small number of iterations.

![Tim_Louvain](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_Louvain.png)

# New friend recommendation

Following the previous idea of 2-degree friends (friends of friends), we can easily add those 2-degree friends who are not yet friends as recommended friends, and the sorting rule is the number of friends they have in common with each other:

```cypher
MATCH (start:player{name: "Tim Duncan"})-[:`follow`]-(f:player)-[:`follow`]-(fof:player)
WHERE NOT (start:player)-[:`follow`]-(fof:player) AND fof != start
RETURN fof.player.name, count(DISTINCT f) AS NrOfMutualF ORDER BY NrOfMutualF DESC;
```

| fof.player.name   | NrOfMutualF |
| :---------------- | :---------- |
| LeBron James      | 2           |
| James Harden      | 1           |
| Chris Paul        | 1           |
| Yao Ming          | 1           |
| Damian Lillard    | 1           |
| JaVale McGee      | 1           |
| Kevin Durant      | 1           |
| Kyle Anderson     | 1           |
| Rudy Gay          | 1           |
| Russell Westbrook | 1           |

Obviously, LeBron is the most recommended! And look at who these mutual friends are.

| fof.player.name   | collect(distinct f.player.name)    |
| :---------------- | :--------------------------------- |
| James Harden      | ["Dejounte Murray"]                |
| LeBron James      | ["Danny Green", "Dejounte Murray"] |
| Chris Paul        | ["Dejounte Murray"]                |
| Yao Ming          | ["Shaquille O'Neal"]               |
| Damian Lillard    | ["LaMarcus Aldridge"]              |
| JaVale McGee      | ["Shaquille O'Neal"]               |
| Kevin Durant      | ["Dejounte Murray"]                |
| Kyle Anderson     | ["Dejounte Murray"]                |
| Rudy Gay          | ["LaMarcus Aldridge"]              |
| Russell Westbrook | ["Dejounte Murray"]                |


Again, let's look for LeBron James in the subgraph we just created! And find the two-step, two-way path between them, and sure enough, it only goes through `["Danny Green", "Dejounte Murray"]` and, without a direct connection.

![Tim_newFriend](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/Tim_newFriend.png)

Now, the system could send reminders to both sides: "HEY, maybe you two should make new friends!"

# Continue Exploration:

- [Pinpointing important content using common neighbors](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-common-neighbor)
- [Push information flow based on friend relationship and geographic location](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-feed-generation)
- [Spatio-temporal relationship mapping](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-spatio-temporal-relationship-tracking)

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))

Identifying influencers in social networks involves using a variety of metrics and methods to identify individuals who have a lot of influence in a given network. This is useful for many business scenarios, such as for marketing or researching the spread of information in a network.

7 Fundamental Use Cases of Social Networks with NebulaGraph Database | EP 2

Two months ago, my colleague brought in a graph of Twitter interactions  and asked me if I could write a post on graph technology to explore social network analysis and see how those connections are recognized through graph technology.

![graph-database-on-social-network](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-02.jpeg)

![graph-database-on-social-network-3](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/graph-database-on-social-network-03.jpeg)

So, how are typical social relationship graphs like the one above, or even more complex social network-based graphs, generated? In the following episodes, I will use the NebulaGraph database to conduct the social network analysis, and you can find step-by-step guides to try all the methods in Playground.

**Also read:** [How to choose a database for a social media app](https://www.nebula-graph.io/posts/how-to-choose-database-for-social-media-app)

# Social Network Analysis on Graph Database

We can use a [graph database](https://www.nebula-graph.io/) in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

For example, graph databases can identify "influential users" in a network, recommend new connections (friendships, favorite content ) based on commonalities between users, or find different groups of people and communities to profile users. Graph databases are ideal for social networking systems where user relationships constantly change because they can support complex multi-hop queries and real-time writes and updates.

There have been quite a few examples of social network graphs practical applications, including:

- Finding key people
- Identifying clusters of people and communities
- Determining the closeness between two users
- Recommending new friends
- Pinpointing important content using common neighbors
- Push information flow based on friend relationships and geographic location
- Use spatio-temporal relationship mapping to query the relationship between people, get the people who intersected in time and space, and the provinces visited

I will dive into each application with hands-on demonstrations to present graph databases' competence in a couple of episodes. Stay tuned!

# Social Network Graph Modeling

To showcase the SNS graph use cases, I'll build most of the examples on a typically small social network. I started by adding extra data on top of the NebulaGraph default dataset, basketballplayer:

Three new tags of vertices:

- address
- place
- post

Five new types of edges:

- created_post
- commented_at
- lived_in
- belong_to

It looks like this:

![schema_sketch](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/schema_sketch.png)

# Importing the data

## Load the default dataset

- In the Command Line Console, we could execute:play basketballplayer` to load the default dataset.

- Or, if we do so from NebulaGraph Studio/Explorer, click the Download from the Demos in the welcome page:

  ![baskertballplayer_studio_starter](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/baskertballplayer_studio_starter.png)

## Add the SNS Graph schema

First, the DDL for those new tags and edges:

```sql
CREATE TAG IF NOT EXISTS post(title string NOT NULL);
CREATE EDGE created_post(post_time timestamp);
CREATE EDGE commented_at(post_time timestamp);
CREATE TAG address(address string NOT NULL, `geo_point` geography(point));
CREATE TAG place(name string NOT NULL, `geo_point` geography(point));
CREATE EDGE belong_to();
CREATE EDGE lived_in();
```

## Load the data

Then we load the DML part to insert vertices and edges:

```sql
INSERT VERTEX post(title) values \
    "post1":("a beautify flower"), "post2":("my first bike"), "post3":("I can swim"), \
    "post4":("I love you, Dad"), "post5":("I hate coriander"), "post6":("my best friend, tom"), \
    "post7":("my best friend, jerry"), "post8":("Frank, the cat"), "post9":("sushi rocks"), \
    "post10":("I love you, Mom"), "post11":("Let's have a party!");

INSERT EDGE created_post(post_time) values \
    "player100"->"post1":(timestamp("2019-01-01 00:30:06")), \
    "player111"->"post2":(timestamp("2016-11-23 10:04:50")), \
    "player101"->"post3":(timestamp("2019-11-11 10:44:06")), \
    "player103"->"post4":(timestamp("2014-12-01 20:45:11")), \
    "player102"->"post5":(timestamp("2015-03-01 00:30:06")), \
    "player104"->"post6":(timestamp("2017-09-21 23:30:06")), \
    "player125"->"post7":(timestamp("2018-01-01 00:44:23")), \
    "player106"->"post8":(timestamp("2019-01-01 00:30:06")), \
    "player117"->"post9":(timestamp("2022-01-01 22:23:30")), \
    "player108"->"post10":(timestamp("2011-01-01 10:00:30")), \
    "player100"->"post11":(timestamp("2021-11-01 11:10:30"));

INSERT EDGE commented_at(post_time) values \
    "player105"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player109"->"post1":(timestamp("2016-11-24 10:04:50")), \
    "player113"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player101"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player102"->"post1":(timestamp("2015-03-03 00:30:06")), \
    "player103"->"post1":(timestamp("2017-09-23 23:30:06")), \
    "player102"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player101"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player106"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player105"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player130"->"post1":(timestamp("2019-01-02 00:30:06")), \
    "player131"->"post2":(timestamp("2016-11-24 10:04:50")), \
    "player131"->"post3":(timestamp("2019-11-13 10:44:06")), \
    "player133"->"post4":(timestamp("2014-12-04 20:45:11")), \
    "player132"->"post5":(timestamp("2015-03-03 00:30:06")), \
    "player134"->"post6":(timestamp("2017-09-23 23:30:06")), \
    "player135"->"post7":(timestamp("2018-01-04 00:44:23")), \
    "player136"->"post8":(timestamp("2019-01-04 00:30:06")), \
    "player137"->"post9":(timestamp("2022-01-02 22:23:30")), \
    "player138"->"post10":(timestamp("2011-01-11 10:00:30")), \
    "player141"->"post1":(timestamp("2019-01-03 00:30:06")), \
    "player142"->"post2":(timestamp("2016-11-25 10:04:50")), \
    "player143"->"post3":(timestamp("2019-11-14 10:44:06")), \
    "player144"->"post4":(timestamp("2014-12-05 20:45:11")), \
    "player145"->"post5":(timestamp("2015-03-04 00:30:06")), \
    "player146"->"post6":(timestamp("2017-09-24 23:30:06")), \
    "player147"->"post7":(timestamp("2018-01-05 00:44:23")), \
    "player148"->"post8":(timestamp("2019-01-05 00:30:06")), \
    "player139"->"post9":(timestamp("2022-01-03 22:23:30")), \
    "player140"->"post10":(timestamp("2011-01-12 10:01:30")), \
    "player141"->"post1":(timestamp("2019-01-04 00:34:06")), \
    "player102"->"post2":(timestamp("2016-11-26 10:06:50")), \
    "player103"->"post3":(timestamp("2019-11-15 10:45:06")), \
    "player104"->"post4":(timestamp("2014-12-06 20:47:11")), \
    "player105"->"post5":(timestamp("2015-03-05 00:32:06")), \
    "player106"->"post6":(timestamp("2017-09-25 23:31:06")), \
    "player107"->"post7":(timestamp("2018-01-06 00:46:23")), \
    "player118"->"post8":(timestamp("2019-01-06 00:35:06")), \
    "player119"->"post9":(timestamp("2022-01-04 22:26:30")), \
    "player110"->"post10":(timestamp("2011-01-15 10:00:30")), \
    "player111"->"post1":(timestamp("2019-01-06 00:30:06")), \
    "player104"->"post11":(timestamp("2022-01-15 10:00:30")), \
    "player125"->"post11":(timestamp("2022-02-15 10:00:30")), \
    "player113"->"post11":(timestamp("2022-03-15 10:00:30")), \
    "player102"->"post11":(timestamp("2022-04-15 10:00:30")), \
    "player108"->"post11":(timestamp("2022-05-15 10:00:30"));

INSERT VERTEX `address` (`address`, `geo_point`) VALUES \
    "addr_0":("Brittany Forge Apt. 718 East Eric  WV 97881", ST_Point(1,2)),\
    "addr_1":("Richard Curve Kingstad  AZ 05660", ST_Point(3,4)),\
    "addr_2":("Schmidt Key Lake Charles  AL 36174", ST_Point(13.13,-87.65)),\
    "addr_3":("5 Joanna Key Suite 704 Frankshire  OK 03035", ST_Point(5,6)),\
    "addr_4":("1 Payne Circle Mitchellfort  LA 73053", ST_Point(7,8)),\
    "addr_5":("2 Klein Mission New Annetteton  HI 05775", ST_Point(9,10)),\
    "addr_6":("1 Vanessa Stravenue Suite 184 Baileyville  NY 46381", ST_Point(11,12)),\
    "addr_7":("John Garden Port John  LA 54602", ST_Point(13,14)),\
    "addr_8":("11 Webb Groves Tiffanyside  MN 14566", ST_Point(15,16)),\
    "addr_9":("70 Robinson Locks Suite 113 East Veronica  ND 87845", ST_Point(17,18)),\
    "addr_10":("24 Mcknight Port Apt. 028 Sarahborough  MD 38195", ST_Point(19,20)),\
    "addr_11":("0337 Mason Corner Apt. 900 Toddmouth  FL 61464", ST_Point(21,22)),\
    "addr_12":("7 Davis Station Apt. 691 Pittmanfort  HI 29746", ST_Point(23,24)),\
    "addr_13":("1 Southport Street Apt. 098 Westport  KY 85907", ST_Point(120.12,30.16)),\
    "addr_14":("Weber Unions Eddieland  MT 64619", ST_Point(25,26)),\
    "addr_15":("1 Amanda Freeway Lisaland  NJ 94933", ST_Point(27,28)),\
    "addr_16":("2 Klein HI 05775", ST_Point(9,10)),\
    "addr_17":("Schmidt Key Lake Charles AL 13617", ST_Point(13.12, -87.60)),\
    "addr_18":("Rodriguez Track East Connorfort  NC 63144", ST_Point(29,30));

INSERT VERTEX `place` (`name`, `geo_point`) VALUES \
    "WV":("West Virginia", ST_Point(1,2.5)),\
    "AZ":("Arizona", ST_Point(3,4.5)),\
    "AL":("Alabama", ST_Point(13.13,-87)),\
    "OK":("Oklahoma", ST_Point(5,6.1)),\
    "LA":("Louisiana", ST_Point(7,8.1)),\
    "HI":("Hawaii", ST_Point(9,10.1)),\
    "NY":("New York", ST_Point(11,12.1)),\
    "MN":("Minnesota", ST_Point(15,16.1)),\
    "ND":("North Dakota", ST_Point(17,18.1)),\
    "FL":("Florida", ST_Point(21,22.1)),\
    "KY":("Kentucky", ST_Point(120.12,30)),\
    "MT":("Montana", ST_Point(25,26.1)),\
    "NJ":("New Jersey", ST_Point(27,28.1)),\
    "NC":("North Carolina", ST_Point(29,30.1));

INSERT EDGE `belong_to`() VALUES \
    "addr_0"->"WV":(),\
    "addr_1"->"AZ":(),\
    "addr_2"->"AL":(),\
    "addr_3"->"OK":(),\
    "addr_4"->"LA":(),\
    "addr_5"->"HI":(),\
    "addr_6"->"NY":(),\
    "addr_7"->"LA":(),\
    "addr_8"->"MN":(),\
    "addr_9"->"ND":(),\
    "addr_10"->"MD":(),\
    "addr_11"->"FL":(),\
    "addr_12"->"HI":(),\
    "addr_13"->"KY":(),\
    "addr_14"->"MT":(),\
    "addr_15"->"NJ":(),\
    "addr_16"->"HI":(),\
    "addr_17"->"AL":(),\
    "addr_18"->"NC":();

INSERT EDGE `lived_in`() VALUES \
    "player100"->"addr_4":(),\
    "player101"->"addr_7":(),\
    "player102"->"addr_2":(),\
    "player103"->"addr_3":(),\
    "player104"->"addr_0":(),\
    "player105"->"addr_5":(),\
    "player106"->"addr_6":(),\
    "player107"->"addr_1":(),\
    "player108"->"addr_8":(),\
    "player109"->"addr_9":(),\
    "player110"->"addr_10":(),\
    "player111"->"addr_11":(),\
    "player112"->"addr_12":(),\
    "player113"->"addr_13":(),\
    "player114"->"addr_14":(),\
    "player115"->"addr_15":(),\
    "player116"->"addr_16":(),\
    "player117"->"addr_17":(),\
    "player118"->"addr_18":();
```

## First glance at the data

Let's start with the stats of the data.

```sql
[basketballplayer]> SUBMIT JOB STATS;

+------------+
| New Job Id |
+------------+
| 10         |
+------------+
[basketballplayer]> SHOW STATS;
+---------+----------------+-------+
| Type    | Name           | Count |
+---------+----------------+-------+
| "Tag"   | "address"      | 19    |
| "Tag"   | "place"        | 14    |
| "Tag"   | "player"       | 51    |
| "Tag"   | "post"         | 10    |
| "Tag"   | "team"         | 30    |
| "Edge"  | "belong_to"    | 19    |
| "Edge"  | "commented_at" | 40    |
| "Edge"  | "created_post" | 10    |
| "Edge"  | "follow"       | 81    |
| "Edge"  | "lived_in"     | 19    |
| "Edge"  | "serve"        | 152   |
| "Space" | "vertices"     | 124   |
| "Space" | "edges"        | 321   |
+---------+----------------+-------+
Got 13 rows (time spent 1038/51372 us)
```

We could get all of the data:

```sql
MATCH ()-[e]->() RETURN e LIMIT 10000
```

As the data volume is quite small, we could render them all in the canvas of NebulaGraph Explorer:

![match_all](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-sns/match_all.png)

Above are the two fundamental steps to embark on the journey of social network exploration. I'll introduce other applications in the next episode:

- [Finding key people](https://www.nebula-graph.io/posts/social-networks-with-graph-database-2#post-identifying-key-people)
- [Identifying clusters of people, communities](https://www.nebula-graph.io/posts/social-networks-with-graph-database-2#post-detect-communities-and-clusters)
- [Determining the closeness between two users](https://www.nebula-graph.io/posts/social-networks-with-graph-database-2#post-friend-closeness)
- [Recommending new friends](https://www.nebula-graph.io/posts/social-networks-with-graph-database-2#post-new-friend-recommendation)
- [Pinpointing important content using common neighbors](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-common-neighbor)
- [Push information flow based on friend relationship and geographic location](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-feed-generation)
- [Spatio-temporal relationship mapping](https://www.nebula-graph.io/posts/social-networks-with-graph-database-3#post-spatio-temporal-relationship-tracking)

[![docker](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/docker-banner.png)](javascript:window.open('https://bit.ly/3XHS873'))


We can use a graph database in a social network system to represent users and their connections or relationships. Graph databases allow efficient querying of relationships between users, making various business implementations on social networks based on connection findings, statistics, and analysis feasible and efficient.

Graph Database for Social Networks: 7 Fundamental Use Cases of  NebulaGraph| EP 1

Qingteng Cloud Security is a host security unicorn company, which focuses on the security of cloud native applications. At present, its main business is the host product named Wanxiang and the container product named Honeycomb. It leads the industry and supports 8 million agents in total. The company's next-generation real-time intrusion detection system based on NebulaGraph and combined with graph technology has been initially put into the market and participated in the 2022 Network Protection, which has achieved good response.

In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

# The status and challenge of intrusion detection

## Mainstream intrusion detection system

Intrusion detection has always been a major direction of security research. The two sets of products of Qingteng, Wanxiang and Honeycomb, are host-based and container-based intrusion detection products, both of which have similar principles, as shown below.

![Intrusion detection system architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-1.png)

The agent is placed on the host and container side, receiving rules from the server side, combining the collected raw events (process/network connection/file read and write, etc.), and reporting them to the server side as alarms after hitting the rules written by security experts, such as the MD5 of the process file, execution command line, machine learning, and other features. The full amount of raw events are also reported to the server side log collection system and stored in the SIEM system.

Once the alarm is reported to the server, the security staff will take the information related to the alarm to SIEM system to check the events before and after the time of occurrence, log in to the relevant machine to check the relevant information if necessary, and make a comprehensive judgment on whether the current attack alarm is valid and do the relevant disposal.

This system is the current mainstream architecture of intrusion detection, but there are many problems:

1. For single-point detection based on single events and rules, it may result in false positives due to too loose rules or missed positives due to too strict rules.
2. Too many alarms may be triggered by the same attack, and security staff have a heavy workload in tracing the source.
3. The original events related to the same alarm need to be manually analyzed by the SIEM system, which lacks visual means.

## The research direction of graph and security

These questions essentially look at each alarm and event independently, when in fact, the alarms and events associated with an attack should be related to each other. We can use graphs to correlate these original events and alarms for overall analysis. In fact, this is also a popular direction of current security research -- traceability graph, with the help of traceability graph we do the following security analysis and detection.

### 1.Graph detection

Traditional security solutions solve similar IOC detection problems, that is, the judgment of single-point, for the process, network connections, files and other entities is not safe. The actual problem faced by the attack, may not be detected at all points, but the actual behavior is dangerous. A single rule is not fully written, even if the rule can be written, such as hitting a command line, but it will trigger many alarms, and eventually the rule cannot be used. We need to comprehensively judge various combination relationships, which is what graph is good at, and also the problem of graph detection in security. Put all related events into one graph to comprehensively judge whether they are valid attacks.

**The research of graph detection started early, but faced with the dual challenges of computational volume and algorithm, the current industry applications are few, and the combined judgments are all in the way of sequence detection**. But the sequence rules can not write too many rules in the rule list, otherwise it will lead to the inability to apply the rules. In addition, if the rules are too detailed, and can not be well fuzzy matching rules, it is easy to be bypassed. Hackers can easily bypass these rules with the help of automation tools.

### 2.Graph association and tracing

Just mentioned alarms, which is another problem all the security products facing today, either the rules do not hit, or too many rule hits, we call this situation **alarm flooding**. Security staff processing capacity is limited, maybe 100 alarms a day is fine, if a day or even an hour 10000 alarms can not be handled, it is no different from no alarms.

This is really a problem of correlation classification and traceability in security:

1. An attack will generate a lot of alarms, such as brute force login, used malicious files, executed malicious commands, etc.. The correlation classification to correlate the related alarms in an attack together, which is something the graph is good at. These alarms also allow a comprehensive assessment of whether the current attack is valid or not.
2. Each alarm will only tell you is a process, file, or network problems, but how this problem occurred, how the hacker came in, where the file was downloaded from, what was done first and then what was done. Security products need to help customers complete this analysis process, the industry is currently with the help of SIEM, THP, SOC and other security products, all the original events are uploaded and saved, and then find security experts, starting from the alarm to check the original event logs, to see what happened before the alarm, which has a relationship may lead to an attack, the process is as short as a few minutes, long as several hours. **Security is a confrontation process, early is better**, the sooner it is found the sooner it is confronted, blocked or isolated, otherwise it is too late even if it is found. The relevant original event entities (process/network, etc.) can be imported into the graph, with the help of the graph to visually explore and trace the entire attack process, which is the process of graph traceability, academically called causal graph, traceability graph.

### 3.Graph knowledge mapping and prediction

We know that **current security is fundamentally based on rules or a priori knowledge**, each vulnerability, Trojan horse, attack tools, attack process, attack organizations have its characteristics, the first few rules are still relatively good description, the attack process and attack organizations are difficult to complete the description.

The current mainstream approach is to build a knowledge base based on security frameworks, the current mainstream has Kill Chain, ATTCK and other frameworks, which is the U.S. Department of Defense-led cyber attack war related to the two companies proposed a security analysis framework, equivalent to the mapping of the tactics and specific attack techniques that delineate the attack. Security academia, such as the University of Illinois and Purdue University have been working on similar issues in the last two years, which is the construction of a security knowledge base (security knowledge mapping). With a complete knowledge base, the ultimate vision of security can be accomplished, for example, if I know your attack process and attack organization, is it possible to stop an attack before it actually starts.

## Domestic and international status

At present, the intrusion detection system based on graph that is really taking the lead in the real world is Crowd Strike, a star security company in the U.S. It builds its security system entirely on graphs, which now amounts to doing both graph detection and graph correlation traceability, and is currently valued at 67 billion. Cloud computing giants AWS and Azure are following its approach. Domestic intrusion detection system based on graph, currently there are public information is ThreatBook and Sangfor, they are equivalent to do part of the work of graph correlation and traceability.

Qingteng Cloud Security's Wanxiang and Honeycomb have been working on intrusion detection for many years and have achieved industry-recognized security detection capabilities, so we chose to start with graph association and traceability, and develop a next-generation real-time intrusion detection system based on NebulaGraph combined with graph technology, focusing first on solving the pain point problems of alarm flooding and association traceability.

# Qingteng cloud security next-generation intrusion detection system

## Detection principle architecture diagram

As shown below, the focus is the reported attack alerts and some of the original events are unified and correlated in real time in the graph.

![Detection principle architecture diagram](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-2.png)

## Product effect

The output after correlation processing by the correlation engine is the attack event, an attack event may be associated with multiple attack alerts and visualized, the current product effect is as follows.

- The same attack using malicious files and trojans

  ![The same attack using malicious files and trojans](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-03.png)

- Mounting of sensitive containers

  ![Mounting of sensitive containers](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-architecture-04.png)

- Custom script detection

  ![Custom script detection](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/intrusion-detection/intrusion-detection-5.png)

## Advantages of NebulaGraph

NebulaGraph was chosen based on the following considerations.

1. The immediate need for graph queries, especially for multi-level process relationships during association and traceability. It is much easier to query a [graph database](https://www.nebula-graph.io) with cypher than to query multi-level relationships in a relational database with SQL.
2. Large-scale storage, involving the storage of a large number of events and alarms.
3. High-performance query scenarios, associations need to ensure near real-time, the current relevant queries are in ms level.

Thanks to the good performance of NebulaGraph, the association engine enters the graph and calculates the association in a near real-time manner. The tool for importing graph data is the real-time graph entry component created by Qingteng based on Flink, which only needs to change the configuration file to complete the map of points and edges.

# Next research direction and plan

At present, the intrusion detection system of Qingteng cloud security mainly supports the association and traceability between single machine and some multi-machine scenarios. The next step is to support more scenarios of multi-machine association with the help of the graph, especially some typical security attack scenarios (rebound/horizontal movement), so as to further enable security with the graph and provide better service for customers.

There are many problems in applying NebulaGraph to graph security applications at present. Some suggestions are given for reference.

## 1.Push down optimization

In our process of use, the most important problem is the optimization of push down. Such as a diffuse process tree, while specifying the root node and leaf nodes, the order of statement writing may lead to query from the leaf node or the root node, query efficiency varies widely, and finally can only specify the leaf node conditions to force the query from the leaf node.

The performance of MATCH is also essentially a push-down optimization problem, this problem is also encountered a lot, our current approach is to maintain a cache pool for some of the statements have been queried, also note that the official enterprise version actually provides a partial cache function, this point is good if you can consider adding to the community version, is a practical application just need.

## 2.Some technical requirements for real-time scenarios

Since we are doing real-time association based on NebulaGraph, one of the major problems is how to achieve a balance between consistency and speed, because currently NebulaGraph cannot actually determine whether the graph entry is complete after writing to it. In practice, the association engine needs to poll downstream to determine whether the current vertices and edges is actually in the graph. This problem has some impact on the actual use and performance, and it is worth considering whether there is a better way.

We can see that some competing products, such as TigerGraph and TuGraph, provide the ability to customize the matching and traversal algorithms. In our multi-host association scenario, we have encountered such a need. At present, we can only achieve it by splitting multiple paths and splicing them together. Efficiency and speed are definitely compromised.

## 3.ToB deployment

Another pain point is the ToB deployment of NebulaGraph. In the security industry, there are still more on-site deployments at home. Especially, state-owned enterprises and government affairs are relatively sensitive to SaaS, so they will face the problem of efficiency and cost.

**The first is single-machine deployment**. The default production version is a distributed deployment with separate storage and computing designed for the cloud, which is still too cumbersome for ToB deployments and consumes relatively more resources than other databases.

**The second is HDD deployment**. Considering the cost, many companies, especially small and medium-sized companies, have limited security budgets, which makes it difficult to provide SSD deployment, not to mention the large capacity and high configuration of SSD machines required by the official website. Therefore, more optimization considerations should be made for HDD and low configuration machines.

Above, for Qingteng cloud security team engineer Zhou Wen to share.

**Thank you for reading this article.**

Want to experience the graph database up close? Now you can use NebulaGraph Cloud on AWS to build your own graph data system, come save a lot of deployment and installation time to get your business done [click the link](https://go.aws/3VZay2I) to use the graph database!

If you want to see the source code, you can go to GitHub to read, use, star it -> [GitHub](https://bit.ly/3f7XHex). Exchange graph database technology and application skills with other NebulaGraph users, and join ["Slack"](https://bit.ly/3SmvFu2) to play together!

Intrusion detection has always been a major direction of security research. In this topic, we will briefly introduce the intrusion detection system based on graph, hoping to have more people to join in the mining of graph and security application combination.

Next-generation Intrusion Detection System Based on Graph

# Background

Traditional risk control mainly relies on manual review and empirical judgment. With the rise of digitalization in the past decade, leading companies began to introduce blacklisting systems to locate the risk points missed in the review stage through simple rules and post-facto audits, but this still cannot detect problems in advance. Since 2018, the anti-fraud system has gradually evolved to the stage of "intelligent decision engine" consisting of rule engines, real-time metric calculation, and machine learning technologies.

## Insurance Fraud Control Pain Points

The pain points of insurance fraud control are mainly reflected in the following aspects:

**Non-exchange information leads to poor risk control**: Data of insurance companies and the related industry is non-exchangeable. Even the data between various systems within the same insurance company is unshared. This can lead to asymmetry in business information and false claims, or the use of information asymmetry to increase claim amounts.

**Requirements of multi-dimensional computing**: Industry fraud diversity, specialization, and grouping will force the anti-fraud system to analyze data from multi-dimensional aspects, while the traditional analysis system based on relational models, dimensional models, or wide table models hardly analyzes data from dozens or even more dimensions.

**Accuracy requirements of algorithm recognition**: Business model tuning requires more feature data input during the training and serving stages. Generally, there are two methods to input data, one is writing more data, and the other is the supplementing of correlation data. However, if we go with the traditional method, it will be difficult to detect abnormal data with more than 4-hop queries, and the query latency is high.

## Underlying Storage Challenges


With the development of the anti-fraud system, the system faces challenges in the aspect of the underlying storage. 

The performance of traditional stand-alone databases has bottlenecks in capacity and query for linked data. In addition, business iterations and peaks and valleys have agility requirements for database scalability. Nowadays, most risk control systems use Hadoop ecosystem tools to co-accomplish system goals, and the corresponding development and operation and maintenance costs are high. In terms of real-time data analysis, it's difficult for traditional big data platforms to perform the analysis of multiple sources of data.


# Use Scenarios

NebulaGraph is used in many scenarios of insurance fraud detection. The following are 5 typical scenarios.


## Scenario 1: Relationship Detection


Customers may skip the family information in the application form. Such uncertain or missing family information are not well determined by the original method. While the graph data structure allows us to find relatives, which is not natively supported by other types of databases.

In addition, the regular form is filled with only immediate relatives, which does not reflect indirect relationships. In NebulaGraph's graph database, more hidden relationships can be discovered through the available data, including social and non-immediate family relationships. It allows for a comprehensive analysis of the family's insurance purchases.

## Scenario 2: Fraud Detection


Traditional fraud detection methods are mainly based on empirical rules and are unable to detect fraud in advance.

Taking car insurance as an example, there are relatively more entities involved in car insurance, including vehicles, owners, drivers, passengers, etc. It is very difficult to identify those entities among these data. However, in a graph database, it is possible to build a graph to see some outliers based on the number of claims or other business indicators, and it is also possible to classify groups with higher fraud rates through graph algorithms. In social relationship networks, the fraud possibility can be determined by the correlation with the blacklisted spots. In general, we can make full use of pattern matching and other graph database technologies to classify the major entities by using graph algorithms and then identify groups or individuals with high fraud rates based on business experience and existing fraud cases.


Fraud cases usually have certain characteristics. For example, a luxury used car with frequent accidents or multiple accident-related people driving a different vehicle frequently, or a high correlation between different vehicle groups and multiple accident-related owners. These are all suspicious characteristics that can be identified quickly and effectively by graph technology.

## Scenario 3: Precision Marketing 

In the scenarios of marketing, the data in a graph database can help identify target customers in depth. Based on the information left by customers, we can find social relationships, friends, and non-immediate family members other than immediate family members. Such relationship networks help us see a larger dimension of the customer's social circle. We can find more hidden relationships in the connections between customers and their friends. This can help us learn more about family insurance purchase behaviors.


In the private domain scenario, the marketing agents can recommend other insurance products based on the customer's purchase history. The groups with associated relationships may tend to prefer the same insurance products. From another perspective, such associated recommendations will be more convincing to customers. In the public domain scenario of getting target customers, graph databases can also help advertise to the target customers through the customer data system.

## Scenario 4: Insurance Underwriting and Claims Fraud Detection


At the underwriting and claim stages, we can find what insurance policies the insured has, the number of claims denied, and the association with known fraud cases by querying the in and out-degree of the insured. As for car insurance, we can detect fraud from "the case" to the "vehicle", to the "owner", to the "driver", and to the "vehicle", as shown in the above figure. In this way, we can find the fraud cases from the root cause. For example, in a DUI transfer case (the drunk driver A contacts his relative B to arrive at the scene and report to the insurance company in place of A to get insurance compensation after the DUI accident), we can analyze behaviors such as outgoing and incoming calls, and the location information of the involver. Through this information, multiple risk factors may be detected such as delayed reporting, non-frequently used cell phone numbers and distance from the scene.

## Scenario 5: Agents Management


A large number of domestic insurance agents and a high turnover rate may lead to difficulty in agent management. 

We can also build the agent's relationship network based on the graph model. The agent's professional habits and complaints received can be labeled in the graph to enhance the screening of agents from the source.

 For some underground insurance scenarios, such as forging the identity of a newcomer for commission. Fraud risks can be identified through cell phone numbers, electronic device numbers, IP associations, etc., all of which can effectively reduce customer complaints and insurance frauds caused by agents.



# Why NebulaGraph?

## Features of NebulaGraph

![NebulaGraph-Database](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/nebulagraph-ecosystem.png)


To address the above anti-fraud pain points and the need for the system-level construct, NebulaGraph has the following features that are suitable for the assurance risk control needs:

1. **Multi-source data fusion**: The sources of insurance data are diverse, including bank data, social network data, government data, medical data, marketing data, and so on. The data fusion of these data sources is a key to the success of the fraud detection system that can establish a comprehensive analysis of customers to better help insurance companies achieve automatic and accurate underwriting.

2. **Support of graph data model**: The construct of an underlying data model is the key to the multi-dimensional analysis. NebulaGraph data models fit the storage and query presentation requirements of linked data and provide quicker analysis compared to the relational model while its schema is flexible.

3. **Cloud-native architecture**: A good storage system should enable upper-tier applications to compute and scale horizontally based on business needs. NebulaGraph is a distributed cloud native graph database. It can be easily scaled out and scaled in to meet the needs of business growth and shrinkage.

4. **Enterprise-level feature support**: NebulaGraph, as a basic software, provides the support of peripheral tools for a better exploration and visual operation and maintenance.

5. **Risk factor detection**: NebulaGraph's association analysis and graph computing engines can help better identify the risk factors with algorithms.

6. **Accuracy of auxiliary algorithms**: Machine learning relies on input data constructed by tuples, which tend to ignore the relationships between data. So the input of graph data can be used to enrich the contextual information and thus improve the accuracy of relevant models.

In a word, NebulaGraph connects dimensional data. It can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. For details about how to deploy and use NebulaGraph, please refer to [live demos](https://www.nebula-graph.io/demo/).  

NebulaGraph can be used in different scenarios to help insurance companies effectively identify and avoid hidden risks at different stages. 

Insurance Fraud Detection with Graph Technology | A Practice of NebulaGraph

> An attempt to use ChatGPT to generate code for a data scraper to predict sports events with the help of the NebulaGraph graph database and graph algorithms.


#  The Hype of ChatGPT

In the hype for FIFA 2022 World Cup, I saw [a blog post from Cambridge Intelligence](https://cambridge-intelligence.com/fifa-world-cup-2022-prediction/), in which they leveraged limited information and correlations among players, teams, and clubs to predict the final winning team. As a graph technology enthusiast, I always would like to try similar things with NebulaGraph to share the ideas of graph algorithms to extract hidden information from the overall connections in a graph in the community.

The initial attempt was to complete it in about 2 hours, but I noticed the dataset needs to be parsed carefully from [Wikipedia](https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_squads) and I happened to be not good at doing this job, so I put the idea on hold for a couple of days.

In the meantime, another craze, the OpenAI ChatGPT, was announced. As I had been a user of DALL-E 2 already(to generate feature images of my blog posts), I also gave it a try very quickly. And I witnessed how other guys(via Twitter, blogs, HackerNews) tried to convince ChatGPT to do so many things that are hard to believe they could do:

- Help to implement a piece of code at any time
- Simulate any prompt interface: shell, python, virtual machine, or even a language you create
- Act out almost any given persona, and chat with you
- Write poetry, rap, prose
- Find a bug in a piece of code
- Explain the meaning of a complex regular expression/Open Cypher Query

ChatGPT's ability to contextualize and understand has never been greater before, so much so that everyone is talking about a new way of working: how to master asking/convincing/triggering machines to help us do our jobs better and faster.

So, after trying to get ChatGPT to help me write complex graph database query statements, explain the meaning of complex graph query statements, and a large chunk of Bison code, which he/she had done them WELL, I realized: why not let ChatGPT write the code that extracts the data for me?

# Grabbing data

I really tried it and the result is... good enough.

The whole process was basically like a coding interviewer, or a product manager, presenting my requirements, then ChatGPT giving me the code implementation. I then try to run the code, find the things that don't make sense in the code, point them out, and give suggestions, and ChatGPT quickly understands the points I point out and makes the appropriate corrections, like:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/chatGPT-correction-process.png)

I won't list this whole process, but I share the generated code and the whole discussion [here](https://gist.github.com/wey-gu/78cb28bee130966e7d6e9d573b51deff).

The final generated data is a CSV file.

- Raw version [world_cup_squads.csv](https://github.com/siwei-io/talks/files/10152775/world_cup_squads.csv)

- Manually modified, separated columns for birthday and age [world_cup_squads_v0.csv](https://github.com/siwei-io/talks/files/10152923/world_cup_squads.csv)

  It contains information/columns of a team, group, number, position, player name, birthday, age, number of international matches played, number of goals scored, and club served.

```CSV
Team,Group,No.,Pos.,Player,DOB,Age,Caps,Goals,Club
Ecuador,A,1,1GK,Hernán Galíndez,(1987-03-30)30 March 1987,35,12,0,Aucas
Ecuador,A,2,2DF,Félix Torres,(1997-01-11)11 January 1997,25,17,2,Santos Laguna
Ecuador,A,3,2DF,Piero Hincapié,(2002-01-09)9 January 2002,20,21,1,Bayer Leverkusen
Ecuador,A,4,2DF,Robert Arboleda,(1991-10-22)22 October 1991,31,33,2,São Paulo
Ecuador,A,5,3MF,José Cifuentes,(1999-03-12)12 March 1999,23,11,0,Los Angeles FC
```

- Final version with header removed [world_cup_squads_no_headers.csv](https://github.com/siwei-io/talks/files/10152974/world_cup_squads_no_headers.csv)



# Graph algorithm to predict the FIFA World Cup 2022

With the help of ChatGPT, I could finally try to predict the winner of the game with Graph Magic, before that, I need to map the data into the graph view.

If you don't care about the process, go to the [predicted result](https://www.nebula-graph.io/posts/predict-fifa-world-cup-with-chatgpt-and-nebulagraph#post-result) directly.



## Graph modeling

> Prerequisites: This article uses [NebulaGraph](https://github.com/vesoft-inc/nebula)(Open-Source) and [NebulaGraph Explorer](https://docs.nebula-graph.io/3.3.0/nebula-explorer/about-explorer/ex-ug-what-is-explorer/)(Proprietary), which you can request a free trial of on [AWS](https://go.aws/3VZay2I).

Graph Modeling is the abstraction and representation of real-world information in the form of a "vertex-> edge" graph. In our case, we will project the information parsed from Wikipedia as:

**Vertices**:

- player
- team
- group
- club

**Edges**:

- groupedin (the team belongs to which group)
- belongto (players belong to the national team)
- serve (players serve in the club)

The age of the players, the number of international caps, and the number of goals scored are naturally fit as properties for the player tag(type of vertex).

The following is a screenshot of this schema in **NebulaGraph Explorer** (I will just call it Explorer later).

![schema_fifa](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/schema_fifa.png)

Then, we can click the save icon in the upper right corner and the button: `Apply to Space`  to create a graph space with the [defined schema]( https://docs.nebula-graph.io/3.3.0/nebula-explorer/db-management/draft/):



## Ingesting into NebulaGraph

With the graph modeling, we can upload the [CSV file](https://github.com/siwei-io/talks/files/10152974/world_cup_squads_no_headers.csv) (the no-header version) into Explorer, by pointing and selecting the vid and properties that map the different columns to the vertices and edges.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/importer_config_mapping.png)

Click Import, we then import the whole graph to NebulaGraph, and after it succeeded, we could also get the entire CSV --> Nebula Importer configuration file: [nebula_importer_config_fifa.yml](https://github.com/siwei-io/talks/files/10164014/config_fifa.yml.txt), so that you reuse it in the future whenever to re-import the same data or share it with others.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/importer_log.png)

> Note: Refer to the [Import Data Document]( https://docs.nebula-graph.io/3.3.0/nebula-explorer/db-management/11.import-data/)

After importing, we can view the [statistics on the schema view](https://docs.nebula-graph.io/3.3.0/nebula-explorer/db-management/10.create-schema/#view_statistics) page, showing us that 831 players participated in the 2022 Qatar World Cup, serving in 295 different clubs.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/data_stats.png)


## Explore the graph

Let's see what insights we could get from the information/ knowledge in the form of a graph.

### Querying the data

Let's start by showing all the data and see what we will get.

First, with the help of NebulaGraph Explorer, I simply did drag and drop to draw any type of vertex type (TAG) and any type of edge between vertex types (TAG), here we know that all the vertices are connected with others, so no isolated vertices will be missed by this query pattern:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/query-builder-0.png)

Let it generate the query statement for me. Here, it defaults to `LIMIT 100`, so let's change it to something larger (LIMIT 10000) and let it execute in the Console.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/query-builder-1.png)

### Initial observation

The result renders out like this, and you can see that it naturally forms a pattern of clusters.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/bird_view.png)

These peripheral clusters are mostly made up of players from clubs that are not traditionally strong ones (now we learned that they could win, though, who knows!). Many of those clubs have only one or two players who serve in one national team or region, so they are somewhat isolated from other clusters.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/edge_teams.png)

### Graph algorithm-based analysis

After I clicked on the two buttons(Sized by Degrees, Colored by Louvain Algorithm) in Explorer (refer to the [document](https://docs.nebula-graph.com.cn/3.3.0/nebula-explorer/graph-explorer/graph-algorithm/) for details), in the browser, we can see that the entire graph has become something like this:

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/Barcelona.png)

Two graph algorithms are utilized to analyze the insights here.

1. change the display size of vertices to highlight importance using their degrees
2. using Louvain's algorithm to distinguish the community of the vertices

You can see that the big red circle is the famous Barcelona, and its players are marked in red, too.

## Winner Prediction Algorithm

In order to be able to make full use of the graph magic(with the implied conditions and information on the graph), my idea([stolen/inspired from this post](https://cambridge-intelligence.com/fifa-world-cup-2022-prediction/)) is to choose a graph algorithm that considers edges for node importance analysis, to find out the vertices that have higher importance, iterate and rank them globally, and thus get the top team rankings.

These methods actually reflect the fact that excellent players have greater community and connectivity at the same time. Meanwhile, to increase the differentiation between traditionally strong teams, I am going to take into account the information on appearances and goals scored.

Ultimately, my algorithm is:

- Take all the `(player)-serve->(club)` relationships and filter them for players with too few goals and too few goals per game (to balance out the disproportionate impact of older players from some weaker teams)
- Explore outwards from all filtered players to get national teams
- Run the [Betweenness Centrality](https://en.wikipedia.org/wiki/Betweenness_centrality) algorithm on the above subgraph to calculate the node importance scores


## Process of the Prediction

First, we take out the subgraph in the pattern of `(player)-serve->(club)` for those who have scored more than ten goals and have an average of more than 0.2 goals per game.

```cypher
MATCH ()-[e]->()
WITH e LIMIT 10000
WITH e AS e WHERE e.goals > 10 AND toFloat(e.goals)/e.caps > 0.2
RETURN e
```

> Note: For convenience, I have included the number of goals and caps as properties in the serve edge, too.

![](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/query_step0.png)

Then, we select all the vertices on the graph in the left toolbar, select the `belongto` edge of the outgoing direction, expand the graph outwards (traverse), and select the icon that marks the newly expanded vertices as flags.

![treversal_step1](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/treversal_step1.png)

Now that we have the final subgraph, we use the graph algorithm function within the browser to execute BNC (Betweenness Centrality):

![bnc_step2](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/bnc_step2.png)

The graph canvas then looks like this:

![bnc_predict](https://www-cdn.nebula-graph.io/nebula-website-5.0/images/blogs/OpenGPT%20FIFA/bnc_predict.png)

# Result

Ultimately, we sorted according to the value of Betweenness Centrality to get the final winning team: Brazil! 🇧🇷, followed by Belgium, Germany, England, France, and Argentina, so let's wait two weeks to come back and see if the prediction is accurate :D.

The sorted data is as follows:

| Vertex                    | **Betweenness Centrality** |
| ------------------------- | -------------------------- |
| **Brazil**🇧🇷              | 3499                       |
| **Paris Saint-Germain**   | 3073.3333333333300         |
| **Neymar**                | 3000                       |
| **Tottenham Hotspur**     | 2740                       |
| **Belgium**🇧🇪             | 2587.833333333330          |
| **Richarlison**           | 2541                       |
| **Kevin De Bruyne**       | 2184                       |
| **Manchester City**       | 2125                       |
| **İlkay Gündoğan**        | 2064                       |
| **Germany**🇩🇪             | 2046                       |
| **Harry Kane (captain**   | 1869                       |
| **England**🏴󠁧󠁢󠁥󠁮󠁧󠁿              | 1864                       |
| **France**🇫🇷              | 1858.6666666666700         |
| **Argentina**🇦🇷           | 1834.6666666666700         |
| **Bayern Munich**         | 1567                       |
| **Kylian Mbappé**         | 1535.3333333333300         |
| **Lionel Messi (captain** | 1535.3333333333300         |
| **Gabriel Jesus**         | 1344                       |

> Feature Image Credit: The image was also generated with OpenAI, through the DALL-E 2 model & DALL-E 2 Outpainting. See the [original image](https://user-images.githubusercontent.com/1651790/205881462-ff007725-e270-4b1e-9062-7702f01021c1.png).


An attempt to use ChatGPT to generate code for a data scraper to predict sports events with the help of the NebulaGraph graph database and graph algorithms.