代码拉取完成，页面将自动刷新
A distributed block storage system that uses mature Raft protocol and is designed for all-flash scenarios.
The current distribution block storage system(Ceph) is facing challenges that hinder its ability to meet the needs of performance, latency, cost, and stability. The main issues are:
fastblock is designed to tackle the challenges of performance and latency issues in distributed block storage system. Its key features include:
The architecture is closely similar to Ceph, including many concepts such as monitor, OSD, and PG. For quick understanding, the architecture diagram is shown below.
osdMap, pg, pgMap, pool, and image.monitor is responsible for maintaining the status of storage nodes and managing node additions and deletions. It also handles the metadata for storage volumes, maintains the cluster's topological structure, responds to user requests for creating pools, and creates Raft groups on OSDs based on the current topology.
As a cluster management tool, monitor does not store data itself and does not aim for extreme performance. Therefore, it is implemented in Golang and uses etcd for multi-replica storage.
The monitor cluster is crucial for ensuring consistency, as it provides a unified view to both clients and OSDs. For all client I/O operations, only the PG layer is visible. Both OSDs and clients initiate a timer at startup to periodically fetch osdmap and pgmap information from monitor. This ensures that all OSDs and clients perceive the same changes in PG status and respond accordingly. This design prevents write operations targeted at a specific PG from being incorrectly directed.
For more details, refer to monitor documentation.
RPC subsystem in fastblock is a crucial system that interconnects various modules. To accommodate heterogeneous networks, the RPC subsystem is implemented in two ways: socket-based (Control RPC) and RDMA-based (Data RPC and Raft RPC). Socket-based RPC follows the classic Linux socket application scenario, while RDMA-based RPC utilizes asynchronous RDMA (i.e., RDMA write) semantics.
There are three types of RPC interactions, as depicted in the diagram:
osdmap, pgmap, and image information between clients and monitor, and between OSDs and monitor. Since these data are not large in volume and are not frequently transmitted, a socket-based implementation is suitable.protobuf. The network communication parts employ RDMA, and RPC data serialization is managed using Protobuf.Raft achieves consistency in distributed systems by electing a leader and entrusting them with the responsibility of managing the replication log. The leader receives log entries from clients, replicates these entries to other servers, and coordinates when these entries can be safely applied to their state machines. There are many open-source implementations of Raft, and we referenced Willemt's C language raft implementation and additionally implemented multi-raft, which mainly includes:
In a multi-group Raft setup, where multiple Raft groups coexist, each group's leader must send heartbeats to its followers. With many Raft groups, this could lead to an excessive number of heartbeats, consuming significant bandwidth and CPU resources. The solution is elegantly simple: since an osd might belong to multiple Raft groups, heartbeats can be consolidated for groups with the same leader and followers. This consolidation significantly reduces the number of heartbeat messages needed. For instance, consider a scenario with two PGs in Raft, namely pg1 and pg2, where both include osd1, osd2, and osd3. osd1 is the leader in this case. Without consolidation, osd1 would need to send separate heartbeat messages for pg1 and pg2 to both osd2 and osd3. However, with heartbeat consolidation, osd1 only needs to send one combined heartbeat message to osd2 and osd3.
The KV (Key-Value) subsystem is tailored for storing both the metadata of Raft and the data of the storage system itself. The design caters to the small scale of the data involved. It employs an in-memory hash map to store all the data, offering basic operations like put, remove, and get. To ensure data persistence, the modified data in the hash map is written to the disk at intervals of 10 milliseconds.
OSD localstore utilizes SPDK blobstore for data storage and is comprised of three key storage modules:
disk_log: This module is responsible for storing the Raft log. Each PG corresponding to a Raft group, is associated with a specific SPDK blob.object_store: This module handles the storage of object data, where each object is mapped to an SPDK blob.kv_store: Each CPU core has its own SPDK blob, which stores all kv data required by that core. This includes Raft's metadata and the data of the storage system itself.
For instance, if two Rafts are running, the localstore provides these three types of storage functionalities - log, object, and KV - for both Rafts.
Clients is designed for creating, modifying, and deleting images. It converts user operations on images into operations on objects (the basic data units processed by OSDs) and then packages these as Data PRC messages to be sent to the leader OSD of the PG. The client also receives and processes responses from the leader OSD and returns the results to the user. Clients operate in various modes to accommodate different environments and uses. These modes include spdk vhost for virtual machines, NBD for bare metal, and CSI for virtual machines. In all these modes, the libfastblock library is called to handle the conversion from images to objects and to communicate with the OSDs. The focus of this explanation is on the spdk vhost for virtual machines.
The client uses the SPDK library to create a vhost app. After initializing SPDK resources, a timer is set up to fetch osdmap, pgmap and image information from monitor. An SPDK script (rpc.py) is used to send requests to the vhost app for creating bdev (bdev_fastblock_create). Upon receiving a request, the vhost app creates an image, sends its information to the monitor, creates a bdev device, and then registers its operation interface (which calls the libfastblock library). The rpc.py script is also used to send requests to the vhost application for creating vhost-blk controller (vhost_create_blk_controller). Upon receiving the request, the vhost app opens the bdev device and registers a vhost driver to handle vhost message. This involves creating a socket for client connections (such as QEMU) and implementing connection services following the vhost protocol, a feature already estabilished in DPDK. Libfastblock converts user operations on images into operations on objects, encapsulates these into Data RPC messages, sends them to the leader OSD of the PG, and handles the responses from the leader OSD.
The source code is primarily organized into three directories: src, monitor, and msg, each serving distinct functions:
src directory primarily contains the implementation of Raft, RDMA communication, the underlying storage engine, and block layer API encapsulations. For more details, you can refer to the documentation of src.monitor directory includes features related to cluster metadata storage management, monitor elections, PG allocation, and distribution of clustermap. Detailed information about this directory can be found in the documentation of monitor.msg directory contains all the implementations of RDMA RPC, along with a simple demo illustrating its use.For the first compilation, you need to acquire dependencies such as SPDK and abseil-cpp by running the following command:
./install-deps.sh
You can compile the Release versions of monitor and osd by running the following command:
./build.sh -t Release -c monitor
./build.sh -t Release -c osd
After compilation, the binaries for fastblock-mon and fastblock-client will be located in the mon/ directory, while fastblock-osd and fastblock-vhost binaries will be found in the build/src/osd/ and build/src/bdev directories, respectively. For subsequent modifications, if there are code changes in the OSD or vhost, recompilation can be done directly in the build/ directory. Similarly, for any updates to the monitor, a simple make command in the mon/ directory suffices.
According to the Deployment and Performance Test Report, in our test environment, with each OSD utilizing only one core, we achieved a latency of less than 100 microseconds for 4K random writes in a single thread, along with concurrent performance of 410,000 IOPS.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。