[open source announcement] Tencent Transaction Processing Testbed System, 3TS
Tencent Transaction Processing Testbed System (3TS) that is jointly developed by Tencent's CynosDB (TDSQL) team and the Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education of Renmin University of China. The system aims to design and construct a unified framework for transaction processing (including distributed transactions). It enables users to quickly build new concurrency control approaches via the access interface provided by the framework. Based on an comprehensive experiment study over the benchmarks, and the applications abstracted, users can select an optimal concurrency control approach. At present, 3TS has been integrated 13 mainstream concurrency control approaches, and provides common benchmarks such as TPC-C、PPS and YCSB.. 3TS further provides a consistency level test benchmark, to address the issue of system selection difficulty caused by the blowout development of distributed database systems, and provides consistency level discrimination and performance test comparison.
In the near future, 3TS will deeply explore the theory and implementation technology of database transaction processing, and its core concepts are: OPEN, DEPTH and EVOLUTION. Open, adhering to the heart of open source, sharing knowledge and technology; Depth, practicing the spirit of systematic researching the essence of transaction processing technology, "We will never return until we defeat the enemy.”. Evolution, keep moving forward, “Although the road is endless and faraway, we still want to pursue the truth in the world."
As a framework related to transaction processing technology, the essential issues that 3TS is committed to exploring mainly include:
3TS conducts systematic research from the following seven aspects:
Transaction processing performance testbed system. Data management is undergoing changes from SQL to NoSQL, from NoSQL to NewSQL. Although NoSQL has the characteristics of high availability and high scalability, it sacrifices transaction processing characteristics and cannot adapt to current mission-critical applications, even to the case under the most basic social network behaviors. In the latter case, user A and B are becoming mutual friends. This behavior involves two operations: adding A as a friend of B, and adding B as a friend of A. These two operations are considered as an inseparable whole. Without transaction, such a simple social network application cannot be well supported in NoSQL systems. Instead, NewSQL inherits the advantages of relational databases and retains the high availability and high scalability characteristics of NoSQL. We remark that NewSQL is the goal of the current development of distributed database technology. However, building a NewSQL system is extremely challenging. Among them, the construction of distributed transactions with high performance and high scalability is one of the most difficult problems. To address this problem in practice, 3TS provides a development framework for distributed transaction concurrency control approaches, including locking, optimistic concurrency control (OCC), timestamp (TO) and multiple versions to achieve. 3TS has been integrated with the current mainstream. 13 Concurrency control approaches, and provides three general benchmarks, including TPC-C, PPS, YCSB. Through the access interfaces provided by 3TS, users can develop new concurrency control approaches and compare their performance with the current mainstream concurrency control approaches. 3TS aims to provide a practical framework for database system kernel developers and university students who are interested in database research, and help to train key technical personnel for database field.
The concurrency operation leads to different data anomalies in the fields of shared object (memory), database system, and distributed storage system, etc. Transaction technology have been studied systematically for decades, many sub-fields have achieved fruitful results. However, some contents have not been studied thoroughly, and there is a "case by case" research method, which makes the object of study lack of unity and integrity.
Among them, the most typical is the database transaction processing technology, which takes ACID as the technical feature, which aims to ensure the consistency of the data operated by the concurrent transaction, that is, the concurrent transaction scheduling can be serialized (the data has no "data anomalies"). However, the basic question of how many kinds of data anomalies there are in this world is not known. On this problem, the limitation of "case by case" research method is more significant, only individual data anomalies are found, but the relationship between known data anomalies is not known. However, the definition of isolation levels and design concurrent access control algorithms depends on the known limited data anomalies.
In addition, with the arrival of the era of big data, distributed database has moved from the research field to the engineering practice field. A logical system is divided into subsystems of N entities and loses the logic monotonicity of cohesion, which makes the consistency of distributed transaction databases generalize into C of ACID and C of CAP. The serializable technology does not take into account the causal relationship between concurrent transactions and is not perfect. This makes a good opportunity for the research of distributed transaction database, that is, what is the relationship between data anomalies and two kinds of consistency(C of ACID and C of CAP)? Is it enough for current concurrent access control algorithms? There are some consistent studies in the academia and industry, but there is not a unified understanding.
3TS is not an engineering system, but a research system. In this system, there are many subsystems, and more subsystems will be extended according to the aforementioned research scope. At present, 3TS includes Tencent's self-developed transaction testing system and 3TS-Deneva subsystem. In the 3TS, many concurrent access control algorithms are supported, include serializability, conflict serializability, SSI, WSI, BOCC, FOCC, Sundial/Tictoc, Maat, OCC, Silo, TO, Locking (No Wait, Wait Die), Calvin and so on. More concurrent access control algorithms and other related technologies will be supported in the future.
We will provide more detailed documentation describing the technical content included in 3TS.
We will update the follow-up R&D plan one after another, and give a detailed description of each item in the plan.
Short-term plans:
We expect that 3TS is a bridge that can help everyone in the academia and industry to master database transaction processing technology quickly, in depth and breadth. We also hope us to carry out research and make progress together. .
3TS is a young penguin, looking forward to growing up together and efforts of you and me.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。