JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in object storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various database engines such as Redis, MySQL, and SQLite based on the scenarios and requirements.
With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage.
📺 Video: What is JuiceFS?
📖 Document: Quick start guide
Architecture | Getting Started | Advanced Topics | POSIX Compatibility | Performance Benchmark | Supported Object Storage | Who is using | Roadmap | Reporting Issues | Contributing | Community | Usage Tracking | License | Credits | FAQ
JuiceFS consists of three parts:
JuiceFS can store the metadata of file system on Redis, which is a fast, open-source, in-memory key-value data storage, particularly suitable for storing metadata; meanwhile, all the data will be stored in object storage through JuiceFS client. Learn more
Each file stored in JuiceFS is split into "Chunk" s at a fixed size with the default upper limit of 64 MiB. Each Chunk is composed of one or more "Slice"(s), and the length of the slice varies depending on how the file is written. Each slice is composed of size-fixed "Block" s, which are 4 MiB by default. These blocks will be stored in object storage in the end; at the same time, the metadata information of the file and its Chunks, Slices, and Blocks will be stored in metadata engines via JuiceFS. Learn more
When using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in object storage. Therefore, the source files stored in JuiceFS cannot be found in the file browser of the object storage platform; instead, there are only a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic! This is just the secret of the high-performance operation of JuiceFS!
Before you begin, make sure you have:
Please refer to Quick Start Guide in the community doc (or doc in this repo) to start using JuiceFS right away!
Check out all the command line options in command reference.
It is also very easy to use JuiceFS on Kubernetes. Please find more information here.
If you wanna use JuiceFS in Hadoop, check Hadoop Java SDK.
Please refer to JuiceFS User Manual for more information.
JuiceFS has passed all of the compatibility tests (8813 in total) in the latest pjdfstest .
All tests successful.
Test Summary Report
-------------------
/root/soft/pjdfstest/tests/chown/00.t (Wstat: 0 Tests: 1323 Failed: 0)
TODO passed: 693, 697, 708-709, 714-715, 729, 733
Files=235, Tests=8813, 233 wallclock secs ( 2.77 usr 0.38 sys + 2.57 cusr 3.93 csys = 9.65 CPU)
Result: PASS
Aside from the POSIX features covered by pjdfstest, JuiceFS also provides:
JuiceFS provides a subcommand that can run a few basic benchmarks to help you understand how it works in your environment:
A sequential read/write benchmark has also been performed on JuiceFS, EFS and S3FS by fio.
Above result figure shows that JuiceFS can provide 10X more throughput than the other two (see more details).
A simple mdtest benchmark has been performed on JuiceFS, EFS and S3FS by mdtest.
The result shows that JuiceFS can provide significantly more metadata IOPS than the other two (see more details).
There is a virtual file called .accesslog
in the root of JuiceFS to show all the details of file system operations and the time they take, for example:
$ cat /jfs/.accesslog
2021.01.15 08:26:11.003330 [uid:0,gid:0,pid:4403] write (17669,8666,4993160): OK <0.000010>
2021.01.15 08:26:11.003473 [uid:0,gid:0,pid:4403] write (17675,198,997439): OK <0.000014>
2021.01.15 08:26:11.003616 [uid:0,gid:0,pid:4403] write (17666,390,951582): OK <0.000006>
The last number on each line is the time (in seconds) that the current operation takes. You can directly use this to debug and analyze performance issues, or try ./juicefs profile /jfs
to monitor real time statistics. Please run ./juicefs profile -h
or refer to here to learn more about this subcommand.
JuiceFS supports almost all object storage services. Learn more.
JuiceFS is still in beta quality, and the core storage format is not stabilized yet. Thus, please do a careful and thorough evaluation before using JuiceFS in a production environment. If you are interested, feel free to do tests and give us feedback.
You are also welcome to share your experience of using JuiceFS with us and others. Additionally, we have collected a summary list in ADOPTERS.md, which includes other open source projects used with JuiceFS.
We use GitHub Issues to track community reported issues. You can also contact the community for any questions.
Thank you for your contribution! Please refer to the CONTRIBUTING.md for more information.
Welcome to join the Discussions and the Slack channel to connect with JuiceFS team members and other users.
JuiceFS collects anonymous usage data by default to help us better understand how the community is using JuiceFS. Only core metrics (e.g. version number) will be reported, and user data and any other sensitive data will not be included. The related code can be viwed here.
You could also disable reporting easily by command line option --no-usage-report
:
$ ./juicefs mount --no-usage-report
JuiceFS is open-sourced under Apache License 2.0, see LICENSE.
The design of JuiceFS was inspired by Google File System, HDFS and MooseFS. Thanks for their great work!
JuiceFS supports many object storage. Please check out this list first. If the object storage you want to use is compatible with S3, you could treat it as S3. Otherwise, try reporting issue.
The simple answer is no. JuiceFS uses Redis transaction to guarantee the atomicity of metadata operations, which is not well supported by cluster mode. For this, sentinal or other Redis HA solution are needed.
See "Redis Best Practices" for more information.
See "Comparison with Others" for more information.
For more FAQs, please see the full list.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。