CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal
This is a mirror (with minor changes) to the site, where CDStore is first released.
CDStore builds on an augmented secret sharing scheme called convergent dispersal, which supports deduplication by using deterministic content-derived hashes as inputs to secret sharing. It combines convergent dispersal with two-stage deduplication to achieve both bandwidth and storage savings and be robust against side-channel attacks.
Mingqiang Li, Chuan Qin, Jingwei Li, and Patrick P. C. Lee. “CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal.” IEEE Internet Computing, 20(3), pp. 45-53, May-June 2016 (Special issue: Cloud Storage).
(An earlier version appeared in USENIX ATC 2015)
Mingqiang Li, Chuan Qin, and Patrick P. C. Lee “CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal.” Proceedings of USENIX Annual Technical Conference (ATC 2015), Santa Clara, CA, July 2015.
CDStore is built on Ubuntu 12.04.3 LTS with gcc version 4.6.3. This software requires the following libraries:
GF-Complete and Leveldb have been packed in
Install OpenSSL, Boost C++ library and snappy (that is necessary for Leveldb) by the following command:
$ sudo apt-get install libssl1.0.0 libboost-all-dev libsnappy-dev
Type the following commands to compile GF_Complete in
$ cd client/lib/gf_complete $ ./configure $ make $ sudo make install
After installing the necessary libraries, following the instructions to configure CDStore server and client.
CDStore requires at least 4 servers for reliability. You need to configure and
make them, respectively. In a successfully configured server, it has a directory
server/meta/ which stores the deduplication index, file recipes and share containers.
After a successful
make, you will get the server program
SERVER. You can start it by the following command, where
[port] is the port that CDStore server serves in.
$ ./SERVER [port]
Modify the configuration file
client/config to specify the server information. For example, if you have run 4 servers with
./SERVER [port] on machines:
0.0.0.0 with port 11030 0.0.0.0 with port 11031 0.0.0.0 with port 11032 0.0.0.0 with port 11033
You need to change
0.0.0.0:11030 0.0.0.0:11031 0.0.0.0:11032 0.0.0.0:11033
Optionally, you can make advanced configure by modifying the total number of servers (4 by default), fault tolerance degree (1 by default), and security degree (0 by default) in
After all configurations, change directory to
client/ and type
make to create client program
We provide scripts
auto_clean.sh for quick configurations. You can run
./auto_config.sh to compile and configure 4 servers, and run
./auto_clean.sh to reset all configurations.
You can use the executable file
client/) in the following way:
usage: ./CLIENT [filename] [userID] [action] [secutiyType] - [filename]: full path of the file; - [userID]: user ID of current client; - [action]: [-u] upload; [-d] download; - [securityType]: [HIGH] AES-256 & SHA-256; [LOW] AES-128 & SHA-1
We show two usage examples:
// upload a file `test` from user 0 using high security mechanism (e.g., AES-256 & SHA-256) $ ./CLIENT test 0 -u HIGH // download a file `test`, from user 1 using low security mechanism (e.g., AES-128 & SHA-1) // the downloaded file will be renamed to be test.d automatically $ ./CLIENT test 1 -d LOW
Limitations & Known Bugs
We assume the upload and download channels are secure (e.g., encrypted and authenticated), and do not implement mechanism for protection. We also assume that the user ID (input by CDStore client) is correct, so as to ensure the two-stage deduplication robust against side-channel attacks.
When encode a test file generated via
urandom, CDStore sometimes aborts with some errors like
Error in region multiply operation. The source & destination pointers must be aligned with respect to each other along a 16 byte boundary.
The error is possibly due to the library
gf_completethat cannot encode random content.
- Version 1.0.2 (Dec 2017)
- Fix bug in download of Version 1.0.1
- Version 1.0.1 (Dec 2015)
- Encode the file pathname through CAONT-RS
- Version 1.0.0 (May 2015)
- First release
- Current maintainers
- Yanjing Ren, UESTC, firstname.lastname@example.org
- JinGang Ma, UESTC, email@example.com
- Original maintainers
- Chuan Qin, CUHK, firstname.lastname@example.org
- Mingqiang Li, CUHK, email@example.com