Skip to content

Commit 9432302

Browse files
SoftlySpokenSoftlySpoken
authored
gpStore Implementation (#49)
* Add gpstore isolated env * remove prebuilt .so * Fix PProcedure LL2double multiple definition * Clean isolate test, to be replaced by Docker image * Add plugins * Update README with Docker * Reorganize data directory structure * Reorganize data directory structure
1 parent d52a76f commit 9432302

File tree

122 files changed

+259505
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

122 files changed

+259505
-2
lines changed

‎gpstore/.gitignore‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
target/
2+
!.mvn/wrapper/maven-wrapper.jar
3+
!**/src/main/**/target/
4+
!**/src/test/**/target/
5+
6+
### IntelliJ IDEA ###
7+
.idea/modules.xml
8+
.idea/jarRepositories.xml
9+
.idea/compiler.xml
10+
.idea/libraries/
11+
*.iws
12+
*.iml
13+
*.ipr
14+
15+
### Eclipse ###
16+
.apt_generated
17+
.classpath
18+
.factorypath
19+
.project
20+
.settings
21+
.springBeans
22+
.sts4-cache
23+
24+
### NetBeans ###
25+
/nbproject/private/
26+
/nbbuild/
27+
/dist/
28+
/nbdist/
29+
/.nb-gradle/
30+
build/
31+
!**/src/main/**/build/
32+
!**/src/test/**/build/
33+
34+
### VS Code ###
35+
.vscode/
36+
37+
### Mac OS ###
38+
.DS_Store

‎gpstore/README.md‎

Lines changed: 140 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,141 @@
1-
# README
1+
# 1. Preparation
22

3-
This is the implementation on gpStore.
3+
## 1.1 Set environment variables
4+
5+
The following absolute paths are for reference only; please adjust them according to the installation paths on your machine.
6+
7+
```Bash
8+
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
9+
export LD_LIBRARY_PATH=/usr/local/lib64:${JAVA_HOME}/jre/lib/amd64/server
10+
export PATH=$PATH:/data/apache-maven-3.9.2/bin
11+
export LC_CTYPE=en_US.UTF-8
12+
export LC_ALL=en_US.UTF-8
13+
```
14+
15+
## 1.2 Download & install dependencies
16+
17+
```bash
18+
# install driver
19+
cd /data
20+
git clone https://github.com/ldbc/ldbc_finbench_driver.git
21+
cd /data/ldbc_finbench_driver
22+
mvn install
23+
24+
# install impl
25+
cd /data
26+
git clone https://github.com/ldbc/ldbc_finbench_transaction_impls.git
27+
cd /data/ldbc_finbench_transaction_impls
28+
mvn clean package
29+
```
30+
31+
Please download our compressed docker image `gpstore-finbench-docker.tar.gz` via [this link](https://disk.pku.edu.cn/link/AA387A00AC18BB464FA58F26F3C37E5CE0).
32+
33+
# 2. Loading and preprocessing
34+
35+
## 2.1 Data
36+
37+
Download FinBench data, decompress it and integrate it into the existing `data/` directory so that the directory hierarchy looks like the following:
38+
39+
```bash
40+
$ cp -r data/intermediate data/finbench-sf1/intermediate
41+
$ cp -r data/intermediate data/finbench-sf10/intermediate
42+
$ tree -d data/ldbc_finbench_transaction_impls/gpstore/data
43+
data/ldbc_finbench_transaction_impls/gpstore/data
44+
├── finbench-sf1
45+
│   ├── incremental
46+
│   ├── intermedidate
47+
│   ├── read_params
48+
│   └── snapshot
49+
└── finbench-sf10
50+
├── incremental
51+
├── intermediate
52+
├── read_params
53+
└── snapshot
54+
```
55+
56+
## 2.2 Convert data
57+
58+
Use sf10 as an example. (Same below without further specification)
59+
60+
```bash
61+
# pwd ldbc_finbench_transaction_impls/gpstore
62+
cp -r scripts data/finbench-sf10/snapshot
63+
mv data/finbench-sf10/intermediate data/sf10/snapshot
64+
cd data/finbench-sf10/snapshot
65+
sh scripts/full.sh
66+
```
67+
68+
## 2.3 Docker image installation & startup
69+
70+
```bash
71+
# Load the image
72+
tar -xzvf gpstore-finbench-docker.tar.gz
73+
docker load -i gpstore-finbench-docker.tar
74+
docker image list # Check that you see REPOSITORY gpstore & TAG finbench
75+
76+
# Startup the container
77+
# Please feel free to replace 9009 by any other available port on your machine; change driver's configuration accordingly (We assume the port is 9090 in the following instructions)
78+
# path/to/datasets: absolute path to ldbc_finbench_transaction_impls/gpstore/data/
79+
# path/to/logs: any path that you have read & write access to
80+
docker run -itd -p 9009:9000 --privileged --cap-add=SYS_PTRACE \
81+
--security-opt seccomp=unconfined --restart always \
82+
-e TZ=Asia/Shanghai \
83+
-v path/to/datasets:/gpstore/datasets \
84+
-v path/to/logs:/gpstore/logs \
85+
--name gpstore-server gpstore:finbench
86+
docker ps -a # Check the STATUS of gpstore-server; it should be UP
87+
```
88+
89+
Please remember to modify the port number in the respective `.properties` file before executing the validation or benchmarking step.
90+
91+
## 2.4 Build plugins [OPTIONAL]
92+
93+
The plugins have been built and deployed inside the docker container; the code is the same as those under the `ldbc_finbench_transaction_impls/gpstore/plugins` path.
94+
95+
Therefore, the following steps are not mandatory. Please only execute them when you have modified one of the plugins and wish to check its effect.
96+
97+
```bash
98+
# Copy the modified plugin into the container
99+
docker cp new_plugin_file_path gpstore-server:/gpstore/plugins
100+
# Build the plugin inside the container
101+
docker exec -it gpstore-server /bin/bash
102+
cd plugins
103+
python3 build.py
104+
# Restart the container to let the modified plugin take effect
105+
exit
106+
docker restart gpstore-server
107+
```
108+
109+
## 2.3 Load data
110+
111+
```bash
112+
curl -X POST -H 'Content-Type: application/json' -d '{"operation":"build","username":"root","password":"123456","db_name":"finbench-sf10"}' http://127.0.0.1:9009/grpc/api
113+
curl -X POST -H 'Content-Type: application/json' -d '{"operation":"load","username":"root","password":"123456","db_name":"finbench-sf10"}' http://127.0.0.1:9009/grpc/api
114+
```
115+
116+
# 3. Benchmark
117+
118+
## 3.1 Validate
119+
120+
### 3.1.1 Create validation
121+
122+
```shell
123+
# create validation
124+
cd /data/ldbc_finbench_transaction_impls/gpstore && sync
125+
bash sf10_finbench_create_validation.sh
126+
```
127+
128+
### 3.1.2 Validate
129+
130+
```shell
131+
cd /data/ldbc_finbench_transaction_impls/gpstore && sync
132+
bash sf10_finbench_validation_database.sh
133+
```
134+
135+
## 3.2 Run benchmark
136+
137+
```shell
138+
# run benchmark
139+
cd /data/ldbc_finbench_transaction_impls/gpstore && sync
140+
bash sf10_finbench_benchmark.sh
141+
```

‎gpstore/benchmark.properties‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
############################################################
2+
# SUT defined configurations #
3+
############################################################
4+
url=http://127.0.0.1:9000
5+
username=root
6+
password=123456
7+
dbname=finbench-sf10
8+
queryDir=queries/
9+
############################################################
10+
# Driver configurations #
11+
############################################################
12+
status=1
13+
thread_count=16
14+
name=LDBC-FinBench
15+
# Modes available: 1.CREATE_VALIDATION 2.VALIDATE_DATABASE 3.EXECUTE_BENCHMARK
16+
mode=EXECUTE_BENCHMARK
17+
results_log=true
18+
time_unit=MICROSECONDS
19+
time_compression_ratio=0.1
20+
peer_identifiers=
21+
workload_statistics=false
22+
spinner_wait_duration=1
23+
help=false
24+
ignore_scheduled_start_times=false
25+
workload=org.ldbcouncil.finbench.driver.workloads.transaction.LdbcFinBenchTransactionWorkload
26+
db=org.ldbcouncil.finbench.impls.gpstore.GpstoreDb
27+
operation_count=1800000
28+
validation_parameters_size=100
29+
validate_workload=true
30+
validate_database=validation_params.csv
31+
warmup=450000
32+
ldbc.finbench.transaction.queries.parameters_dir=data/sf10/read_params
33+
ldbc.finbench.transaction.queries.updates_dir=data/sf10/incremental
34+
# param and update files suffix, `csv` or `parquet`, default is `csv`
35+
ldbc.finbench.transaction.queries.files_suffix=csv
36+
ldbc.finbench.transaction.queries.simple_read_dissipation=0.2
37+
ldbc.finbench.transaction.queries.update_interleave=500
38+
ldbc.finbench.transaction.queries.scale_factor=1
39+
# Frequency of complex read queries
40+
ldbc.finbench.transaction.queries.ComplexRead1_freq=12
41+
ldbc.finbench.transaction.queries.ComplexRead2_freq=45
42+
ldbc.finbench.transaction.queries.ComplexRead3_freq=23
43+
ldbc.finbench.transaction.queries.ComplexRead4_freq=57
44+
ldbc.finbench.transaction.queries.ComplexRead5_freq=83
45+
ldbc.finbench.transaction.queries.ComplexRead6_freq=21
46+
ldbc.finbench.transaction.queries.ComplexRead7_freq=36
47+
ldbc.finbench.transaction.queries.ComplexRead8_freq=51
48+
ldbc.finbench.transaction.queries.ComplexRead9_freq=21
49+
ldbc.finbench.transaction.queries.ComplexRead10_freq=6
50+
ldbc.finbench.transaction.queries.ComplexRead11_freq=12
51+
ldbc.finbench.transaction.queries.ComplexRead12_freq=34
52+
# For debugging purposes
53+
ldbc.finbench.transaction.queries.ComplexRead1_enable=true
54+
ldbc.finbench.transaction.queries.ComplexRead2_enable=true
55+
ldbc.finbench.transaction.queries.ComplexRead3_enable=true
56+
ldbc.finbench.transaction.queries.ComplexRead4_enable=true
57+
ldbc.finbench.transaction.queries.ComplexRead5_enable=true
58+
ldbc.finbench.transaction.queries.ComplexRead6_enable=true
59+
ldbc.finbench.transaction.queries.ComplexRead7_enable=true
60+
ldbc.finbench.transaction.queries.ComplexRead8_enable=true
61+
ldbc.finbench.transaction.queries.ComplexRead9_enable=true
62+
ldbc.finbench.transaction.queries.ComplexRead10_enable=true
63+
ldbc.finbench.transaction.queries.ComplexRead11_enable=true
64+
ldbc.finbench.transaction.queries.ComplexRead12_enable=true
65+
ldbc.finbench.transaction.queries.SimpleRead1_enable=true
66+
ldbc.finbench.transaction.queries.SimpleRead2_enable=true
67+
ldbc.finbench.transaction.queries.SimpleRead3_enable=true
68+
ldbc.finbench.transaction.queries.SimpleRead4_enable=true
69+
ldbc.finbench.transaction.queries.SimpleRead5_enable=true
70+
ldbc.finbench.transaction.queries.SimpleRead6_enable=true
71+
ldbc.finbench.transaction.queries.Write1_enable=true
72+
ldbc.finbench.transaction.queries.Write2_enable=true
73+
ldbc.finbench.transaction.queries.Write3_enable=true
74+
ldbc.finbench.transaction.queries.Write4_enable=true
75+
ldbc.finbench.transaction.queries.Write5_enable=true
76+
ldbc.finbench.transaction.queries.Write6_enable=true
77+
ldbc.finbench.transaction.queries.Write7_enable=true
78+
ldbc.finbench.transaction.queries.Write8_enable=true
79+
ldbc.finbench.transaction.queries.Write9_enable=true
80+
ldbc.finbench.transaction.queries.Write10_enable=true
81+
ldbc.finbench.transaction.queries.Write11_enable=true
82+
ldbc.finbench.transaction.queries.Write12_enable=true
83+
ldbc.finbench.transaction.queries.Write13_enable=true
84+
ldbc.finbench.transaction.queries.Write14_enable=true
85+
ldbc.finbench.transaction.queries.Write15_enable=true
86+
ldbc.finbench.transaction.queries.Write16_enable=true
87+
ldbc.finbench.transaction.queries.Write17_enable=true
88+
ldbc.finbench.transaction.queries.Write18_enable=true
89+
ldbc.finbench.transaction.queries.Write19_enable=true
90+
ldbc.finbench.transaction.queries.ReadWrite1_enable=true
91+
ldbc.finbench.transaction.queries.ReadWrite2_enable=true
92+
ldbc.finbench.transaction.queries.ReadWrite3_enable=true

‎gpstore/create_validation.properties‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
############################################################
2+
# SUT defined configurations #
3+
############################################################
4+
url=http://222.128.12.69:9009
5+
username=root
6+
password=123456
7+
dbname=finbench-sf10
8+
queryDir=queries/
9+
############################################################
10+
# Driver configurations #
11+
############################################################
12+
status=1
13+
thread_count=1
14+
name=LDBC-FinBench
15+
# Modes available: 1.CREATE_VALIDATION 2.VALIDATE_DATABASE 3.EXECUTE_BENCHMARK
16+
mode=CREATE_VALIDATION
17+
results_log=true
18+
time_unit=MICROSECONDS
19+
time_compression_ratio=1
20+
peer_identifiers=
21+
workload_statistics=false
22+
spinner_wait_duration=1
23+
help=false
24+
ignore_scheduled_start_times=false
25+
workload=org.ldbcouncil.finbench.driver.workloads.transaction.LdbcFinBenchTransactionWorkload
26+
db=org.ldbcouncil.finbench.impls.gpstore.GpstoreDb
27+
operation_count=10000
28+
validation_parameters_size=10000
29+
validate_workload=true
30+
validate_database=validation_params.csv
31+
warmup=0
32+
ldbc.finbench.transaction.queries.parameters_dir=data/sf10/read_params
33+
ldbc.finbench.transaction.queries.updates_dir=data/sf10/incremental
34+
# param and update files suffix, `csv` or `parquet`, default is `csv`
35+
ldbc.finbench.transaction.queries.files_suffix=csv
36+
ldbc.finbench.transaction.queries.simple_read_dissipation=0.2
37+
ldbc.finbench.transaction.queries.update_interleave=10000000
38+
ldbc.finbench.transaction.queries.scale_factor=1
39+
# Frequency of complex read queries
40+
ldbc.finbench.transaction.queries.ComplexRead1_freq=1
41+
ldbc.finbench.transaction.queries.ComplexRead2_freq=1
42+
ldbc.finbench.transaction.queries.ComplexRead3_freq=1
43+
ldbc.finbench.transaction.queries.ComplexRead4_freq=1
44+
ldbc.finbench.transaction.queries.ComplexRead5_freq=1
45+
ldbc.finbench.transaction.queries.ComplexRead6_freq=1
46+
ldbc.finbench.transaction.queries.ComplexRead7_freq=1
47+
ldbc.finbench.transaction.queries.ComplexRead8_freq=1
48+
ldbc.finbench.transaction.queries.ComplexRead9_freq=1
49+
ldbc.finbench.transaction.queries.ComplexRead10_freq=1
50+
ldbc.finbench.transaction.queries.ComplexRead11_freq=1
51+
ldbc.finbench.transaction.queries.ComplexRead12_freq=1
52+
# For debugging purposes
53+
ldbc.finbench.transaction.queries.ComplexRead1_enable=true
54+
ldbc.finbench.transaction.queries.ComplexRead2_enable=true
55+
ldbc.finbench.transaction.queries.ComplexRead3_enable=true
56+
ldbc.finbench.transaction.queries.ComplexRead4_enable=true
57+
ldbc.finbench.transaction.queries.ComplexRead5_enable=true
58+
ldbc.finbench.transaction.queries.ComplexRead6_enable=true
59+
ldbc.finbench.transaction.queries.ComplexRead7_enable=true
60+
ldbc.finbench.transaction.queries.ComplexRead8_enable=true
61+
ldbc.finbench.transaction.queries.ComplexRead9_enable=true
62+
ldbc.finbench.transaction.queries.ComplexRead10_enable=true
63+
ldbc.finbench.transaction.queries.ComplexRead11_enable=true
64+
ldbc.finbench.transaction.queries.ComplexRead12_enable=true
65+
ldbc.finbench.transaction.queries.SimpleRead1_enable=true
66+
ldbc.finbench.transaction.queries.SimpleRead2_enable=true
67+
ldbc.finbench.transaction.queries.SimpleRead3_enable=true
68+
ldbc.finbench.transaction.queries.SimpleRead4_enable=true
69+
ldbc.finbench.transaction.queries.SimpleRead5_enable=true
70+
ldbc.finbench.transaction.queries.SimpleRead6_enable=true
71+
ldbc.finbench.transaction.queries.Write1_enable=true
72+
ldbc.finbench.transaction.queries.Write2_enable=true
73+
ldbc.finbench.transaction.queries.Write3_enable=true
74+
ldbc.finbench.transaction.queries.Write4_enable=true
75+
ldbc.finbench.transaction.queries.Write5_enable=true
76+
ldbc.finbench.transaction.queries.Write6_enable=true
77+
ldbc.finbench.transaction.queries.Write7_enable=true
78+
ldbc.finbench.transaction.queries.Write8_enable=true
79+
ldbc.finbench.transaction.queries.Write9_enable=true
80+
ldbc.finbench.transaction.queries.Write10_enable=true
81+
ldbc.finbench.transaction.queries.Write11_enable=true
82+
ldbc.finbench.transaction.queries.Write12_enable=true
83+
ldbc.finbench.transaction.queries.Write13_enable=true
84+
ldbc.finbench.transaction.queries.Write14_enable=true
85+
ldbc.finbench.transaction.queries.Write15_enable=true
86+
ldbc.finbench.transaction.queries.Write16_enable=true
87+
ldbc.finbench.transaction.queries.Write17_enable=true
88+
ldbc.finbench.transaction.queries.Write18_enable=true
89+
ldbc.finbench.transaction.queries.Write19_enable=true
90+
ldbc.finbench.transaction.queries.ReadWrite1_enable=true
91+
ldbc.finbench.transaction.queries.ReadWrite2_enable=true
92+
ldbc.finbench.transaction.queries.ReadWrite3_enable=true

‎gpstore/data/intermediate/Account.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
accountId ID
2+
createTime LONG
3+
isBlocked BOOLEAN
4+
accountType STRING
5+
nickname STRING
6+
phonenum STRING
7+
email STRING
8+
freqLoginType STRING
9+
lastLoginTime LONG
10+
accountLevel STRING
11+

‎gpstore/data/intermediate/AccountRepayLoan.csv‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
accountId START_ID
2+
loanId END_ID
3+
amount FLOAT
4+
createTime LONG
5+
comment STRING
6+

‎gpstore/data/intermediate/AccountTransferAccount.csv‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
fromId START_ID
2+
toId END_ID
3+
amount FLOAT
4+
createTime LONG
5+
orderNum LONG
6+
comment STRING
7+
payType STRING
8+
goodsType STRING
9+

0 commit comments

Comments
 (0)