payload: {"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/5156","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/5156/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/5156/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/5156/events","html_url":"https://github.com/scylladb/scylla/pull/5156","id":504701533,"node_id":"MDExOlB1bGxSZXF1ZXN0MzI2MjkzOTky","number":5156,"title":"distributed_loader: populate non-system keyspaces in parallel","user":{"login":"piodul","id":4950936,"node_id":"MDQ6VXNlcjQ5NTA5MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/4950936?v=4","gravatar_id":"","url":"https://api.github.com/users/piodul","html_url":"https://github.com/piodul","followers_url":"https://api.github.com/users/piodul/followers","following_url":"https://api.github.com/users/piodul/following{/other_user}","gists_url":"https://api.github.com/users/piodul/gists{/gist_id}","starred_url":"https://api.github.com/users/piodul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piodul/subscriptions","organizations_url":"https://api.github.com/users/piodul/orgs","repos_url":"https://api.github.com/users/piodul/repos","events_url":"https://api.github.com/users/piodul/events{/privacy}","received_events_url":"https://api.github.com/users/piodul/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2019-10-09T14:47:13Z","updated_at":"2019-10-10T12:13:50Z","closed_at":"2019-10-10T12:13:50Z","author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/scylladb/scylla/pulls/5156","html_url":"https://github.com/scylladb/scylla/pull/5156","diff_url":"https://github.com/scylladb/scylla/pull/5156.diff","patch_url":"https://github.com/scylladb/scylla/pull/5156.patch"},"body":"Before this change, when populating non-system keyspaces, each data\r\ndirectory was scanned and for each entry (keyspace directory),\r\na keyspace was populated. This was done in a serial fashion - populating\r\nof one keyspace was not started until the previous one was done.\r\n\r\nLoading keyspaces in such fashion can introduce unnecessary waiting\r\nin case of a large number of keyspaces in one data directory. Population\r\nprocess is I/O intensive and barely uses CPU.\r\n\r\nThis change enables parallel loading of keyspaces per data directory.\r\nPopulating the next keyspace does not wait for the previous one.\r\n\r\nA benchmark was performed measuring startup time, with the following\r\nsetup:\r\n  - 1 data directory,\r\n  - 200 keyspaces,\r\n  - 2 tables in each keyspace, with the following schema:\r\n      CREATE TABLE tbl (a int, b int, c int, PRIMARY KEY(a, b))\r\n        WITH CLUSTERING ORDER BY (b DESC),\r\n  - 1024 rows in each table, with values (i, 2*i, 3*i) for i in 0..1023,\r\n  - ran on 6-core virtual machine running on i7-8750H CPU,\r\n  - compiled in dev mode,\r\n  - parameters: --smp 6 --max-io-requests 4 --developer-mode=yes\r\n      --datadir $DIR --commitlog-directory $DIR\r\n      --hints-directory $DIR --view-hints-directory $DIR\r\n\r\nThe benchmark tested:\r\n  - boot time, by comparing timestamp of the first message in log,\r\n    and timestamp of the following message:\r\n      \"init - Scylla version ... initialization completed.\"\r\n  - keyspace population time, by comparing timestamps of messages:\r\n      \"init - loading non-system sstables\"\r\n    and\r\n      \"init - starting view update generator\"\r\n\r\nThe benchmark was run 5 times for sequential and parallel version,\r\nwith the following results:\r\n  - sequential: boot 31.620s, keyspace population 6.051s\r\n  - parallel:   boot 29.966s, keyspace population 4.360s\r\n\r\nKeyspace population time decreased by ~27.95%, and overall boot time\r\nby about ~5.23%.\r\n\r\nTests: unit(release)\r\n\r\nFixes #2007"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/540543437","html_url":"https://github.com/scylladb/scylla/pull/5156#issuecomment-540543437","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/5156","id":540543437,"node_id":"MDEyOklzc3VlQ29tbWVudDU0MDU0MzQzNw==","user":{"login":"avikivity","id":1017210,"node_id":"MDQ6VXNlcjEwMTcyMTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1017210?v=4","gravatar_id":"","url":"https://api.github.com/users/avikivity","html_url":"https://github.com/avikivity","followers_url":"https://api.github.com/users/avikivity/followers","following_url":"https://api.github.com/users/avikivity/following{/other_user}","gists_url":"https://api.github.com/users/avikivity/gists{/gist_id}","starred_url":"https://api.github.com/users/avikivity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avikivity/subscriptions","organizations_url":"https://api.github.com/users/avikivity/orgs","repos_url":"https://api.github.com/users/avikivity/repos","events_url":"https://api.github.com/users/avikivity/events{/privacy}","received_events_url":"https://api.github.com/users/avikivity/received_events","type":"User","site_admin":false},"created_at":"2019-10-10T12:13:50Z","updated_at":"2019-10-10T12:13:50Z","author_association":"CONTRIBUTOR","body":"Queued as c04e8c37aa7b1bfcf301b06cfea6be978958e4ec"}}
other: {"actor":{"display_login":"avikivity"}}