-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ..::[ The Bitcoin Foundation: NMON Performance Tests ]::.. [ Date: 2015.05.26 ] [ Author: mod6 [R.01] ] 0x0] Introduction Over the month of May, mod6 conducted a series of system level performance tests during a full sync of the bitcoin blockchain for each version. The metrics were captured on a three second interval with NMON [R.02] against three different versions of the Reference Implementation: 0x01: v0.5.3 [Original] [R.03] with SHA256 [R.04], 0x02: v0.5.3.1-RELEASE [R.05] with SHA256 [R.06], 0x03: v0.5.3.1-RELEASE [R.05] with SHA256 [R.06] plus asciilifeform's [R.07] OrphanageThermonuke Patch submitted [R.08] to the btc-dev [R.09] email list. All of the charts and captured data have been posted to the Bitcoin Foundation website in the following web directories: 0x01: v0.5.3 [Original] [R.0A] 0x02: v0.5.3.1-RELEASE [R.0B] 0x03: v0.5.3.1-RELEASE + patch { OrphanageThermonuke } [R.0C] mod6 measured only one significant difference between the three variants: while 0x01 and 0x02 exhaust system memory and trigger OOM-kills, 0x03 syncs the full blockchain without requiring manual intervention. 0x1] Test Environment [ Host Environment ]: The performance tests were conducted on an `m3.medium' AWS Debian-6 ``Squeeze'' AMD64 EBS AMI. This size of instance comes with 3.75Gb of RAM allocated. Main disk storage [/dev/sda] for the host environment was not utilized; all disk IO for the tests were conducted on attached volume and not reflected in disk R/W charts [Disk.png]. [ Local Environment Processes ]: During the conducting of these performance tests, the only third party processes running were `bitcoind' and `nmon'. 0x2] Notable Test Observations & Test Results [ OOM-KILL ]: The `bitcoind' process received the ``OOM-KILL'' signal from the kernel two times during the full sync of the blockchain for versions: [ 0x01: v0.5.3 [R.03] ]: In the following charts for the 0x01 performance test, the OOM-KILL signal occurred on 20150520 & 20150522. The times where this occurred is correlated in the graph data as follows: All_CPUs.png [R.0D] : CPU Utilization drop to 5% on 20150520 and 10% on 20150522. CPU.png [R.0E] : CPU Utilization drop to 5% on 20150520 and 10% on 20150522. Memory.png [R.0F] : Active and Inactive Memory drops to about 200MB on 20150520 and about 1`200MB on 20150522. It seems that the ``OOM-KILL'' signal is received when Active Memory nears the 3`000MB boundary. Network.png [R.10] : Total Network Utilization drops to about 60KB/s on 20150520 and about 40KB/s on 20150522. Process_Switches.png [R.11] : Process Switches spiked to about 2900 Switches/s on 20150520 and about 3300 Switches/s on 20150522. Run_Queue.png [R.12] : The Run Queue length dropped to about 1.2 on 20150520 and about 1.3 on 20150522. There was a third drop on 20150524, but is not related to a OOM-KILL signal. [ 0x02: v0.5.3.1-RELEASE [R.05] ]: All_CPUs.png [R.13] : CPU Utilization drop to 0% on 20150515 and 0% on 20150517. CPU.png [R.14] : CPU Utilization drop to 0% on 20150515 and 0% on 20150517. Memory.png [R.15] : Active and Inactive Memory drops to about 0MB on 20150515 and about 0MB on 20150517. It seems that the ``OOM-KILL'' signal is received when Active Memory nears the 3`000MB boundary. Network.png [R.16] : Total Network Utilization drops to about 0KB/s on 20150515 and about 0KB/s on 20150517. Process_Switches.png [R.17] : Process Switches spiked to about 4`000 Switches/s on 20150515 and about 2`000 Switches/s on 20150517; both of these spikes occurred right before they went to near 0. Run_Queue.png [R.18] : The Run Queue length dropped to about 1.0 on 20150515 and about 1.0 on 20150517. [ Differences Noted in Test Phase ]: A major difference between these two performance test runs: when the ``OOM-KILL'' signal occured in test 0x02 there was a greater time interval [hours] between the signal and when `bitcoind' was restarted. This may account for why CPU/Mem/Network/Process Switches/Run Queue reached minimal values, where they did not completely in test 0x01. During 0x01, `bitcoind' was restarted much quicker, within about 60 minutes. [ 0x03: No ``OOM-KILL'' with OrphanageThermonuke patch applied ]: It is well worth noting here that this performance test of 0x03 never never received an ``OOM-KILL'' signal during full blockchain sync. The charts for this test also reflect this fact as there are no precipitous graph fluctuations as there were with 0x01 and 0x02. [ CPU Utilization ]: Across all three full blockchain sync tests the data shows that the range of CPU Utilization hovers in the range between 30% to 50% [R.0D] [R.13] [R.19], with the highest spike near 60% [R.13] on test 0x02. [ Memory Utilization ]: Across all three full blockchain sync tests the Memory charts [R.0F] [R.15] [R.1A] show that the range of utilized Active Memory is approximately 1`800MB to 3`000MB. However, there is a notable difference with 0x03: Active Memory seems to peak around 2`400MB and then drop and stabilize around 1`800MB for the duration of the sync process. As was noted in the above section on ``OOM-KILL'' it seems to be that when Active Memory approaches 3`000MB there is a ``OOM-KILL'' signal following shortly thereafter. This environment has 3.75Gb of RAM available to it [3`750MB] which translates to about an 80% utilization boundary of Active Memory. [ Total Network Utilization [Reads/Writes] ]: Network utilization during 0x01 and 0x02 the tests range [while running] between troughs of around 80KB/s and peaks of about 340KB/s. The 0x03 test, has a notably higher utilization range of around 150KB/s troughs with peaks about 600KB/s. It should also be noted that Total Network Utilization during test 0x03, the Reference Implementation operated in the upper bounds of the range for about the first day and a half of the blockchain sync process and then proceeded to taper to lower levels. 0x3] Conclusion NMON proved to be a useful tool for gathering System Level performance metrics while the Reference Implementation was achieving full blockchain sync. In the future, we may utilize this tool again. However, we plan to do further testing and analysis with tools that are aimed at the process level to achieve better granularity. [ References ]: [R.01]: 027A 8D7C 0FB8 A166 4372 0F40 7217 05A8 B71E ADAF [R.02]: http://nmon.sourceforge.net/pmwiki.php [R.03] http://thebitcoin.foundation/v0.5.3-0-gd05c03a.tar.gz [R.04]: aab1f8ea8c7f131ff69dfa3b9437ba35531018be760132dd6373f41a591f6382 [R.05]: http://thebitcoin.foundation/bitcoin-v0_5_3_1-RELEASE.tar.gz [R.06]: 5c41fe6cf286770a25bf61ab0c35747d0c760f8656754296d2e1d3c4274b5686 [R.07]: 1721 5D11 8B72 3950 7FAF ED98 B982 28A0 01AB FFC7 [R.08]: http://thebitcoin.foundation/ml/btc-dev/2015-May/000088.html [R.09]: http://thebitcoin.foundation/mailman/listinfo/btc-dev URL="http://thebitcoin.foundation/test/perf" [R.0A]: $URL/v0_5_3-Orig/20150526 [R.0B]: $URL/v0_5_3_1-RELEASE/20150519 [R.0C]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512 [R.0D]: $URL/v0_5_3-Orig/20150526/All_CPUs.png [R.0E]: $URL/v0_5_3-Orig/20150526/CPU.png [R.0F]: $URL/v0_5_3-Orig/20150526/Memory.png [R.10]: $URL/v0_5_3-Orig/20150526/Network.png [R.11]: $URL/v0_5_3-Orig/20150526/Process_Switches.png [R.12]: $URL/v0_5_3-Orig/20150526/Run_Queue.png [R.13]: $URL/v0_5_3_1-RELEASE/20150519/All_CPUs.png [R.14]: $URL/v0_5_3_1-RELEASE/20150519/CPU.png [R.15]: $URL/v0_5_3_1-RELEASE/20150519/Memory.png [R.16]: $URL/v0_5_3_1-RELEASE/20150519/Network.png [R.17]: $URL/v0_5_3_1-RELEASE/20150519/Process_Switches.png [R.18]: $URL/v0_5_3_1-RELEASE/20150519/Run_Queue.png [R.19]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/All_CPUs.png [R.1A]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/Memory.png [R.1B]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/Network.png EOF -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQIcBAEBCgAGBQJVZmxZAAoJEHIXBai3Hq2vFhEQAIePvmYMe4YjNZZMOmayA97m q6OBVxekCu7x4OR8Lij1oDGvLtugkGqn5c2f8eHGVGBSv7AgAer2ZIQlWfwKSNjL mdDxK0BTGCbk2VGRG9ONInUUOtsoNQvSdHNqlMeQPmtdyNaCa8FuF7oQXoYTOIMO e9X3Isj52U/Ojlv5KSQTSxEkZ9BDV4x5pEGt5gHB6JTzPCghJI5QSGmBe1LGe7/y InTRXdESiifERgY+nqQhxl2hzaYPufsdJ0fyRn+sYPOFI1zu/rMPN+jQ51f7sakk SAFS8gUUlrBNDpL9BimxbvOk3mXgoekr4pv01h1WNHHvGA73xe3X1SKxVPFFOuzP CtY+QKao7ZR8Gg6JfCIChkbKBK5pEQ2Y/qt70Oki0TF+App7YNtjX4Im26RJt/Gx MhbI7Q/5ZmdNe62x8k6oJ3iO0lrGfIiQIKUM7F727e53tPKRpHwbAGm0uWYG7Ul1 5l2BC6DW6K16fB6/XYCdTfjrdO0bQeRbNNZtrJHo0q7hqjvaL4pWaccU7ISJ+miN X6maYnAQCey6MC88xsMVjbH2dxClwyiTUdoHI5QsNXj19UIEKXs/EiXFfbLhJ1mg KqDlGP/LtcbeMBNICLbtQ4pLrsAPf7LyNJ51ZW8noX+XaFhgDbeSvajKHwymEP28 pFg8V5opdZrPfrPERVIk =hOY6 -----END PGP SIGNATURE-----