[BTC-dev] Reference Implementation NMON Performance Tests

Shane Kinney modsix at gmail.com
Thu May 28 02:29:46 UTC 2015

URL: <http://therealbitcoin.org/ml/btc-dev/attachments/20150527/attachment_b94f6d4b2e8df13c0cfe0cfd4efe5a85cec3757b.txt>
Hash: SHA512

 ..::[ The Bitcoin Foundation: NMON Performance Tests ]::..

     [                  Date: 2015.05.26              ]
     [                Author: mod6 [R.01]             ]

0x0] Introduction

   Over the month of May, mod6 conducted a series of system level performance
   tests during a full sync of the bitcoin blockchain for each version.  The
   metrics were captured on a three second interval with NMON [R.02] against
   three different versions of the Reference Implementation:

   0x01: v0.5.3 [Original] [R.03] with SHA256 [R.04],
   0x02: v0.5.3.1-RELEASE [R.05] with SHA256 [R.06],
   0x03: v0.5.3.1-RELEASE [R.05] with SHA256 [R.06] plus asciilifeform's [R.07]
         OrphanageThermonuke Patch submitted [R.08] to the btc-dev [R.09] email

   All of the charts and captured data have been posted to the Bitcoin
   Foundation website in the following web directories:

   0x01: v0.5.3 [Original] [R.0A]
   0x02: v0.5.3.1-RELEASE [R.0B]
   0x03: v0.5.3.1-RELEASE + patch { OrphanageThermonuke } [R.0C]

   mod6 measured only one significant difference between the three variants:
   while 0x01 and 0x02 exhaust system memory and trigger OOM-kills, 0x03 syncs
   the full blockchain without requiring manual intervention.

0x1] Test Environment

   [ Host Environment ]:

     The performance tests were conducted on an `m3.medium' AWS Debian-6
     ``Squeeze'' AMD64 EBS AMI.  This size of instance comes with 3.75Gb of RAM

     Main disk storage [/dev/sda] for the host environment was not utilized;
     all disk IO for the tests were conducted on attached volume and not
     reflected in disk R/W charts [Disk.png].

   [ Local Environment Processes ]:

     During the conducting of these performance tests, the only third party
     processes running were `bitcoind' and `nmon'.

0x2] Notable Test Observations & Test Results

   [ OOM-KILL ]:

     The `bitcoind' process received the ``OOM-KILL'' signal from the kernel two
     times during the full sync of the blockchain for versions:

       [ 0x01: v0.5.3 [R.03] ]:

         In the following charts for the 0x01 performance test, the OOM-KILL
         signal occurred on 20150520 & 20150522.  The times where this occurred
         is correlated in the graph data as follows:

         All_CPUs.png [R.0D] : CPU Utilization drop to 5% on 20150520 and 10%
                               on 20150522.

         CPU.png [R.0E]      : CPU Utilization drop to 5% on 20150520 and 10%
                               on 20150522.

         Memory.png [R.0F]   : Active and Inactive Memory drops to about 200MB
                               on 20150520 and about 1`200MB on 20150522.  It
                               seems that the ``OOM-KILL'' signal is received
                               when Active Memory nears the 3`000MB boundary.

         Network.png [R.10]  : Total Network Utilization drops to about 60KB/s
                               on 20150520 and about 40KB/s on 20150522.

         Process_Switches.png [R.11] : Process Switches spiked to about
                                       2900 Switches/s on 20150520 and about
                                       3300 Switches/s on 20150522.

         Run_Queue.png [R.12] : The Run Queue length dropped to about 1.2 on
                                20150520 and about 1.3 on 20150522. There was
                                a third drop on 20150524, but is not related
                                to a OOM-KILL signal.

       [ 0x02: v0.5.3.1-RELEASE [R.05] ]:

         All_CPUs.png [R.13] : CPU Utilization drop to 0% on 20150515 and 0%
                               on 20150517.

         CPU.png [R.14]      : CPU Utilization drop to 0% on 20150515 and 0%
                               on 20150517.

         Memory.png [R.15]   : Active and Inactive Memory drops to about 0MB
                               on 20150515 and about 0MB on 20150517.  It seems
                               that the ``OOM-KILL'' signal is received when
                               Active Memory nears the 3`000MB boundary.

         Network.png [R.16]  : Total Network Utilization drops to about 0KB/s
                               on 20150515 and about 0KB/s on 20150517.

         Process_Switches.png [R.17] : Process Switches spiked to about
                                       4`000 Switches/s on 20150515 and about
                                       2`000 Switches/s on 20150517; both of
                                       these spikes occurred right before they
                                       went to near 0.

         Run_Queue.png [R.18] : The Run Queue length dropped to about 1.0 on
                                20150515 and about 1.0 on 20150517.

     [ Differences Noted in Test Phase ]:

       A major difference between these two performance test runs: when the
       ``OOM-KILL'' signal occured in test 0x02 there was a greater time
       interval [hours] between the signal and when `bitcoind' was restarted.
       This may account for why CPU/Mem/Network/Process Switches/Run Queue
       reached minimal values, where they did not completely in test 0x01.
       During 0x01, `bitcoind' was restarted much quicker, within about 60

     [ 0x03: No ``OOM-KILL'' with OrphanageThermonuke patch applied ]:

       It is well worth noting here that this performance test of 0x03 never
       never received an ``OOM-KILL'' signal during full blockchain sync.
       The charts for this test also reflect this fact as there are no
       precipitous graph fluctuations as there were with 0x01 and 0x02.

   [ CPU Utilization ]:

     Across all three full blockchain sync tests the data shows that the range
     of CPU Utilization hovers in the range between 30% to 50% [R.0D] [R.13]
     [R.19], with the highest spike near 60% [R.13] on test 0x02.

   [ Memory Utilization ]:

     Across all three full blockchain sync tests the Memory charts [R.0F] [R.15]
     [R.1A] show that the range of utilized Active Memory is approximately
     1`800MB to 3`000MB. However, there is a notable difference with 0x03:
     Active Memory seems to peak around 2`400MB and then drop and stabilize
     around 1`800MB for the duration of the sync process.  As was noted in the
     above section on ``OOM-KILL'' it seems to be that when Active Memory
     approaches 3`000MB there is a ``OOM-KILL'' signal following shortly
     thereafter.  This environment has 3.75Gb of RAM available to it [3`750MB]
     which translates to about an 80% utilization boundary of Active Memory.

   [ Total Network Utilization [Reads/Writes] ]:

     Network utilization during 0x01 and 0x02 the tests range [while running]
     between troughs of around 80KB/s and peaks of about 340KB/s. The 0x03 test,
     has a notably higher utilization range of around 150KB/s troughs with peaks
     about 600KB/s.  It should also be noted that Total Network Utilization
     during test 0x03, the Reference Implementation operated in the upper bounds
     of the range for about the first day and a half of the blockchain sync
     process and then proceeded to taper to lower levels.

0x3] Conclusion

   NMON proved to be a useful tool for gathering System Level performance
   metrics while the Reference Implementation was achieving full blockchain
   sync.  In the future, we may utilize this tool again.  However, we plan to
   do further testing and analysis with tools that are aimed at the process
   level to achieve better granularity.

[ References ]:

  [R.01]: 027A 8D7C 0FB8 A166 4372 0F40 7217 05A8 B71E ADAF
  [R.02]: http://nmon.sourceforge.net/pmwiki.php
  [R.03]  http://thebitcoin.foundation/v0.5.3-0-gd05c03a.tar.gz
  [R.04]: aab1f8ea8c7f131ff69dfa3b9437ba35531018be760132dd6373f41a591f6382
  [R.05]: http://thebitcoin.foundation/bitcoin-v0_5_3_1-RELEASE.tar.gz
  [R.06]: 5c41fe6cf286770a25bf61ab0c35747d0c760f8656754296d2e1d3c4274b5686
  [R.07]: 1721 5D11 8B72 3950 7FAF ED98 B982 28A0 01AB FFC7
  [R.08]: http://thebitcoin.foundation/ml/btc-dev/2015-May/000088.html
  [R.09]: http://thebitcoin.foundation/mailman/listinfo/btc-dev

  [R.0A]: $URL/v0_5_3-Orig/20150526
  [R.0B]: $URL/v0_5_3_1-RELEASE/20150519
  [R.0C]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512
  [R.0D]: $URL/v0_5_3-Orig/20150526/All_CPUs.png
  [R.0E]: $URL/v0_5_3-Orig/20150526/CPU.png
  [R.0F]: $URL/v0_5_3-Orig/20150526/Memory.png
  [R.10]: $URL/v0_5_3-Orig/20150526/Network.png
  [R.11]: $URL/v0_5_3-Orig/20150526/Process_Switches.png
  [R.12]: $URL/v0_5_3-Orig/20150526/Run_Queue.png
  [R.13]: $URL/v0_5_3_1-RELEASE/20150519/All_CPUs.png
  [R.14]: $URL/v0_5_3_1-RELEASE/20150519/CPU.png
  [R.15]: $URL/v0_5_3_1-RELEASE/20150519/Memory.png
  [R.16]: $URL/v0_5_3_1-RELEASE/20150519/Network.png
  [R.17]: $URL/v0_5_3_1-RELEASE/20150519/Process_Switches.png
  [R.18]: $URL/v0_5_3_1-RELEASE/20150519/Run_Queue.png
  [R.19]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/All_CPUs.png
  [R.1A]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/Memory.png
  [R.1B]: $URL/v0_5_3_1-RELEASE-Plus-OrphanageThermonuke/20150512/Network.png

Version: GnuPG v1.4.13 (FreeBSD)


More information about the BTC-dev mailing list