EventMachine(Thin) vs Twisted vs Tornado
Notice: The results here is not accurate. But, you can get a rough idea how everything works. Refer to this: http://www.brianp.net/2008/10/03/changing-the-length-of-the-time_wait-state-on-mac-os-x/
The recent release of the realtime Tornado frameworb from Facebook(friendfeed) is really cool. But, the real question is: how is the performance of Tornado?
After reading the post from apparatus camparing Twisted with Tornado, I decided to do a performance test on Twisted equivalent in Ruby - EventMachine. I used Thin as the server and Rack middleware to test the speed of EventMachine. And, since there was someone in HackerNews curious about how MochiWeb performance on the test. So, I decided to include MochiWeb in my test.
I ran into problems of 'apr_socket_recv: Connection reset by peer (54)' when I tried to run the test on twisted(cocurrency > 75) and mochiweb(cocurrency > 50).
Results:
twisted:
test1:
Concurrency Level: 1
Time taken for tests: 5.815 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 1719.73 [#/sec] (mean)
Time per request: 0.581 [ms] (mean)
Time per request: 0.581 [ms] (mean, across all concurrent requests)
Transfer rate: 233.44 [Kbytes/sec] received
test2:
Concurrency Level: 1
Time taken for tests: 5.811 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 1720.74 [#/sec] (mean)
Time per request: 0.581 [ms] (mean)
Time per request: 0.581 [ms] (mean, across all concurrent requests)
Transfer rate: 233.58 [Kbytes/sec] received
test3:
Concurrency Level: 1
Time taken for tests: 5.823 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 1717.46 [#/sec] (mean)
Time per request: 0.582 [ms] (mean)
Time per request: 0.582 [ms] (mean, across all concurrent requests)
Transfer rate: 233.13 [Kbytes/sec] received
tornado:
test1:
Concurrency Level: 1
Time taken for tests: 4.122 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680000 bytes
HTML transferred: 120000 bytes
Requests per second: 2425.79 [#/sec] (mean)
Time per request: 0.412 [ms] (mean)
Time per request: 0.412 [ms] (mean, across all concurrent requests)
Transfer rate: 397.98 [Kbytes/sec] received
test2:
Concurrency Level: 1
Time taken for tests: 4.174 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680000 bytes
HTML transferred: 120000 bytes
Requests per second: 2395.56 [#/sec] (mean)
Time per request: 0.417 [ms] (mean)
Time per request: 0.417 [ms] (mean, across all concurrent requests)
Transfer rate: 393.02 [Kbytes/sec] received
test3:
Concurrency Level: 1
Time taken for tests: 4.163 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680000 bytes
HTML transferred: 120000 bytes
Requests per second: 2402.26 [#/sec] (mean)
Time per request: 0.416 [ms] (mean)
Time per request: 0.416 [ms] (mean, across all concurrent requests)
Transfer rate: 394.12 [Kbytes/sec] received
thin+rack(eventmachine):
test1:
Concurrency Level: 1
Time taken for tests: 2.954 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 3385.44 [#/sec] (mean)
Time per request: 0.295 [ms] (mean)
Time per request: 0.295 [ms] (mean, across all concurrent requests)
Transfer rate: 466.16 [Kbytes/sec] received
test2:
Concurrency Level: 1
Time taken for tests: 2.915 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 3430.92 [#/sec] (mean)
Time per request: 0.291 [ms] (mean)
Time per request: 0.291 [ms] (mean, across all concurrent requests)
Transfer rate: 472.42 [Kbytes/sec] received
test3:
Concurrency Level: 1
Time taken for tests: 2.929 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 3414.41 [#/sec] (mean)
Time per request: 0.293 [ms] (mean)
Time per request: 0.293 [ms] (mean, across all concurrent requests)
Transfer rate: 470.15 [Kbytes/sec] received
mochiweb:
test1:
Concurrency Level: 1
Time taken for tests: 5.176 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 1932.01 [#/sec] (mean)
Time per request: 0.518 [ms] (mean)
Time per request: 0.518 [ms] (mean, across all concurrent requests)
Transfer rate: 318.86 [Kbytes/sec] received
test2:
Concurrency Level: 1
Time taken for tests: 14.485 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 690.36 [#/sec] (mean)
Time per request: 1.449 [ms] (mean)
Time per request: 1.449 [ms] (mean, across all concurrent requests)
Transfer rate: 113.94 [Kbytes/sec] received
test3:
Concurrency Level: 1
Time taken for tests: 6.173 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 1620.07 [#/sec] (mean)
Time per request: 0.617 [ms] (mean)
Time per request: 0.617 [ms] (mean, across all concurrent requests)
Transfer rate: 267.38 [Kbytes/sec] receivedtwisted:
test1:
Concurrency Level: 25
Time taken for tests: 4.147 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2411.52 [#/sec] (mean)
Time per request: 10.367 [ms] (mean)
Time per request: 0.415 [ms] (mean, across all concurrent requests)
Transfer rate: 327.34 [Kbytes/sec] received
test2:
Concurrency Level: 25
Time taken for tests: 15.073 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 663.42 [#/sec] (mean)
Time per request: 37.684 [ms] (mean)
Time per request: 1.507 [ms] (mean, across all concurrent requests)
Transfer rate: 90.05 [Kbytes/sec] received
test3:
Concurrency Level: 25
Time taken for tests: 4.104 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2436.59 [#/sec] (mean)
Time per request: 10.260 [ms] (mean)
Time per request: 0.410 [ms] (mean, across all concurrent requests)
Transfer rate: 330.75 [Kbytes/sec] received
tornado:
test1:
Concurrency Level: 25
Time taken for tests: 2.362 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680336 bytes
HTML transferred: 120024 bytes
Requests per second: 4234.33 [#/sec] (mean)
Time per request: 5.904 [ms] (mean)
Time per request: 0.236 [ms] (mean, across all concurrent requests)
Transfer rate: 694.83 [Kbytes/sec] received
test2:
Concurrency Level: 25
Time taken for tests: 3.312 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680672 bytes
HTML transferred: 120048 bytes
Requests per second: 3018.97 [#/sec] (mean)
Time per request: 8.281 [ms] (mean)
Time per request: 0.331 [ms] (mean, across all concurrent requests)
Transfer rate: 495.50 [Kbytes/sec] received
test3:
Concurrency Level: 25
Time taken for tests: 2.364 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680336 bytes
HTML transferred: 120024 bytes
Requests per second: 4230.54 [#/sec] (mean)
Time per request: 5.909 [ms] (mean)
Time per request: 0.236 [ms] (mean, across all concurrent requests)
Transfer rate: 694.21 [Kbytes/sec] received
thin+rack(eventmachine):
test1:
Concurrency Level: 25
Time taken for tests: 1.912 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 5229.33 [#/sec] (mean)
Time per request: 4.781 [ms] (mean)
Time per request: 0.191 [ms] (mean, across all concurrent requests)
Transfer rate: 720.05 [Kbytes/sec] received
test2:
Concurrency Level: 25
Time taken for tests: 4.709 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410282 bytes
HTML transferred: 120024 bytes
Requests per second: 2123.55 [#/sec] (mean)
Time per request: 11.773 [ms] (mean)
Time per request: 0.471 [ms] (mean, across all concurrent requests)
Transfer rate: 292.46 [Kbytes/sec] received
test3:
Concurrency Level: 25
Time taken for tests: 12.924 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410423 bytes
HTML transferred: 120036 bytes
Requests per second: 773.77 [#/sec] (mean)
Time per request: 32.309 [ms] (mean)
Time per request: 1.292 [ms] (mean, across all concurrent requests)
Transfer rate: 106.58 [Kbytes/sec] received
mochiweb:
test1:
Concurrency Level: 25
Time taken for tests: 2.975 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 3360.99 [#/sec] (mean)
Time per request: 7.438 [ms] (mean)
Time per request: 0.298 [ms] (mean, across all concurrent requests)
Transfer rate: 554.70 [Kbytes/sec] received
test2:
Concurrency Level: 25
Time taken for tests: 21.965 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 455.28 [#/sec] (mean)
Time per request: 54.912 [ms] (mean)
Time per request: 2.196 [ms] (mean, across all concurrent requests)
Transfer rate: 75.14 [Kbytes/sec] received
test3:
Concurrency Level: 25
Time taken for tests: 7.974 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1690000 bytes
HTML transferred: 120000 bytes
Requests per second: 1254.08 [#/sec] (mean)
Time per request: 19.935 [ms] (mean)
Time per request: 0.797 [ms] (mean, across all concurrent requests)
Transfer rate: 206.97 [Kbytes/sec] receivedtwisted:
test1:
Concurrency Level: 50
Time taken for tests: 4.095 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2441.77 [#/sec] (mean)
Time per request: 20.477 [ms] (mean)
Time per request: 0.410 [ms] (mean, across all concurrent requests)
Transfer rate: 331.45 [Kbytes/sec] received
test2:
Concurrency Level: 50
Time taken for tests: 15.061 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 663.97 [#/sec] (mean)
Time per request: 75.305 [ms] (mean)
Time per request: 1.506 [ms] (mean, across all concurrent requests)
Transfer rate: 90.13 [Kbytes/sec] received
test3:
Concurrency Level: 50
Time taken for tests: 5.101 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 1960.27 [#/sec] (mean)
Time per request: 25.507 [ms] (mean)
Time per request: 0.510 [ms] (mean, across all concurrent requests)
Transfer rate: 266.09 [Kbytes/sec] received
tornado:
test1:
Concurrency Level: 50
Time taken for tests: 2.333 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680672 bytes
HTML transferred: 120048 bytes
Requests per second: 4286.57 [#/sec] (mean)
Time per request: 11.664 [ms] (mean)
Time per request: 0.233 [ms] (mean, across all concurrent requests)
Transfer rate: 703.55 [Kbytes/sec] received
test2:
Concurrency Level: 50
Time taken for tests: 21.458 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680504 bytes
HTML transferred: 120036 bytes
Requests per second: 466.03 [#/sec] (mean)
Time per request: 107.290 [ms] (mean)
Time per request: 2.146 [ms] (mean, across all concurrent requests)
Transfer rate: 76.48 [Kbytes/sec] received
test3:
Concurrency Level: 50
Time taken for tests: 2.338 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1681344 bytes
HTML transferred: 120096 bytes
Requests per second: 4277.70 [#/sec] (mean)
Time per request: 11.689 [ms] (mean)
Time per request: 0.234 [ms] (mean, across all concurrent requests)
Transfer rate: 702.37 [Kbytes/sec] received
thin+rack(eventmachine):
test1:
Concurrency Level: 50
Time taken for tests: 1.906 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410705 bytes
HTML transferred: 120060 bytes
Requests per second: 5246.96 [#/sec] (mean)
Time per request: 9.529 [ms] (mean)
Time per request: 0.191 [ms] (mean, across all concurrent requests)
Transfer rate: 722.84 [Kbytes/sec] received
test2:
Concurrency Level: 50
Time taken for tests: 12.911 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 774.54 [#/sec] (mean)
Time per request: 64.554 [ms] (mean)
Time per request: 1.291 [ms] (mean, across all concurrent requests)
Transfer rate: 106.65 [Kbytes/sec] received
test3:
Concurrency Level: 50
Time taken for tests: 1.907 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410564 bytes
HTML transferred: 120048 bytes
Requests per second: 5243.21 [#/sec] (mean)
Time per request: 9.536 [ms] (mean)
Time per request: 0.191 [ms] (mean, across all concurrent requests)
Transfer rate: 722.25 [Kbytes/sec] receivedtwisted:
test1:
Concurrency Level: 75
Time taken for tests: 4.106 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2435.40 [#/sec] (mean)
Time per request: 30.796 [ms] (mean)
Time per request: 0.411 [ms] (mean, across all concurrent requests)
Transfer rate: 330.59 [Kbytes/sec] received
test2:
Concurrency Level: 75
Time taken for tests: 6.309 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390278 bytes
HTML transferred: 120024 bytes
Requests per second: 1585.12 [#/sec] (mean)
Time per request: 47.315 [ms] (mean)
Time per request: 0.631 [ms] (mean, across all concurrent requests)
Transfer rate: 215.21 [Kbytes/sec] received
test3:
Concurrency Level: 75
Time taken for tests: 23.031 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 434.20 [#/sec] (mean)
Time per request: 172.730 [ms] (mean)
Time per request: 2.303 [ms] (mean, across all concurrent requests)
Transfer rate: 58.94 [Kbytes/sec] received
tornado:
test1:
Concurrency Level: 75
Time taken for tests: 9.284 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680168 bytes
HTML transferred: 120012 bytes
Requests per second: 1077.14 [#/sec] (mean)
Time per request: 69.629 [ms] (mean)
Time per request: 0.928 [ms] (mean, across all concurrent requests)
Transfer rate: 176.74 [Kbytes/sec] received
test2:
Concurrency Level: 75
Time taken for tests: 9.286 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680840 bytes
HTML transferred: 120060 bytes
Requests per second: 1076.94 [#/sec] (mean)
Time per request: 69.642 [ms] (mean)
Time per request: 0.929 [ms] (mean, across all concurrent requests)
Transfer rate: 176.77 [Kbytes/sec] received
test3:
Concurrency Level: 75
Time taken for tests: 9.328 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680168 bytes
HTML transferred: 120012 bytes
Requests per second: 1072.03 [#/sec] (mean)
Time per request: 69.961 [ms] (mean)
Time per request: 0.933 [ms] (mean, across all concurrent requests)
Transfer rate: 175.90 [Kbytes/sec] received
thin+rack(eventmachine):
test1:
Concurrency Level: 75
Time taken for tests: 1.911 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 5233.07 [#/sec] (mean)
Time per request: 14.332 [ms] (mean)
Time per request: 0.191 [ms] (mean, across all concurrent requests)
Transfer rate: 720.57 [Kbytes/sec] received
test2:
Concurrency Level: 75
Time taken for tests: 20.880 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410423 bytes
HTML transferred: 120036 bytes
Requests per second: 478.92 [#/sec] (mean)
Time per request: 156.601 [ms] (mean)
Time per request: 2.088 [ms] (mean, across all concurrent requests)
Transfer rate: 65.97 [Kbytes/sec] received
test3:
Concurrency Level: 75
Time taken for tests: 1.926 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410564 bytes
HTML transferred: 120048 bytes
Requests per second: 5191.80 [#/sec] (mean)
Time per request: 14.446 [ms] (mean)
Time per request: 0.193 [ms] (mean, across all concurrent requests)
Transfer rate: 715.17 [Kbytes/sec] receivedtwisted:
test1:
Concurrency Level: 100
Time taken for tests: 4.151 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2409.15 [#/sec] (mean)
Time per request: 41.508 [ms] (mean)
Time per request: 0.415 [ms] (mean, across all concurrent requests)
Transfer rate: 327.02 [Kbytes/sec] received
test2:
Concurrency Level: 100
Time taken for tests: 4.151 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2409.24 [#/sec] (mean)
Time per request: 41.507 [ms] (mean)
Time per request: 0.415 [ms] (mean, across all concurrent requests)
Transfer rate: 327.03 [Kbytes/sec] received
test3:
Concurrency Level: 100
Time taken for tests: 4.176 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1390000 bytes
HTML transferred: 120000 bytes
Requests per second: 2394.58 [#/sec] (mean)
Time per request: 41.761 [ms] (mean)
Time per request: 0.418 [ms] (mean, across all concurrent requests)
Transfer rate: 325.05 [Kbytes/sec] receivedtornado:
test1:
Concurrency Level: 100
Time taken for tests: 2.412 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1681344 bytes
HTML transferred: 120096 bytes
Requests per second: 4146.64 [#/sec] (mean)
Time per request: 24.116 [ms] (mean)
Time per request: 0.241 [ms] (mean, across all concurrent requests)
Transfer rate: 680.85 [Kbytes/sec] received
test2:
Concurrency Level: 100
Time taken for tests: 9.366 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1682856 bytes
HTML transferred: 120204 bytes
Requests per second: 1067.73 [#/sec] (mean)
Time per request: 93.657 [ms] (mean)
Time per request: 0.937 [ms] (mean, across all concurrent requests)
Transfer rate: 175.47 [Kbytes/sec] received
test3:
Concurrency Level: 100
Time taken for tests: 2.414 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1681176 bytes
HTML transferred: 120084 bytes
Requests per second: 4142.38 [#/sec] (mean)
Time per request: 24.141 [ms] (mean)
Time per request: 0.241 [ms] (mean, across all concurrent requests)
Transfer rate: 680.08 [Kbytes/sec] received
thin+rack(eventmachine):
test1:
Concurrency Level: 100
Time taken for tests: 1.967 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 5083.27 [#/sec] (mean)
Time per request: 19.672 [ms] (mean)
Time per request: 0.197 [ms] (mean, across all concurrent requests)
Transfer rate: 699.94 [Kbytes/sec] received
test2:
Concurrency Level: 100
Time taken for tests: 20.919 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410987 bytes
HTML transferred: 120084 bytes
Requests per second: 478.03 [#/sec] (mean)
Time per request: 209.191 [ms] (mean)
Time per request: 2.092 [ms] (mean, across all concurrent requests)
Transfer rate: 65.87 [Kbytes/sec] received
test3:
Concurrency Level: 100
Time taken for tests: 1.945 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1410000 bytes
HTML transferred: 120000 bytes
Requests per second: 5142.11 [#/sec] (mean)
Time per request: 19.447 [ms] (mean)
Time per request: 0.194 [ms] (mean, across all concurrent requests)
Transfer rate: 708.04 [Kbytes/sec] received
Code:
Twisted:
Tornado:
Thin+Rack:
MochiWeb:
Feel free to comment on the mistakes I made.
3 comments
Just decrease TIME_WAIT timeout


You probably want to look for a better benchmarking tool than ab. The error it spat out at you in a couple of your tests is just one of its shortcomings. I'd also take a careful look at the systems you're using in these benchmarks. You seem to have an *extremely* large variability between your tests - much larger than the difference between any of the servers you tested. This makes the results highly suspicious at best, and useless at worst.