Starman vs. uWSGI: PSGI server performance comparison

This benchmark is about Perl PSGI web application servers but the story starts with Python. I needed to deploy a Python based web application on a server running nginx. I was not fully up to date about the current WSGI application server offerings and therefore searched for some comparisons. I found a comprehensive WSGI server benchmark by Nicholas Piël which was very interesting, although slightly dated already.

I noticed that uWSGI looks like a good modern application server which integrates nicely with nginx. And not only for running Python apps: It can run Perl PSGI/Plack apps as well! That made me very interested.

Earlier PSGI server benchmarks

I searched the interwebs for PSGI/Plack server performance comparisons but did not find any which included uWSGI. It looked like the benchmark in the Starman documentation is the only one available and quoted in many places. It declares Starman as the fastest PSGI server:

– server: Starman (workers=10)
Requests per second: 6849.16 [#/sec] (mean)
— server: Twiggy
Requests per second: 3911.78 [#/sec] (mean)
— server: AnyEvent::HTTPD
Requests per second: 2738.49 [#/sec] (mean)
— server: HTTP::Server::PSGI
Requests per second: 2218.16 [#/sec] (mean)
— server: HTTP::Server::PSGI (workers=10)
Requests per second: 2792.99 [#/sec] (mean)
— server: HTTP::Server::Simple
Requests per second: 1435.50 [#/sec] (mean)
— server: Corona
Requests per second: 2332.00 [#/sec] (mean)
— server: POE
Requests per second: 503.59 [#/sec] (mean)

I needed to have a look at this myself and compare Starman to uWSGI.

Test machine setup

The test environment consisted of:

  • Server: QEMU/KVM based virtual machine with 4 processors and 768 MB of memory, running Ubuntu 12.04 64bit version.
  • Client/load generator: another virtual machine in the same physical box, running Debian sid.

The virtual machines were setup using virtio network drivers and interconnected with a Linux bridge in the host.

Linux kernel TCP tweaks

The test application

The web application used in this benchmark was the simplest possible “hello world” implementation in PSGI:

Starman configuration

The following upstart job was used to start five Starman workers:

The nginx front-end was configured to talk to Starman as follows:

uWSGI configuration

uWSGI was configured to match the Starman configuration with five workers and listen backlog of 1000 (which is the Starman default):

Nginx has a native support for uwsgi protocol which was used over TCP:

Load generator

I used a patched version of weighttp for testing. I do not like ab (ApacheBench) because it does not recognize failed responses (nginx returns HTTP error code 500 when it is unable to speak to the upstream server). I do not like httperf because it does not scale to a high concurrency (it uses select() and thus has a limit of 1024 open connections on Linux).

There is also a problem with weighttp: it includes unsuccessful requests in the “requests per second” statistics. That ruins the results if there are failures because nginx is very fast at serving the error pages. I patched weighttp as follows:

I used keep-alive connections between weighttp and nginx (weighttp option -k) because I did not want to introduce extra overhead from the extra TCP handshakes.

Software versions

All the server side software was as distributed in Ubuntu 12.04 64 bit version at the time of writing this article, except for nginx which was installed from the Launchpad nginx stable PPA. Versions are listed below:

  • uwsgi: 1.0.3+dfsg-1ubuntu0.1
  • starman: 0.2014-1
  • nginx-full: 1.2.1-1ubuntu0

Starman vs. uWSGI

Enough introductions, let’s get to the point. I fired off weighttp from a script with increasing levels of concurrency and got the following results:

uWSGI vs. Starman performance (SVG graph)

uWSGI was able to serve roughly three times as many requests per second as Starman!

Another important thing to consider is the error rate. Here is a graph of the error rate with Starman:

Starman errors (SVG graph)

There is a significant amount of page load errors with Starman when the load gets too high. Not too good. How about uWSGI?

uWSGI errors (SVG graph)

There were zero page load errors with uWSGI. It always served the page that the client was looking for. Much better.

Let’s look at the server average CPU utilization next, Starman first:

Starman CPU utilization (SVG graph)

And then uWSGI:

Starman CPU utilization (SVG graph)

Looks like there is not too much difference in the average CPU consumption.

The following table shows the memory consumption (resident set size) of the server processes after the tests were run:

Server kB per master kB per worker # of workers total MB
Starman 9396 8916 5 52.7
uWSGI 4836 2708 5 17.9

uWSGI uses just a third of the memory compared to Starman.

Conclusions

uWSGI clearly outperforms Starman in every aspect covered in this benchmark.

But that is not all. There are some extra benefits from using uWSGI:

  • It can run Python/WSGI and PHP applications as well. There is no need to have a separate application web server for each different language.
  • uWSGI packaging in Ubuntu/Debian is very nicely done. There is no need to write init scripts or upstart jobs for bringing up servers. You just link a configuration file in /etc/uwsgi/apps-enabled/ and the app starts and stops using the normal system daemon startup mechanisms.
  • uWSGI integrates very nicely with nginx using the high-performance uwsgi binary protocol.
  • uWSGI is programmed to be very robust.
  • uWSGI can do many other things as well. Look at the uWSGI documentation for further details.

I can warmly recommend uWSGI for your PSGI (and other) web application serving needs!

What are your experiences with Starman and uWSGI? Are there any other modern well-performing PSGI or multi-platform application servers which I have overlooked? Feel free to comment below.

  • Pingback: Starman vs. uWSGI: PSGI server performance comparison | bitrate | ちゅどん道中記

  • http://bulknews.typepad.com/ Tatsuhiko Miyagawa


    It doesn’t surprise me at all when C-based implementation such as uWSGI wins on micro benchmarking simple Hello World apps.

    I wonder what’s the actual errors you see when you mention “error rates”. I guess nginx is returning 500 when the communication between nginx and upstream failed. Do you see any specific errors in the nginx log?

    (Note: I’m the author of Perl PSGI spec and Starman web server plus most other PSGI implementations, and a contributor to the uWSGI)

    • Janne Snabb


      Thank you for your comments. I very much appreciate the work you have done with PSGI and Starman, they are both excellent!

      You are absolutely correct about the limited usefulness of micro-benchmarks like this. We can see that in my follow-up test with Catalyst and Dancer “hello world” applications the relative performance difference is already much smaller. With some complex application the difference would be so small that it would be completely insignificant.

      In most real world scenarios the main bottlenecks are somewhere else: in the application itself or in the data store (such as a database or a file system). However in some specific cases small differences like this can matter.

      The error rates in the graphs indicate situations where nginx has responded with the HTTP error 500. At those cases nginx logs two kinds of different error messages:

      2012/07/31 09:00:11 [error] 16894#0: *2752864 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.66, server: test1.example.com, request: “GET / HTTP/1.1″, upstream: “http://127.0.0.1:5001/”, host: “test1.example.com”

      2012/07/31 09:00:23 [error] 16894#0: *2753362 upstream prematurely closed connection while reading response header from upstream, client: 192.168.0.66, server: test1.example.com, request: “GET / HTTP/1.1″, upstream: “http://127.0.0.1:5001/”, host: “test1.example.com”

      I think the first error message happens when Starman socket’s listen backlog (controlled with the “–backlog” option) has become full, which is normal behavior when the application can not serve requests fast enough.

      I am uncertain what exact underlying reason causes the second error message (“upstream prematurely closed connection”). I will try to have a more detailed look at it when I have time, to understand the exact reason for this error condition.

      • http://bulknews.typepad.com/ Tatsuhiko Miyagawa


        > However in some specific cases small differences like this can matter.

        I agree – that’s why I put the performance comparison on Starman doc in the first place, although I guess we have to update it with a caveat like “these difference won’t matter much in your real-world app”. Anyway being fast isn’t a bad thing at all, and micro benchmarking is a relatively easy way to attract people on making a choice, or at least giving a shot ;)

        > I think the first error message happens when Starman socket’s listen backlog (controlled with the “–backlog” option) has become full, which is normal behavior when the application can not serve requests fast enough.

        Makes sense – I agree that this is a pretty normal and sane behavior, and in a real world full deployment scenario where this could be a real issue, you’ll have to have a cluster of workers and configure nginx to try them in return when the backlog is full (with a relatively small backlog number).This is actually described at https://metacpan.org/module/starman#OPTIONS

        > I am uncertain what exact underlying reason causes the second error message (“upstream prematurely closed connection”). I will try to have a more detailed look at it when I have time, to understand the exact reason for this error condition.

        That would be really helpful.
        Thanks for the post!

  • AnyEvent


    try Feersum instead of Starman. You will definitely have a different result.

  • Slavik Goltser


    Another difference is using a binary protocol vs a text protocol. How does uWSGI tested here compare to uWSGI using HTTP behind nginx (to make it actually closer to the Starman set up you have) and to uWSGI using HTTP directly. Also consider running this benchmark against Starman directly (since it accepts HTTP). I do not quite agree that your benchmarking methodology is completely fair simply due to the use of different protocols between nginx and the PSGI backend of choice. I also do not quite agree that your benchmarking set up is exhaustive as you are not testing against the backend directly (maybe nginx adds a noticeable performance penalty).

    • Janne Snabb


      Thank you for your comments.

      This benchmark was done as a part of a project where I needed to find the fastest Perl PSGI application server to go with nginx. uWSGI is the winner for this specific use case. I would have configured Starman to use the faster uwsgi protocol if it supported it but it does not :).

      Thanks for pointing out the limitations of this benchmark (which I was aware of but failed to point out in the article). I am happy to make a more comprehensive and more generic benchmark if someone wishes to sponsor one!

  • Pingback: Starman vs uWSGI: PSGI server comparison « Beyond the Volga