Skip to content

Drop httpx in favour of aiohttp

Mateusz Żebrak requested to merge mzebrak/httpx-error into develop

Closes: #34 (closed)

Requires: !157 (merged) to be merged first

Getting aiohttp to work with ClientSession as a singleton causes an:

aiohttp.client_exceptions.ClientOSError: [Errno None] Can not write request body

from underlying:

E           ConnectionResetError: Cannot write to closing transport

to be thrown periodically (see this workaround) - looks like hived cant keep connection

but the solution with creating a new session for every request seems not to have significant effect on the performance since aiohttp is much faster than httpx.

Additionally after !157 (merged) , a slight decrease in performance should not be a problem for now, and in the future we can take care of the topic more and make it able to handle requests with aiohttp on a singleton session

As we can observe in this benchamrk:

current solution (aiohttp session per request):

Running 10s test @ http://localhost:8000/aiohttp/single
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.31ms    7.72ms  99.45ms   95.58%
    Req/Sec   533.61     84.57   610.00     84.50%
  10628 requests in 10.01s, 1.48MB read
Requests/sec:   1062.20
Transfer/sec:    151.45KB

aiohttp single session:

Running 10s test @ http://localhost:8000/aiohttp/session
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.43ms    1.51ms  26.04ms   94.24%
    Req/Sec     0.94k    83.98     1.08k    75.00%
  18634 requests in 10.01s, 2.59MB read
Requests/sec:   1862.19
Transfer/sec:    265.51KB

and before we got http2 session based solution:

Running 10s test @ http://localhost:8000/httpx/session/http2
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.39ms    1.60ms  39.17ms   96.12%
    Req/Sec   601.91     42.89   656.00     81.50%
  11981 requests in 10.00s, 1.67MB read
Requests/sec:   1197.51
Transfer/sec:    170.74KB

and just to compare to session per every request with httpx:

Running 10s test @ http://localhost:8000/httpx/single/http2
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    70.13ms   17.16ms 141.46ms   81.04%
    Req/Sec    71.26     20.15   101.00     48.99%
  1420 requests in 10.01s, 202.46KB read
Requests/sec:    141.87
Transfer/sec:     20.23KB

so we keep pretty much the same performance as with httpx single session (around 10-20% performance drop) while introducing many advantages for the moment:

  • No more problem with random failing CI tests: (10x checked here and here)
  • Removed dependency of httpx, while we also use aiohttp (why use 2 dependencies for same thing?)
  • Easier usage of Communication - since request methods are now class ones, there is no more need of calling start/close methods
Edited by Mateusz Żebrak

Merge request reports