aiohttp¶

HTTP client/server for asyncio (PEP 3156).

Features¶

Supports both HTTP Client and HTTP Server.
Supports both Server WebSockets and Client WebSockets out-of-the-box.
Web-server has Middlewares, Signals and pluggable routing.

Library Installation¶

$ pip install aiohttp

You may want to install optional cchardet library as faster replacement for chardet:

$ pip install cchardet

Getting Started¶

Client example:

import asyncio
import aiohttp

async def fetch_page(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            assert response.status == 200
            return await response.read()

loop = asyncio.get_event_loop()
with aiohttp.ClientSession(loop=loop) as session:
    content = loop.run_until_complete(
        fetch_page(session, 'http://python.org'))
    print(content)

Server example:

from aiohttp import web

async def handle(request):
    name = request.match_info.get('name', "Anonymous")
    text = "Hello, " + name
    return web.Response(body=text.encode('utf-8'))

app = web.Application()
app.router.add_route('GET', '/{name}', handle)

web.run_app(app)

Note

Throughout this documentation, examples utilize the async/await syntax introduced by PEP 492 that is only valid for Python 3.5+.

If you are using Python 3.4, please replace await with yield from and async def with a @coroutine decorator. For example, this:

async def coro(...):
    ret = await f()

should be replaced by:

@asyncio.coroutine
def coro(...):
    ret = yield from f()

Source code¶

The project is hosted on GitHub

Please feel free to file an issue on the bug tracker if you have found a bug or have some suggestion in order to improve the library.

The library uses Travis for Continuous Integration.

Dependencies¶

Python Python 3.4.1+
chardet library
Optional cchardet library as faster replacement for chardet.

Install it explicitly via:
```
$ pip install cchardet
```

Discussion list¶

aio-libs google group: https://groups.google.com/forum/#!forum/aio-libs

Feel free to post your questions and ideas here.

Contributing¶

Please read the instructions for contributors before making a Pull Request.

Authors and License¶

The aiohttp package is written mostly by Nikolay Kim and Andrew Svetlov.

It’s Apache 2 licensed and freely available.

Feel free to improve this package and send a pull request to GitHub.

Contents¶

HTTP Client¶

Make a Request¶

Begin by importing the aiohttp module:

import aiohttp

Now, let’s try to get a web-page. For example let’s get GitHub’s public time-line

with aiohttp.ClientSession() as session:
    async with session.get('https://api.github.com/events') as resp:
        print(resp.status)
        print(await resp.text())

Now, we have a ClientSession called session and a ClientResponse object called resp. We can get all the information we need from the response. The mandatory parameter of ClientSession.get() coroutine is an HTTP url.

In order to make an HTTP POST request use ClientSession.post() coroutine:

session.post('http://httpbin.org/post', data=b'data')

Other HTTP methods are available as well:

session.put('http://httpbin.org/put', data=b'data')
session.delete('http://httpbin.org/delete')
session.head('http://httpbin.org/get')
session.options('http://httpbin.org/get')
session.patch('http://httpbin.org/patch', data=b'data')

Passing Parameters In URLs¶

You often want to send some sort of data in the URL’s query string. If you were constructing the URL by hand, this data would be given as key/value pairs in the URL after a question mark, e.g. httpbin.org/get?key=val. Requests allows you to provide these arguments as a dict, using the params keyword argument. As an example, if you wanted to pass key1=value1 and key2=value2 to httpbin.org/get, you would use the following code:

params = {'key1': 'value1', 'key2': 'value2'}
async with session.get('http://httpbin.org/get',
                       params=params) as resp:
    assert resp.url == 'http://httpbin.org/get?key2=value2&key1=value1'

You can see that the URL has been correctly encoded by printing the URL.

For sending data with multiple values for the same key MultiDict may be used as well.

It is also possible to pass a list of 2 item tuples as parameters, in that case you can specify multiple values for each key:

params = [('key', 'value1'), ('key', 'value2')]
async with session.get('http://httpbin.org/get',
                       params=params) as r:
    assert r.url == 'http://httpbin.org/get?key=value2&key=value1'

You can also pass str content as param, but beware - content is not encoded by library. Note that + is not encoded:

async with session.get('http://httpbin.org/get',
                       params='key=value+1') as r:
        assert r.url = 'http://httpbin.org/get?key=value+1'

Response Content¶

We can read the content of the server’s response. Consider the GitHub time-line again:

async with session.get('https://api.github.com/events') as resp:
    print(await resp.text())

will printout something like:

'[{"created_at":"2015-06-12T14:06:22Z","public":true,"actor":{...

aiohttp will automatically decode the content from the server. You can specify custom encoding for the text() method:

await resp.text(encoding='windows-1251')

Binary Response Content¶

You can also access the response body as bytes, for non-text requests:

print(await resp.read())

b'[{"created_at":"2015-06-12T14:06:22Z","public":true,"actor":{...

The gzip and deflate transfer-encodings are automatically decoded for you.

JSON Response Content¶

There’s also a built-in JSON decoder, in case you’re dealing with JSON data:

async with session.get('https://api.github.com/events') as resp:
    print(await resp.json())

In case that JSON decoding fails, json() will raise an exception. It is possible to specify custom encoding and decoder functions for the json() call.

Streaming Response Content¶

While methods read(), json() and text() are very convenient you should use them carefully. All these methods load the whole response in memory. For example if you want to download several gigabyte sized files, these methods will load all the data in memory. Instead you can use the content attribute. It is an instance of the aiohttp.StreamReader class. The gzip and deflate transfer-encodings are automatically decoded for you:

async with session.get('https://api.github.com/events') as resp:
    await resp.content.read(10)

In general, however, you should use a pattern like this to save what is being streamed to a file:

with open(filename, 'wb') as fd:
    while True:
        chunk = await resp.content.read(chunk_size)
        if not chunk:
            break
        fd.write(chunk)

It is not possible to use read(), json() and text() after explicit reading from content.

Releasing Response¶

Don’t forget to release response after use. This will ensure explicit behavior and proper connection pooling.

The easiest way to correctly response releasing is async with statement:

async with session.get(url) as resp:
    pass

But explicit release() call also may be used:

await resp.release()

But it’s not necessary if you use read(), json() and text() methods. They do release connection internally but better don’t rely on that behavior.

Custom Headers¶

If you need to add HTTP headers to a request, pass them in a dict to the headers parameter.

For example, if you want to specify the content-type for the previous example:

import json
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
headers = {'content-type': 'application/json'}

await session.post(url,
                   data=json.dumps(payload),
                   headers=headers)

Custom Cookies¶

To send your own cookies to the server, you can use the cookies parameter:

url = 'http://httpbin.org/cookies'
cookies = dict(cookies_are='working')

async with session.get(url, cookies=cookies) as resp:
    assert await resp.json() == {"cookies":
                                     {"cookies_are": "working"}}

POST a Multipart-Encoded File¶

To upload Multipart-encoded files:

url = 'http://httpbin.org/post'
files = {'file': open('report.xls', 'rb')}

await session.post(url, data=files)

You can set the filename, content_type explicitly:

url = 'http://httpbin.org/post'
data = FormData()
data.add_field('file',
               open('report.xls', 'rb'),
               filename='report.xls',
               content_type='application/vnd.ms-excel')

await session.post(url, data=data)

If you pass a file object as data parameter, aiohttp will stream it to the server automatically. Check StreamReader for supported format information.

See also

Working with Multipart

Streaming uploads¶

aiohttp supports multiple types of streaming uploads, which allows you to send large files without reading them into memory.

As a simple case, simply provide a file-like object for your body:

with open('massive-body', 'rb') as f:
   await session.post('http://some.url/streamed', data=f)

Or you can provide an coroutine that yields bytes objects:

@asyncio.coroutine
def my_coroutine():
   chunk = yield from read_some_data_from_somewhere()
   if not chunk:
      return
   yield chunk

Warning

yield expression is forbidden inside async def.

Note

It is not a standard coroutine as it yields values so it can not be used like yield from my_coroutine(). aiohttp internally handles such coroutines.

Also it is possible to use a StreamReader object. Lets say we want to upload a file from another request and calculate the file SHA1 hash:

async def feed_stream(resp, stream):
    h = hashlib.sha256()

    while True:
        chunk = await resp.content.readany()
        if not chunk:
            break
        h.update(chunk)
        s.feed_data(chunk)

    return h.hexdigest()

resp = session.get('http://httpbin.org/post')
stream = StreamReader()
loop.create_task(session.post('http://httpbin.org/post', data=stream))

file_hash = await feed_stream(resp, stream)

Because the response content attribute is a StreamReader, you can chain get and post requests together (aka HTTP pipelining):

r = await session.get, 'http://python.org')
await session.post('http://httpbin.org/post',
                   data=r.content)

Uploading pre-compressed data¶

To upload data that is already compressed before passing it to aiohttp, call the request function with compress=False and set the used compression algorithm name (usually deflate or zlib) as the value of the Content-Encoding header:

async def my_coroutine(session, headers, my_data):
    data = zlib.compress(my_data)
    headers = {'Content-Encoding': 'deflate'}
    async with session.post('http://httpbin.org/post',
                            data=data,
                            headers=headers,
                            compress=False):
        pass

Connectors¶

To tweak or change transport layer of requests you can pass a custom connector to ClientSession and family. For example:

conn = aiohttp.TCPConnector()
session = aiohttp.ClientSession(connector=aiohttp.TCPConnector())

See also

Connectors section for more information about different connector types and configuration options.

Limiting connection pool size¶

To limit amount of simultaneously opened connection to the same endpoint ((host, port, is_ssl) triple) you can pass limit parameter to connector:

conn = aiohttp.TCPConnector(limit=30)

The example limits amount of parallel connections to 30.

SSL control for TCP sockets¶

TCPConnector constructor accepts mutually exclusive verify_ssl and ssl_context params.

By default it uses strict checks for HTTPS protocol. Certification checks can be relaxed by passing verify_ssl=False:

conn = aiohttp.TCPConnector(verify_ssl=False)
session = aiohttp.ClientSession(connector=conn)
r = await session.get('https://example.com')

If you need to setup custom ssl parameters (use own certification files for example) you can create a ssl.SSLContext instance and pass it into the connector:

sslcontext = ssl.create_default_context(cafile='/path/to/ca-bundle.crt')
conn = aiohttp.TCPConnector(ssl_context=sslcontext)
session = aiohttp.ClientSession(connector=conn)
r = await session.get('https://example.com')

You may also verify certificates via MD5, SHA1, or SHA256 fingerprint:

# Attempt to connect to https://www.python.org
# with a pin to a bogus certificate:
bad_md5 = b'\xa2\x06G\xad\xaa\xf5\xd8\\J\x99^by;\x06='
conn = aiohttp.TCPConnector(fingerprint=bad_md5)
session = aiohttp.ClientSession(connector=conn)
exc = None
try:
    r = yield from session.get('https://www.python.org')
except FingerprintMismatch as e:
    exc = e
assert exc is not None
assert exc.expected == bad_md5

# www.python.org cert's actual md5
assert exc.got == b'\xca;I\x9cuv\x8es\x138N$?\x15\xca\xcb'

Note that this is the fingerprint of the DER-encoded certificate. If you have the certificate in PEM format, you can convert it to DER with e.g. openssl x509 -in crt.pem -inform PEM -outform DER > crt.der.

Tip: to convert from a hexadecimal digest to a binary byte-string, you can use binascii.unhexlify:

md5_hex = 'ca3b499c75768e7313384e243f15cacb'
from binascii import unhexlify
assert unhexlify(md5_hex) == b'\xca;I\x9cuv\x8es\x138N$?\x15\xca\xcb'

Unix domain sockets¶

If your HTTP server uses UNIX domain sockets you can use UnixConnector:

conn = aiohttp.UnixConnector(path='/path/to/socket')
session = aiohttp.ClientSession(connector=conn)

Proxy support¶

aiohttp supports proxy. You have to use ProxyConnector:

conn = aiohttp.ProxyConnector(proxy="http://some.proxy.com")
session = aiohttp.ClientSession(connector=conn)
async with session.get('http://python.org') as resp:
    print(resp.status)

ProxyConnector also supports proxy authorization:

conn = aiohttp.ProxyConnector(
    proxy="http://some.proxy.com",
    proxy_auth=aiohttp.BasicAuth('user', 'pass'))
session = aiohttp.ClientSession(connector=conn)
async with session.get('http://python.org') as r:
    assert r.status == 200

Authentication credentials can be passed in proxy URL:

conn = aiohttp.ProxyConnector(
    proxy="http://user:pass@some.proxy.com")

Response Status Codes¶

We can check the response status code:

async with session.get('http://httpbin.org/get') as resp:
    assert resp.status == 200

Response Headers¶

We can view the server’s response ClientResponse.headers using a CIMultiDictProxy:

>>> resp.headers
{'ACCESS-CONTROL-ALLOW-ORIGIN': '*',
 'CONTENT-TYPE': 'application/json',
 'DATE': 'Tue, 15 Jul 2014 16:49:51 GMT',
 'SERVER': 'gunicorn/18.0',
 'CONTENT-LENGTH': '331',
 'CONNECTION': 'keep-alive'}

The dictionary is special, though: it’s made just for HTTP headers. According to RFC 7230, HTTP Header names are case-insensitive. It also supports multiple values for the same key as HTTP protocol does.

So, we can access the headers using any capitalization we want:

>>> resp.headers['Content-Type']
'application/json'

>>> resp.headers.get('content-type')
'application/json'

All headers converted from binary data using UTF-8 with surrogateescape option. That works fine on most cases but sometimes unconverted data is needed if a server uses nonstandard encoding. While these headers are malformed from RFC 7230 perspective they are may be retrieved by using ClientResponse.raw_headers property:

>>> resp.raw_headers
((b'SERVER', b'nginx'),
 (b'DATE', b'Sat, 09 Jan 2016 20:28:40 GMT'),
 (b'CONTENT-TYPE', b'text/html; charset=utf-8'),
 (b'CONTENT-LENGTH', b'12150'),
 (b'CONNECTION', b'keep-alive'))

Response Cookies¶

If a response contains some Cookies, you can quickly access them:

url = 'http://example.com/some/cookie/setting/url'
async with session.get(url) as resp:
    print(resp.cookies['example_cookie_name'])

Note

Response cookies contain only values, that were in Set-Cookie headers of the last request in redirection chain. To gather cookies between all redirection requests you can use aiohttp.ClientSession object.

Response History¶

If a request was redirected, it is possible to view previous responses using the history attribute:

>>> resp = await session.get('http://example.com/some/redirect/')
>>> resp
<ClientResponse(http://example.com/some/other/url/) [200]>
>>> resp.history
(<ClientResponse(http://example.com/some/redirect/) [301]>,)

If no redirects occurred or allow_redirects is set to False, history will be an empty sequence.

WebSockets¶

New in version 0.15.

aiohttp works with client websockets out-of-the-box.

You have to use the aiohttp.ClientSession.ws_connect() coroutine for client websocket connection. It accepts a url as a first parameter and returns ClientWebSocketResponse, with that object you can communicate with websocket server using response’s methods:

session = aiohttp.ClientSession()
async with session.ws_connect('http://example.org/websocket') as ws:

    async for msg in ws:
        if msg.tp == aiohttp.MsgType.text:
            if msg.data == 'close cmd':
                await ws.close()
                break
            else:
                ws.send_str(msg.data + '/answer')
        elif msg.tp == aiohttp.MsgType.closed:
            break
        elif msg.tp == aiohttp.MsgType.error:
            break

You must use the only websocket task for both reading (e.g await ws.receive() or async for msg in ws:) and writing but may have multiple writer tasks which can only send data asynchronously (by ws.send_str('data') for example).

Timeouts¶

The example wraps a client call in Timeout context manager, adding timeout for both connecting and response body reading procedures:

with aiohttp.Timeout(0.001):
    async with aiohttp.get('https://github.com') as r:
        await r.text()

blog comments powered by Disqus

HTTP Client Reference¶

Client Session¶

Client session is the recommended interface for making HTTP requests.

Session encapsulates connection pool (connector instance) and supports keepalives by default.

Usage example:

import aiohttp
import asyncio

async def fetch(client):
    async with client.get('http://python.org') as resp:
        assert resp.status == 200
        print(await resp.text())

with aiohttp.ClientSession() as client:
    asyncio.get_event_loop().run_until_complete(fetch(client))

New in version 0.17.

The client session supports context manager protocol for self closing.

class aiohttp.ClientSession(*, connector=None, loop=None, cookies=None, headers=None, skip_auto_headers=None, auth=None, request_class=ClientRequest, response_class=ClientResponse, ws_response_class=ClientWebSocketResponse)[source]¶

The class for creating client sessions and making requests.

Parameters:

connector (aiohttp.connector.BaseConnector) – BaseConnector sub-class instance to support connection pooling.
loop –
event loop used for processing HTTP requests.

If loop is None the constructor borrows it from connector if specified.

asyncio.get_event_loop() is used for getting default event loop otherwise.
cookies (dict) – Cookies to send with the request (optional)
headers –
HTTP Headers to send with the request (optional).

May be either iterable of key-value pairs or Mapping (e.g. dict, CIMultiDict).
skip_auto_headers –
set of headers for which autogeneration should be skipped.

aiohttp autogenerates headers like User-Agent or Content-Type if these headers are not explicitly passed. Using skip_auto_headers parameter allows to skip that generation. Note that Content-Length autogeneration can’t be skipped.

Iterable of str or upstr (optional)
auth (aiohttp.BasicAuth) – an object that represents HTTP Basic Authorization (optional)
request_class – Request class implementation. ClientRequest by default.
response_class – Response class implementation. ClientResponse by default.
ws_response_class –
WebSocketResponse class implementation. ClientWebSocketResponse by default.

New in version 0.16.

Changed in version 0.16: request_class default changed from None to ClientRequest

Changed in version 0.16: response_class default changed from None to ClientResponse

closed¶

True if the session has been closed, False otherwise.

A read-only property.

connector¶

aiohttp.connector.BaseConnector derived instance used for the session.

A read-only property.

cookies¶

The session cookies, http.cookies.SimpleCookie instance.

A read-only property. Overriding session.cookies = new_val is forbidden, but you may modify the object in-place if needed.

coroutine request(method, url, *, params=None, data=None, headers=None, skip_auto_headers=None, auth=None, allow_redirects=True, max_redirects=10, encoding='utf-8', version=HttpVersion(major=1, minor=1), compress=None, chunked=None, expect100=False, read_until_eof=True)[source]¶

Performs an asynchronous HTTP request. Returns a response object.

Parameters:

method (str) – HTTP method
url (str) – Request URL
params –
Mapping, iterable of tuple of key/value pairs or string to be sent as parameters in the query string of the new request (optional)

Allowed values are:
- collections.abc.Mapping e.g. dict, aiohttp.MultiDict or aiohttp.MultiDictProxy
- collections.abc.Iterable e.g. tuple or list
- str with preferably url-encoded content (Warning: content will not be encoded by aiohttp)
data – Dictionary, bytes, or file-like object to send in the body of the request (optional)
headers (dict) – HTTP Headers to send with the request (optional)
skip_auto_headers –
set of headers for which autogeneration should be skipped.

aiohttp autogenerates headers like User-Agent or Content-Type if these headers are not explicitly passed. Using skip_auto_headers parameter allows to skip that generation.

Iterable of str or upstr (optional)
auth (aiohttp.BasicAuth) – an object that represents HTTP Basic Authorization (optional)
allow_redirects (bool) – If set to False, do not follow redirects. True by default (optional).
version (aiohttp.protocol.HttpVersion) – Request HTTP version (optional)
compress (bool) – Set to True if request has to be compressed with deflate encoding. None by default (optional).
chunked (int) – Set to chunk size for chunked transfer encoding. None by default (optional).
expect100 (bool) – Expect 100-continue response from server. False by default (optional).
read_until_eof (bool) – Read response until EOF if response does not have Content-Length header. True by default (optional).

Return ClientResponse:

a client response object.

coroutine get(url, *, allow_redirects=True, **kwargs)[source]¶

Perform a GET request.