Developer Interface

Main Interface

For when you want to get things up and running.

Driver

class pyrace.Driver[source]

Bases: object

An interface to create Threads, provide them with a work, and drive them through their work in a synchronized manner.

logger

logging.Logger – A Logger instance for this Driver.

send_event

threading.Event – An Event indicating that all connections should finish sending a Request.

read_event

threading.Event – An Event indicating that all connections should begin fetching a Response.

_thread_class

pyrace.Thread – Our custom Thread class. Exposed for easy injection of other custom classes.

threads

list – A list of Threads from the most recent process call.

create_threads(work_queue, thread_count, extra_args={})[source]

Creates Threads, initializing them with the provided work_queue and optional extra race_args keyword arguments.

drive_threads(threads, timeout=10, send_delay=0.1)[source]

Drives a list of Threads through their respective work_queues.

Parameters:
  • threads (list) – The list of Threads to drive through their work_queues.
  • timeout (float, optional) – How long to wait, in seconds, before considering a Thread to have timed out. Timed out threads may continue to run, but are no longer driven directly or waited on. Default is 10 seconds.
  • send_delay (float, optional) – How long to wait after syncing before allowing Threads to finish sending data. Increasing this may improve timing precision by allowing sockets to fully flush. Increasing this value too much may result in connection timeouts. Default is 0.10 seconds.
process(work_queue, thread_count=2, timeout=10, send_delay=0.1, **race_args)[source]

Creates and drives a collection of Threads through a list of work items.

Parameters:
  • work_queue (list) – A list of Requests or callables for the threads to process. If work_queue isn’t a list, it will be converted to a single-item list. See Thread documentation for additional details.
  • thread_count (int, optional) – The number of threads to create. Default is 2.
  • timeout (float, optional) – How long to wait, in seconds, before considering a Thread to have timed out. Timed out Threads may continue to run, but are no longer driven directly or waited on. Default is 10 seconds.
  • send_delay (float, optional) – How long to wait after synchronizing before allowing BaseConnections to finish sending data. Increasing this may improve timing precision by allowing sockets to fully flush. Increasing this value too much may result in socket timeouts. Default is 0.10 seconds.
  • **race_args (dict) – Keyword arguments to pass to the Thread and BaseConnection constructors.
Returns:

A list of Thread objects used. Of particular interest are the response and all_responses attributes.
Return type:

list

Lower-Level Classes

For when you want finer controls over how things work.

Thread

class pyrace.Thread(work_queue, race_args)[source]

Bases: threading.Thread

A worker thread that acts as the middleman between Drivers and HTTPAdapters.

Parameters:

  • work_queue (list) – A list of work items for this class to process when run() is executed. Work items must be a requests.Request or a callable function:

    • Requests are sent and their Responses are appended to the all_responses list.
    • Functions will be executed with the Thread instance as the only parameter (i.e. self). Functions are encouraged inspect the Thread state, process any responses, and to modify work_queue to add or change work as necessary.

  • race_args (dict) – Arguments for this class and its child BaseConnections. Entries relevant to this class are:

    thread_num : int

    The thread number associated with this Thread and its children classes.

    shared : dict

    A shared dict between this Thread, its Driver, and its children classes.

    send_kwargs : dict, optional

    Dict of extra arguments to pass to requests.Session.send(). Default is {}.

    fake_send : bool, optional

    Don’t actually send the Request, just create a fake Response from it. The PreparedRequest is stored in requests.Response.request. Useful for debugging. Default is False.

    do_eval : bool, optional

    Indicates if embedded statements in Requests should be evaluated. Requests are evaluated before being requests.Session.prepare()d. See _eval_attrs, _eval_pattern, and _eval_action for details. Default is False.

    save_sent_cookies : bool, optional

    Cookies manually set in Request headers or cookie attributes should be extracted and saved in our Session‘s cookie jar. Extraction occurs after embedded statement evaluation is applied. Default is True.
work_queue

list – Local copy of __init__ parameter.

race_args

dict – Local copy of __init__ parameter.

thread_num

int – Unpacked value from race_args.

shared

dict – Unpacked value from race_args.

fake_send

bool – Unpacked value from race_args.

do_eval

bool – Unpacked value from race_args.

save_sent_cookies

bool – Unpacked value from race_args.

logger

logging.Logger – A Logger instance for this Thread.

_eval_attrs

list – The list of Request attributes to apply _eval_pattern to.

_eval_pattern

str – The regex pattern that searches for evaluable code in the prepared request attributes. Default pattern matches <<<statement_goes_here>>>.

_eval_flags

int – The regex flags to apply to _eval_pattern. Default is re.VERBOSE | re.DOTALL.

_eval_action

callable – The action to take with the re.MatchObject found by _eval_pattern. Default is converting re.MatchObject group 1 to a string, then eval() it.

session

requests.Session – The session-level storage associated with the most recent run() call. The Session is created when run() begins and is closed when it ends.

adapter

pyrace.HTTPAdapter – The adapter responsible for handling our Session‘s HTTP requests. See transport adapter documentation for details.

response

requests.Response – The Response object from the most recently executed Request.

all_responses

list – All Response objects from the most recent run() call, in order.

Raises:InvalidWorkItem – A work item in work_queue wasn’t a Request or callable.
run()[source]

The work body of a threading.Thread. Processes work items from work_queue in sequential order.

HTTP[S]Connection

class pyrace.BaseConnection(*args, **kwargs)[source]

Bases: object

A HTTPConnection wrapper that synchronizes the final bytes of a request.

Parameters:

race_args (dict) – Arguments for this class and its parent Thread. Entries relevant to this class are:

thread_num : int

The thread number associated with our parent Thread.

shared : dict

A shared dict between this Connection and its parent classes.

sync_event : threading.Event

Per-thread state synchronization Event. We set sync_event just before waiting on send_event and/or read_event. This indicates to the parent Driver that our class is ready to proceed. Once all Threads synchronize, the Driver clears each sync_event.

send_event : threading.Event

Global connection/socket sending Event controlled by the parent Driver. Until this Event is set, we withhold the final few bytes of all sent data. When set, it indicates the remaining buffered data may be sent.

read_event : threading.Event

Global connection/socket reading Event controlled by the parent Driver. When set, it indicates that Connections may read server responses.

Note

Due to CPython’s Global Interpreter Lock, only one thread can execute Python bytecode at a time. The lock is only guaranteed to be released during I/O, but that means some threads may begin fetching a Response before others have finished sending their Request. To prevent this, all threads must complete their sends before reading any results.

connect_mode : {‘normal’, ‘same’, ‘random’, ‘different’}, optional

Determines how the hostname’s server IP address is chosen:

  • Normal: always call getaddrinfo() and use what it returns. The results are usually similar but may vary from call to call.
  • Same: globally cache the getaddrinfo() result and use it. This ensures that each Connection across all Threads use the same IP address.
  • Random: randomize which IP address each Connection connects to.
  • Different: each Connection should connect to a different IP address. The getaddrinfo() result is chosen based on the Connection‘s thread_num. If there are fewer IP addresses than threads, the selection wraps around.

Default is same.

Note

The connect_mode argument isn’t foolproof. If the connection to the preferred IP address fails, it will try the remaining IP addresses until a successful connection is made (this is the default behavior of create_connection()). This means that diff mode can never fully guarantee that each Connection actually connects to a different IP address.
race_args

dict – Local copy of __init__ parameter.

thread_num

int – Unpacked value from race_args.

shared

dict – Unpacked value from race_args.

sync_event

Event – Unpacked value from race_args.

send_event

Event – Unpacked value from race_args.

read_event

Event – Unpacked value from race_args.

logger

logging.Logger – A logging.Logger instance for this Connection.

_send_buffer

str – Whenever send() is called, the data is appended to this buffer. If this buffer is over _buffer_size bytes long, all but the final _buffer_size bytes are sent.

_buffer_size

int – Defines the maximum length of _send_buffer before flushing occurs.

getaddrinfo(host, port)[source]

A thread-safe, caching version of getaddrinfo().

getresponse(**kwargs)[source]

Flushes all buffered data and returns the server response after read_event is set.

send(data, flush=False)[source]

Sends all data except for a the last few bytes until flushed.

class pyrace.HTTPConnection(*args, **kwargs)[source]

Bases: pyrace.connection.BaseConnection, urllib3.connection.HTTPConnection

class pyrace.HTTPSConnection(*args, **kwargs)[source]

Bases: pyrace.connection.BaseConnection, urllib3.connection.VerifiedHTTPSConnection

Intermediate Classes

For when you’re a developer.

HTTPAdapter

class pyrace.HTTPAdapter(*args, **kwargs)[source]

Bases: requests.adapters.HTTPAdapter

init_poolmanager(*args, **pool_kwargs)[source]

Creates a PoolManager that this HTTPAdapter gets HTTPConnectionPools from.

Notes

Every requests.HTTPAdapter is responsible for maintaining a urllib3.PoolManager. The PoolManager is responsible for maintaining a ConnectionPool for each scheme. The ConnectionPools are responsible for maintaining Connection objects.

To pass race_args to our HTTPConnection initialization, the following must take place:

  1. Inject race_args into pool_kwargs.
  2. Initialize a PoolManager with **pool_kwargs which will save any unrecognized keyword arguments in connection_pool_kw.
  3. Change the PoolManager‘s ConnectionPool class references to pint to our custom ConnectionPools instead.
  4. When HTTPConnectionPool is initialized with **connection_pool_kw, it saves any unrecognized keyword arguments in conn_kw. (ConnectionPools are created the first time a Connection is needed for a scheme.)
  5. When a urllib3.Connection is created, it is initialized with **conn_kw (which now contains a race_args key).

HTTP[S]ConnectionPool

class pyrace.HTTPConnectionPool(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, **conn_kw)[source]

Bases: urllib3.connectionpool.HTTPConnectionPool

ConnectionCls

alias of HTTPConnection

class pyrace.HTTPSConnectionPool(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, key_file=None, cert_file=None, cert_reqs=None, ca_certs=None, ssl_version=None, assert_hostname=None, assert_fingerprint=None, ca_cert_dir=None, **conn_kw)[source]

Bases: urllib3.connectionpool.HTTPSConnectionPool

ConnectionCls

alias of HTTPSConnection