Developer Reference

Technique

What makes pyrace special is its ability to land multiple HTTP[S] requests in the smallest possible time frame. The technique used to accomplish this is rather straightforward:

1. Create multiple threads.
2. Allow the threads send all but the final few bytes of their data.
3. Force the threads to synchronize using Events.
4. Allow the threads to send their final bytes, completing all requests simultaneously.

This technique relies on the fact that a request isn’t received by a server until the entirety of the request arrives. In practice, this means we only need to synchronize the arrival of the final bytes of requests, not the bulk of their contents. For implementation details, see the BaseConnection documentation.

If the technique is so simple, why does this module have so many classes?

The primary classes are Driver, Thread, and HTTPConnection, but they are separated by many layers of abstraction. Communication between the classes is acheived using the race_args object, but the classes are separated by multiple layers, as shown below. To pass race_args to the primary classes, the intermediate classes must be modified or overwritten.

For example, in order to pass race_args to an HTTPConnection, we must control its parent HTTPConnectionPool. To control an HTTPConnectionPool, we must also control its parent PoolManager. So on and so forth.

Class Relationships

Overriding connection classes at a session level requires modifying many of the classes in between. Here is a diagram illustrating how pyrace (and requests/urllib3) classes interact with each other:

       pyrace.Driver
            /|\
           V V V
       pyrace.Thread
             |
             V
      requests.Session
             |
             V
     pyrace.HTTPAdapter
             |
             V
    urllib3.PoolManager
            /|\
           V V V
pyrace.HTTP[S]ConnectionPool
            /|\
           V V V
  pyrace.HTTP[S]Connection

All of the classes from Session to HTTPConnection required modification in some way, but only HTTPAdapter, HTTPConnectionPool, and HTTPconnection had to be overridden. The other classes, Session and PoolManager, could be modified on a per-instance basis.

Class Summaries

pyrace.Driver

A Driver has one or more Threads.

The Driver is responsible for creating Threads, providing them with a work_queue, and driving them through their respective workloads. The work_queue is a list of Requests and callable functions to be run. Driving is accomplished by using Events to indicate when Thread BaseConnections should synchronize, finish sending data, or begin reading responses.

pyrace.Thread

A Thread has a single Session and a single work_queue. The Thread handles creation of a Session and processing work_queue entries.

A Session is created in order to:

• Persist data (e.g. cookies) across requests within the work_queue.
• Allow a custom HTTPAdapter to be mounted to handle HTTP[S] requests.

Processing work_queue entries includes:

• Sending Requests.
• Executing callable functions within the Thread‘s context.
• Optionally extracting outgoing request cookies to the Session cookie jar.
• Optionally evaluating embedded statements within outgoing requests.

requests.Session

A Session has one or more HTTPAdapters to handle per-service configurations. Sessions persist data across multiple Requests and allow for mounting custom HTTPAdapters.

This Session‘s parent Thread will mount a custom HTTPAdapter to handle HTTP[S] requests.

For more information, see session object documentation.

pyrace.HTTPAdapter

An HTTPAdapter has a single PoolManager. Transport adapters control the interaction with other services (i.e. HTTP[S]).

An instance of this class will be mounted to our parent Thread‘s Session. One instance will handle both HTTP and HTTPS requests.

This is a minimally modified version of requests.HTTPAdapter. The modifications include:

• Allow for the passing arguments (i.e. race_args) to all children classes.
• Replacing the PoolManager‘s HTTPConnectionPool references. This allows for the (eventual) usage of our custom HTTPConnection class.

For implementation details, see HTTPAdapter documentation. For general information, see transport adapter documentation.

urllib3.PoolManager

A PoolManager has one or more ConnectionPools, one for each scheme (i.e. HTTP/HTTPS). When an HTTPConnection is required for a given scheme, the PoolManager routes the request to the correct HTTPConnectionPool.

This PoolManager‘s parent HTTPAdapter will modify the HTTPConnectionPool class references used during creation of new HTTPConnectionPools.

For more information, see urllib3.PoolManager documentation.

pyrace.HTTP[S]ConnectionPool

An HTTPConnectionPool has one or more HTTPConnections. When an HTTPAdapter requests a connection, the HTTPConnection will return one from its pool or create a new one. Because HTTP and HTTPS connections are incompatible due to SSL wrapping, the two are kept separate.

This is a minimally modified version of urllib3.HTTPConnectionPool. The modification changes the ConnectionCls value for HTTP and HTTPS.

pyrace.HTTP[S]Connection

An HTTPConnection has a single Socket. The HTTPConnection handles creating a connection and sending/receiving raw data. This is where the bulk of the Technique is implemented.

This is a modified version of urllib3.connection.HTTPConnection. The modifications include:

• Accepting arguments from the parent Driver (via race_args).
• Withholding the final few bytes of all sent data until an Event is set.
• Optionally manipulating which of the host’s IP address to connect to.