DESCRIPTION

These modules contain implementations of various protocols forming the Web platform's transport layer.

Following protocols are supported by these modules and underlying platform (partially or fully; see documentation of relevant modules): TCP (for IPv4 and IPv6), UNIX domain sockets, TLS (including SNI, OCSP stapling), HTTP (including Cookies, application/x-www-form-urlencoded, multipart/form-data, Basic authentication, OAuth 1.0, OAuth 2.0 Bearer, CONNECT), SOCKS4, SOCKS5, DNS, MIME types, MIME sniffing.

MODULES

There are following modules that expose public APIs:

Web::Transport::BasicClient

A basic HTTP client.

Web::Transport::WSClient

A WebSocket client.

Web::Transport::PSGIServerConnection

A PSGI interface of HTTP server.

Web::Transport::ProxyServerConnection

A HTTP proxy server.

Web::Transport::PlatformResolver

A name resolver using system's resolver.

Web::Transport::CachedResolver

A DNS cache.

Web::Transport::ENVProxyManager

An environment variable based proxy manager.

Web::MIME::Type::Parser

MIME type parser.

Web::MIME::Sniffing

MIME type sniffing.

Web::Transport::AWS

AWS API signature calculation.

Web::Transport::Base64

A Base64 API.

Web::Transport::DataURL::Parser

A data: URL processor.

Web::Transport::FindPort

Find TCP ports for testing.

The module Web::Transport::ConnectionClient is deprecated.

REQUEST OPTIONS

For the request method of a Web::Transport::BasicClient object, following key/value pairs can be used to specify the parameters of the request to send:

url => $url

Specify the request URL. If specified, the value must be a Web::URL object.

The scheme of the URL must be one of http, https, ftp, ws, or wss.

Any username, password, or fragment of the URL is ignored.

Exactly one of url, path, and path_string options must be specified.

path => [$seg1, $seg2, ...]

Specify the path segments of the request URL. If specified, the value must be an array reference of zero or more character strings. They are encoded in UTF-8 and then concatenated with /.

If a "prefix" is defined by the context, the path segments given by this option is appended to the prefix.

Exactly one of url, path, and path_string options must be specified.

path_string => $string

Specify the path of the request URL. If specified, the value must be a character string. It is encoded in UTF-8. If the first character is not /, a leading / is prepended.

If a "prefix" is defined by the context, the path segments given by this option is appended to the prefix.

Exactly one of url, path, and path_string options must be specified.

method => $string

Specify the HTTP request method. Note that HTTP request methods are case-sensitive. If not specified, GET is used.

headers => {$name => $value, ...}

headers => [[$name => $value], ...]

Specify HTTP headers. The value must be a headers value (see "HEADERS").

cookies => {$string => $string, ...}

Specify cookies, to be included in the Cookie: header. The value must be a hash reference of zero or more key/value pairs, where keys are cookie names and values are corresponding cookie values. Any key whose value is undef is ignored.

Names and values must be strings of zero or more characters. They are encoded in UTF-8 and percent-encoded before included in the header value. If this behavior is inappropriate, use headers option instead.

This option also adds Cache-Control: no-store and Pragma: no-store headers.

params => {$string => $string, ...}

Parameters to be included in the request.

The value must be a hash reference of zero or more key/value pairs, where keys are parameter names and values are corresponding parameter values.

Parameter names and values must be strings of zero or more characters. Parameter values can be either a string or an array reference of zero or more strings, representing parameter values sharing same parameter name. Any parameter whose value is undef is ignored.

They are encoded in the application/x-www-form-urlencoded format (or multipart/form-data, if files option is also specified) in UTF-8. If the method is POST and there is no conflicting options, they are put into the request body. Otherwise, they are appended to the request URL.

files => {$string => $file, ...}

Files to be included in the request.

The value must be a hash reference of zero or more key/value pairs, where keys are parameter names and values are corresponding files.

Parameter names must be strings of zero or more characters.

Files can be either a hash reference or an array reference of zero or more hash references, representing files sharing same parameter name. Any pair whose value is undef is ignored.

A file hash reference must contain a body_ref key, whose value must be a reference to a scalar containing zero or more bytes, i.e. the file content.

A file hash reference can also contain mime_type key, whose value is a MIME type string (Content-Type: header value) and mime_filename key, whose value is a file name string (Content-Disposition: header's filename parameter value; defaulted to a string file.dat if missing). Their defaults, used when the key is omitted or the value is undef, are application/octet-stream and the empty string, respectively.

They are encoded in the multipart/form-data format in UTF-8.

basic_auth => [$userid, $password]

Specifies the credentials of the Basic authentication. The value must be an array reference of two strings, which are used as user ID and password.

bearer => $string

Specifies the credentials of the Bearer authentication.

oauth1 => [$string, $string, $string, $string]

If a non-undef value is specified, the request is to be signed using OAuth 1.0 HMAC-SHA1 signature method.

The value must be an array reference of strings, which are consumer key, consumer secret, access token, and access token secret. They must be specified even though they can be empty strings.

This option also adds Cache-Control: no-store and Pragma: no-store headers.

oauth1_container => 'authorization' | 'query' | 'body'

Where to add OAuth 1.0 request parameters. This option is ignored unless the oauth1 option is also specified.

The value authorization designates the HTTP Authorization: header.

The value query designates the URL query component.

The value body designates the request body.

If this option is not specified, or undef is specified, parameters are added to the HTTP Authorization: header, unless there is another Authorization: header, in which case they are added to the same slot as params.

oauth_verifier => $string

The OAuth 1.0 oauth_verifier request parameter value. If a non-undef value is specified and the oauth1 option is also specified, this value is taken into account as a request parameter. Otherwise this option is ignored.

oauth_callback => $string

The OAuth 1.0 oauth_callback request parameter value. If a non-undef value is specified and the oauth1 option is also specified, this value is taken into account as a request parameter. Otherwise this option is ignored.

aws4 => [$access_key_id, $secret_access_key, $region, $service]

If specified, signature of the request is attached using AWS Signature Version 4 and the Authorization: header.

The value must be an array reference of four values. The zeroth item must be the access key ID. The first item must be the secret access key. The second item must be the AWS region, such as us-east-1. The third item must be the AWS service, such as s3.

aws4_signed_headers => {$name1 => 1, $name2 => 1, ...}

Specifies additionally signed headers. This option is ignored unless aws4 option is also specified.

If specified, the value must be a hash reference, whose keys are header names (ASCII case-insensitive) and values are true values.

ws_protocols => [$name1, $name2, ...]

Specifies WebSocket protocol names. The value must be an array reference of zero or more byte strings. Only applicable to WebSocket requests. To establish a WebSocket connection with WebSocket protocol names as a WebSocket client, this option must be used (rather than directly specifying the Sec-WebSocket-Proto: header value) as the protocol names are directly handled by the client module as part of the protocol handshake.

superreload => $boolean

If true, Cache-Control: no-cache and Pragma: no-store headers are added.

body => $bytes

The request body. If a non-undef value is specified, it must be a string of zero or more bytes.

This option is not allowed when body_stream is specified.

body_stream => $readable_stream

The request body. If a non-undef value is specified, it must be a ReadableStream with type bytes (i.e. a readable byte stream) which is not locked.

If this option has non-undef value, the length option must also be specified. This option is not allowed when body is specified.

length => $byte_length

The byte length of the request body. If a non-undef value is specified, it must be equal to the number of bytes contained by the body_stream readable byte stream.

If this option has non-undef value, the body_stream option must also be specified. This option is not allowed when body is specified.

stream => $boolean

Whether the result of the operation should contain a ReadableStream object (true) or a byte string (false) for the response body. See Web::Transport::BasicClient for more information.

forwarding => $boolean (for proxy)

Whether this client is a component of a proxy, forwarding a request to the upstream. This option is only applicable to the request hash reference of the argument (or return value) to the handle_request callback of Web::Transport::ProxyServerConnection.

Different subsets of these options are also supported by relevant methods of deprecated modules Web::Transport::ConnectionClient and Web::Transport::WSClient.

HEADERS

A headers value is either a headers hash reference or a headers array reference.

A headers hash reference is a reference to a hash whose keys are header names and its values are corresponding header values. If a value is an array reference, that array's items are used as the header values (i.e. multiple headers with same name are generated).

A headers array reference is a reference to an array which contain zero or more array references which contains header name (index 0) and header value (index 1). A canonical headers array reference is a headers array reference whose items contain third item (index 2), which is the ASCII lowercased variant of the header name.

Header names and values must be byte strings. Header names and values cannot contain 0x0D or 0x0A bytes. Header names are case-insensitive. Header names cannot be empty and cannot contain certain bytes (e.g. 0x3A).

PROXY MANAGERS

A proxy manager is used to retrieve proxy configurations used for fetching. It is an object with the get_proxies_for_url method.

The get_proxies_for_url method is expected to return the list of the proxies for the specified URL. It has a required argument, which must be a URL record (Web::URL), for which a proxy configuration is requested.

Additionally, following named argument chould be specified:

signal => $signal

The abort signal (AbortSignal) for the operation. If the abort controller (AbortController) associated with the signal is aborted, the operation is expected to be aborted and the method's promise is rejected with an object whose name is AbortError whenever possible.

The method is expected to return a promise (Promise), which is to be resolved with a proxy configuration list (see "PROXY CONFIGURATIONS"), or to be rejected with an error.

There are following proxy managers in this repository: Web::Transport::ENVProxyManager and Web::Transport::ConstProxyManager. Any other object can be used as well.

PROXY CONFIGURATIONS

A proxy configuration list is ...

XXX

host => ...

XXX

For a proxy configuration returned by a proxy manager, the value must be a Web::Host object. For a proxy configuration used as part of an input to the Web::Transport::ConstProxyManager's constructor, the value may be a string which is a valid host string.

debug => $mode

Specify the debug mode. The default value is the WEBUA_DEBUG environment variable's value. See WEBUA_DEBUG section for available mode values.

This option is only applicable to http and https protocols.

RESOLVERS

XXX

The resolve method is expected to return a promise. XXX If aborted, or the resolver is unable to resolve (note that this is different from the resolver is ready but the result is "not found"), the promise is expected to be rejected with the exception describing the failure.

There are following resolvers in this repository: Web::Transport::PlatformResolver, Web::Transport::CachedResolver.

CERTIFICATE MANAGER

A certificate manager is an object from which TLS certifciates can be retrieved.

It is expected to have following methods:

$cm->prepare ($name => $value, ...)->then (sub { ... })

Run preparation steps for the certificate manager. The concrete steps are implementation specific. The method must return a promise, which is fulfilled or rejected when the certificate manager is ready to return relevant certificates.

It receives zero or more options as name/value pair arguments. The server option has a boolean value, representing server (true) or client (false). If the server value is true but the certificate manager is not ready to return the server certificates (including CA certificates), the method must reject the promise.

{$name => $value, ...} = $cm->to_anyevent_tls_args_sync

Return a hash reference containing zero or more arguments to the new method of the AnyEvent::TLS class. It should only contain certificate-related options.

If the certificate manager's prepare method has been invoked with server option set to true and its promise has been fulfilled, the hash reference must contain the server certificates and all other relevant options. Otherwise, if the certificate manager's prepare method has been invoked and its promise has been fulfilled, the hash reference must contain all relevant options.

{$name => $value, ...} = $cm->to_anyevent_tls_args_for_host_sync ($host)

Same as to_anyevent_tls_args_sync but returns argument for a host-specific certificate.

The method is invoked with an argument of type Web::Host, representing the domain name given in the SNI extension's field received from the client.

The method can return undef if there is no host-specific certificate (such that the default certificate returned by to_anyevent_tls_args_sync should be used instead).

There is following module in this repository: Web::Transport::DefaultCertificateManager.

UNDERLYING PLATFORM INFORMATION OBJECT

An underlying platform information object represents the underlying platform on which the application is running, used to retrive platform-dependent configurations.

It is expected to have following methods:

$string = $info->user_agent

Return the default User-Agent value.

$string = $info->accept_language

Return the appropriate value for the HTTP Accept-Language header.

There is following module in this repository: Web::Transport::PlatformInfo.

ENVIRONMENT VARIABLES

The WEBUA_DEBUG and WEBSERVER_DEBUG environment variables can be used to enable the debug mode of client and server modules, respectively. If a true value is specified, debug output, such as some of network input and output, are printed to the standard error output. If its value is greater than 1, more verbose messages are printed.

DEPENDENCY

These module requires Perl 5.14 or later. They require several core modules such as Digest::SHA and Math::BigInt.

They require AnyEvent and Net::SSLeay. The Net::SSLeay module requires OpenSSL or equivalent (e.g. LibreSSL). For Web compatibility and security, OpenSSL version must be latest enough.

They require following modules (which are submodules of this Git repository):

• modules from perl-promise repository <https://github.com/wakaba/perl-promise>, e.g. Promise and AbortController
• modules from perl-web-url repository <https://github.com/manakai/perl-web-url>, e.g. Web::URL, Web::Host, and Web::Origin
• modules from perl-web-encodings package <https://github.com/manakai/perl-web-encodings>, e.g. Web::Encoding
• modules from perl-web-datetime package <https://github.com/manakai/perl-web-datetime>, e.g. Web::DateTime::Parser

Additionally, modules Web::Transport::TCPStream, Web::Transport::UnixStream, Web::Transport::TLSStream, Web::Transport::SOCKS4Stream, Web::Transport::SOCKS5Stream, Web::Transport::H1CONNECTStream, Web::Transport::BasicClient, Web::Transport::ProxyServerConnection, and Web::Transport::PSGIServerConnection require following modules (which are also part of submodule of this Git repository):

• modules from perl-streams package <https://github.com/manakai/perl-streams>, e.g. ArrayBuffer and ReadableStream

AUTHOR

Wakaba <wakaba@suikawiki.org>.

ACKNOWLEDGEMENTS

Some of modules derived from various earlier effort on these areas. See documentations of modules and comments in source codes for more information.

LICENSE

Copyright 2009-2013 Hatena <https://www.hatena.ne.jp/>.

Copyright 2014-2022 Wakaba <wakaba@suikawiki.org>.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The license of the tls-certs.pem file is:

  This Source Code Form is subject to the terms of the Mozilla Public
  License, v. 2.0. If a copy of the MPL was not distributed with this
  file, You can obtain one at <http://mozilla.org/MPL/2.0/>.