Table Of Contents

Previous topic

Object

This Page

Psst... hey. Did you know you can read about Swift at docs.openstack.org also? Plus you can get to past versions atSwift 1.3 docs and Swift 1.2 docs.

Misc

Exceptions

exception swift.common.exceptions.AuditException

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.AuthException

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ChunkReadTimeout(seconds=None, exception=None)

Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ChunkWriteTimeout(seconds=None, exception=None)

Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ConnectionTimeout(seconds=None, exception=None)

Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.DiskFileError

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DiskFileNotExist

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DriveNotMounted

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DuplicateDeviceError

Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.EmptyRingError

Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.ListingIterError

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ListingIterNotAuthorized(aresp)

Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.ListingIterNotFound

Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.LockTimeout(seconds=None, msg=None)

Bases: swift.common.exceptions.MessageTimeout

exception swift.common.exceptions.MessageTimeout(seconds=None, msg=None)

Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.RingBuilderError

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.RingValidationError

Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.SwiftConfigurationError

Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.SwiftException

Bases: exceptions.Exception

Constraints

swift.common.constraints.ACCOUNT_LISTING_LIMIT = 10000

Max container list length of a get request for an account

swift.common.constraints.CONTAINER_LISTING_LIMIT = 10000

Max object list length of a get request for a container

swift.common.constraints.FORMAT2CONTENT_TYPE = {'xml': 'application/xml', 'plain': 'text/plain', 'json': 'application/json'}

Query string format= values to their corresponding content-type values

swift.common.constraints.MAX_FILE_SIZE = 5368709122L

Max file size allowed for objects

swift.common.constraints.MAX_META_COUNT = 90

Max number of metadata items

swift.common.constraints.MAX_META_NAME_LENGTH = 128

Max length of the name of a key for metadata

swift.common.constraints.MAX_META_OVERALL_SIZE = 4096

Max overall size of metadata

swift.common.constraints.MAX_META_VALUE_LENGTH = 256

Max length of the value of a key for metadata

swift.common.constraints.MAX_OBJECT_NAME_LENGTH = 1024

Max object name length

swift.common.constraints.check_float(string)

Helper function for checking if a string can be converted to a float.

Parameters:string – string to be verified as a float
Returns:True if the string can be converted to a float, False otherwise
swift.common.constraints.check_metadata(req, target_type)

Check metadata sent in the request headers.

Parameters:
  • req – request object
  • target_type – str: one of: object, container, or account: indicates which type the target storage for the metadata is
Raises HTTPBadRequest:
 

bad metadata

swift.common.constraints.check_mount(root, drive)

Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidently filling up the root partition.

Parameters:
  • root – base path where the devices are mounted
  • drive – drive name to be checked
Returns:

True if it is a valid mounted device, False otherwise

swift.common.constraints.check_object_creation(req, object_name)

Check to ensure that everything is alright about an object to be created.

Parameters:
  • req – HTTP request object
  • object_name – name of object to be created
Raises:
  • HTTPRequestEntityTooLarge – the object is too large
  • HTTPLengthRequered – missing content-length header and not a chunked request
  • HTTPBadRequest – missing or bad content-type header, or bad metadata
swift.common.constraints.check_utf8(string)

Validate if a string is valid UTF-8 str or unicode

Parameters:string – string to be validated
Returns:True if the string is valid utf-8 str or unicode, False otherwise

Utils

Miscellaneous utility functions for use with Swift.

class swift.common.utils.ContextPool(size=1000)

Bases: eventlet.greenpool.GreenPool

GreenPool subclassed to kill its coros when it gets gc’ed

class swift.common.utils.LogAdapter(logger, server)

Bases: logging.LoggerAdapter, object

A Logger like object which performs some reformatting on calls to exception(). Can be used to store a threadlocal transaction id and client ip.

notice(msg, *args, **kwargs)

Convenience function for syslog priority LOG_NOTICE. The python logging lvl is set to 25, just above info. SysLogHandler is monkey patched to map this log lvl to the LOG_NOTICE syslog priority.

process(msg, kwargs)

Add extra info to message

set_statsd_prefix(prefix)

The StatsD client prefix defaults to the “name” of the logger. This method may override that default with a specific value. Currently used in the proxy-server to differentiate the Account, Container, and Object controllers.

statsd_delegate(statsd_func_name)

Factory which creates methods which delegate to methods on self.logger.statsd_client (an instance of StatsdClient). The created methods conditionally delegate to a method whose name is given in ‘statsd_func_name’. The created delegate methods are a no-op when StatsD logging is not configured. The created delegate methods also handle the defaulting of sample_rate (to either the default specified in the config with ‘log_statsd_default_sample_rate’ or the value passed into delegate function).

Parameters:statsd_func_name – the name of a method on StatsdClient.
class swift.common.utils.ModifiedParseResult

Bases: urlparse.ParseResult

Parse results class for urlparse.

class swift.common.utils.NullLogger

A no-op logger for eventlet wsgi.

class swift.common.utils.SwiftLogFormatter(fmt=None, datefmt=None)

Bases: logging.Formatter

Custom logging.Formatter will append txn_id to a log message if the record has one and the message does not.

swift.common.utils.audit_location_generator(devices, datadir, mount_check=True, logger=None)

Given a devices path and a data directory, yield (path, device, partition) for all files in that directory

Parameters:
  • devices – parent directory of the devices to be audited
  • datadir – a directory located under self.devices. This should be one of the DATADIR constants defined in the account, container, and object servers.
  • mount_check – Flag to check if a mount check should be performed on devices
  • logger – a logger object
swift.common.utils.cache_from_env(env)

Get memcache connection pool from the environment (which had been previously set by the memcache middleware

Parameters:env – wsgi environment dict
Returns:swift.common.memcached.MemcacheRing from environment
swift.common.utils.capture_stdio(logger, **kwargs)

Log unhandled exceptions, close stdio, capture stdout and stderr.

param logger: Logger object to use

swift.common.utils.compute_eta(start_time, current_value, final_value)

Compute an ETA. Now only if we could also have a progress bar...

Parameters:
  • start_time – Unix timestamp when the operation began
  • current_value – Current value
  • final_value – Final value
Returns:

ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.drop_buffer_cache(fd, offset, length)

Drop ‘buffer’ cache for the given range of the given file.

Parameters:
  • fd – file descriptor
  • offset – start offset
  • length – length
swift.common.utils.drop_privileges(user)

Sets the userid/groupid of the current process, get session leader, etc.

Parameters:user – User name to change privileges to
swift.common.utils.dump_recon_cache(cache_dict, cache_file, logger, lock_timeout=2)

Update recon cache values

Parameters:
  • cache_dict – Dictionary of cache key/value pairs to write out
  • cache_file – cache file to update
  • logger – the logger to use to log an encountered error
  • lock_timeout – timeout (in seconds)
swift.common.utils.fallocate(fd, size)

Pre-allocate disk space for a file.

Parameters:
  • fd – file descriptor
  • size – size to allocate (in bytes)
swift.common.utils.get_logger(conf, name=None, log_to_console=False, log_route=None, fmt='%(server)s %(message)s')

Get the current system logger using config settings.

Log config and defaults:

log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
log_udp_host = (disabled)
log_udp_port = logging.handlers.SYSLOG_UDP_PORT
log_address = /dev/log
log_statsd_host = (disabled)
log_statsd_port = 8125
log_statsd_default_sample_rate = 1
log_statsd_metric_prefix = (empty-string)
Parameters:
  • conf – Configuration dict to read settings from
  • name – Name of the logger
  • log_to_console – Add handler which writes to console on stderr
  • log_route – Route for the logging, not emitted to the log, just used to separate logging configurations
  • fmt – Override log format
swift.common.utils.get_param(req, name, default=None)

Get parameters from an HTTP request ensuring proper handling UTF-8 encoding.

Parameters:
  • req – Webob request object
  • name – parameter name
  • default – result to return if the parameter is not found
Returns:

HTTP request parameter value

swift.common.utils.get_time_units(time_amount)

Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)

Parameters:time_amount – length of time in seconds
Returns:A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)
swift.common.utils.get_valid_utf8_str(str_or_unicode)

Get valid parts of utf-8 str from str, unicode and even invalid utf-8 str

Parameters:str_or_unicode – a string or an unicode which can be invalid utf-8
swift.common.utils.hash_path(account, container=None, object=None, raw_digest=False)

Get the connonical hash for an account/container/object

Parameters:
  • account – Account
  • container – Container
  • object – Object
  • raw_digest – If True, return the raw version rather than a hex digest
Returns:

hash string

swift.common.utils.human_readable(value)

Returns the number in a human readable format; for example 1048576 = “1Mi”.

swift.common.utils.item_from_env(env, item_name)

Get a value from the wsgi environment

Parameters:
  • env – wsgi environment dict
  • item_name – name of item to get
Returns:

the value from the environment

swift.common.utils.iter_devices_partitions(devices_dir, item_type)

Iterate over partitions accross all devices.

Parameters:
  • devices_dir – Path to devices
  • item_type – One of ‘accounts’, ‘containers’, or ‘objects’
Returns:

Each iteration returns a tuple of (device, partition)

swift.common.utils.load_libc_function(func_name, log_error=True)

Attempt to find the function in libc, otherwise return a no-op func.

Parameters:func_name – name of the function to pull from libc.
swift.common.utils.lock_file(*args, **kwds)

Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:
  • filename – file to be locked
  • timeout – timeout (in seconds)
  • append – True if file should be opened in append mode
  • unlink – True if the file should be unlinked at the end
swift.common.utils.lock_parent_directory(filename, timeout=10)

Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:
  • filename – file path of the parent directory to be locked
  • timeout – timeout (in seconds)
swift.common.utils.lock_path(*args, **kwds)

Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.

Parameters:
  • directory – directory to be locked
  • timeout – timeout (in seconds)
swift.common.utils.mkdirs(path)

Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.

Parameters:path – path to create
swift.common.utils.normalize_timestamp(timestamp)

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx format.

Parameters:timestamp – unix timestamp
Returns:normalized timestamp as a string
swift.common.utils.parse_options(parser=None, once=False, test_args=None)

Parse standard swift server/daemon options with optparse.OptionParser.

Parameters:
  • parser – OptionParser to use. If not sent one will be created.
  • once – Boolean indicating the “once” option is available
  • test_args – Override sys.argv; used in testing
:returns : Tuple of (config, options); config is an absolute path to the
config file, options is the parser options as a dictionary.
Raises SystemExit:
 First arg (CONFIG) is required, file must exist
swift.common.utils.public(func)

Decorator to declare which methods are publicly accessible as HTTP requests

Parameters:func – function to make public
swift.common.utils.ratelimit_sleep(running_time, max_rate, incr_by=1, rate_buffer=5)

Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.

Parameters:
  • running_time – the running time of the next allowable request. Best to start at zero.
  • max_rate – The maximum rate per second allowed for the process.
  • incr_by – How much to increment the counter. Useful if you want to ratelimit 1024 bytes/sec and have differing sizes of requests. Must be >= 0.
  • rate_buffer – Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy.
swift.common.utils.readconf(conffile, section_name=None, log_name=None, defaults=None, raw=False)

Read config file and return config items as a dict

Parameters:
  • conffile – path to config file, or a file-like object (hasattr readline)
  • section_name – config section to read (will return all sections if not defined)
  • log_name – name to be used with logging (will use section_name if not defined)
  • defaults – dict of default values to pre-populate the config with
Returns:

dict of config items

swift.common.utils.remove_file(path)

Quiet wrapper for os.unlink, OSErrors are suppressed

Parameters:path – first and only argument passed to os.unlink
swift.common.utils.renamer(old, new)

Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.

Parameters:
  • old – old path to be renamed
  • new – new path to be renamed to
swift.common.utils.rsync_ip(ip)

Transform ip string to an rsync-compatible form

Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.

Parameters:ip – an ip string (ipv4 or ipv6)
Returns:a string ip address
swift.common.utils.search_tree(root, glob_match, ext)

Look in root, for any files/dirs matching glob, recurively traversing any found directories looking for files ending with ext

Parameters:
  • root – start of search path
  • glob_match – glob to match in root, matching dirs are traversed with os.walk
  • ext – only files that end in ext will be returned
Returns:

list of full paths to matching files, sorted

swift.common.utils.split_path(path, minsegs=1, maxsegs=None, rest_with_last=False)

Validate and split the given HTTP request path.

Examples:

['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters:
  • path – HTTP Request path to be split
  • minsegs – Minimum number of segments to be extracted
  • maxsegs – Maximum number of segments to be extracted
  • rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.
Returns:

list of segments with a length of maxsegs (non-existant segments will return as None)

Raises :

ValueError if given an invalid path

swift.common.utils.storage_directory(datadir, partition, hash)

Get the storage directory

Parameters:
  • datadir – Base data directory
  • partition – Partition
  • hash – Account, container or object hash
Returns:

Storage directory

swift.common.utils.streq_const_time(s1, s2)

Constant-time string comparison.

Params s1:the first string
Params s2:the second string
Returns:True if the strings are equal.

This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.

Remove any file in a given path that that was last modified before mtime.

Parameters:path – path to remove file from
Mtime :timestamp of oldest file to keep
swift.common.utils.urlparse(url)

urlparse augmentation. This is necessary because urlparse can’t handle RFC 2732 URLs.

Parameters:url – URL to parse.
swift.common.utils.validate_device_partition(device, partition)

Validate that a device and a partition are valid and won’t lead to directory traversal when used.

Parameters:
  • device – device to validate
  • partition – partition to validate
Raises :

ValueError if given an invalid device or partition

swift.common.utils.whataremyips()

Get the machine’s ip addresses

Returns:list of Strings of ip addresses
swift.common.utils.write_file(path, contents)

Write contents to file at path

Parameters:
  • path – any path, subdirs will be created as needed
  • contents – data to write to file, will be converted to string
swift.common.utils.write_pickle(obj, dest, tmp=None, pickle_protocol=0)

Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location

Parameters:
  • obj – python object to be pickled
  • dest – path of final destination file
  • tmp – path to tmp to use, defaults to None
  • pickle_protocol – protocol to pickle the obj with, defaults to 0

TempAuth

class swift.common.middleware.tempauth.TempAuth(app, conf)

Bases: object

Test authentication and authorization system.

Add to your pipeline in proxy-server.conf, such as:

[pipeline:main]
pipeline = catch_errors cache tempauth proxy-server

Set account auto creation to true in proxy-server.conf:

[app:proxy-server]
account_autocreate = true

And add a tempauth filter section, such as:

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

See the proxy-server.conf-sample for more information.

Parameters:
  • app – The next WSGI app in the pipeline
  • conf – The dict of configuration values
authorize(req)

Returns None if the request is authorized to continue or a standard WSGI response callable if not.

denied_response(req)

Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.

get_groups(env, token)

Get groups for the given token.

Parameters:
  • env – The current WSGI environment dictionary.
  • token – Token to validate and return a group string for.
Returns:

None if the token is invalid or a string containing a comma separated list of groups the authenticated user is a member of. The first group in the list is also considered a unique identifier for that user.

handle(env, start_response)

WSGI entry point for auth requests (ones that match the self.auth_prefix). Wraps env in webob.Request object and passes it down.

Parameters:
  • env – WSGI environment dictionary
  • start_response – WSGI callable
handle_get_token(req)

Handles the various request for token and service end point(s) calls. There are various formats to support the various auth servers in the past. Examples:

GET <auth-prefix>/v1/<act>/auth
    X-Auth-User: <act>:<usr>  or  X-Storage-User: <usr>
    X-Auth-Key: <key>         or  X-Storage-Pass: <key>
GET <auth-prefix>/auth
    X-Auth-User: <act>:<usr>  or  X-Storage-User: <act>:<usr>
    X-Auth-Key: <key>         or  X-Storage-Pass: <key>
GET <auth-prefix>/v1.0
    X-Auth-User: <act>:<usr>  or  X-Storage-User: <act>:<usr>
    X-Auth-Key: <key>         or  X-Storage-Pass: <key>

On successful authentication, the response will have X-Auth-Token and X-Storage-Token set to the token to use with Swift and X-Storage-URL set to the URL to the default Swift cluster to use.

Parameters:req – The webob.Request to process.
Returns:webob.Response, 2xx on success with data set as explained above.
handle_request(req)

Entry point for auth requests (ones that match the self.auth_prefix). Should return a WSGI-style callable (such as webob.Response).

Parameters:req – webob.Request object
swift.common.middleware.tempauth.filter_factory(global_conf, **local_conf)

Returns a WSGI filter app for use with paste.deploy.

ACLs

swift.common.middleware.acl.clean_acl(name, value)

Returns a cleaned ACL header value, validating that it meets the formatting requirements for standard Swift ACL strings.

The ACL format is:

[item[,item...]]

Each item can be a group name to give access to or a referrer designation to grant or deny based on the HTTP Referer header.

The referrer designation format is:

.r:[-]value

The .r can also be .ref, .referer, or .referrer; though it will be shortened to just .r for decreased character count usage.

The value can be * to specify any referrer host is allowed access, a specific host name like www.example.com, or if it has a leading period . or leading *. it is a domain name specification, like .example.com or *.example.com. The leading minus sign - indicates referrer hosts that should be denied access.

Referrer access is applied in the order they are specified. For example, .r:.example.com,.r:-thief.example.com would allow all hosts ending with .example.com except for the specific host thief.example.com.

Example valid ACLs:

.r:*
.r:*,.r:-.thief.com
.r:*,.r:.example.com,.r:-thief.example.com
.r:*,.r:-.thief.com,bobs_account,sues_account:sue
bobs_account,sues_account:sue

Example invalid ACLs:

.r:
.r:-

By default, allowing read access via .r will not allow listing objects in the container – just retrieving objects from the container. To turn on listings, use the .rlistings directive.

Also, .r designations aren’t allowed in headers whose names include the word ‘write’.

ACLs that are “messy” will be cleaned up. Examples:

Original Cleaned
bob, sue bob,sue
bob , sue bob,sue
bob,,,sue bob,sue
.referrer : * .r:*
.ref:*.example.com .r:.example.com
.r:*, .rlistings .r:*,.rlistings
Parameters:
  • name – The name of the header being cleaned, such as X-Container-Read or X-Container-Write.
  • value – The value of the header being cleaned.
Returns:

The value, cleaned of extraneous formatting.

Raises ValueError:
 

If the value does not meet the ACL formatting requirements; the error message will indicate why.

swift.common.middleware.acl.parse_acl(acl_string)

Parses a standard Swift ACL string into a referrers list and groups list.

See clean_acl() for documentation of the standard Swift ACL format.

Parameters:acl_string – The standard Swift ACL string to parse.
Returns:A tuple of (referrers, groups) where referrers is a list of referrer designations (without the leading .r:) and groups is a list of groups to allow access.
swift.common.middleware.acl.referrer_allowed(referrer, referrer_acl)

Returns True if the referrer should be allowed based on the referrer_acl list (as returned by parse_acl()).

See clean_acl() for documentation of the standard Swift ACL format.

Parameters:
  • referrer – The value of the HTTP Referer header.
  • referrer_acl – The list of referrer designations as returned by parse_acl().
Returns:

True if the referrer should be allowed; False if not.

WSGI

WSGI tools for use with swift.

class swift.common.wsgi.WSGIContext(wsgi_app)

Bases: object

This class provides a means to provide context (scope) for a middleware filter to have access to the wsgi start_response results like the request status and headers.

swift.common.wsgi.get_socket(conf, default_port=8080)

Bind socket to bind ip:port in conf

Parameters:
  • conf – Configuration dict to read settings from
  • default_port – port to use if not specified in conf
:returns : a socket object as returned from socket.listen or
ssl.wrap_socket if conf specifies cert_file
swift.common.wsgi.make_pre_authed_env(env, method=None, path=None, agent='Swift', query_string=None)

Returns a new fresh WSGI environment with escalated privileges to do backend checks, listings, etc. that the remote user wouldn’t be able to accomplish directly.

Parameters:
  • env – The WSGI environment to base the new environment on.
  • method – The new REQUEST_METHOD or None to use the original.
  • path – The new path_info or none to use the original. path should NOT be quoted. When building a url, a Webob Request (in accordance with wsgi spec) will quote env[‘PATH_INFO’]. url += quote(environ[‘PATH_INFO’])
  • query_string – The new query_string or none to use the original. When building a url, a Webob Request will append the query string directly to the url. url += ‘?’ + env[‘QUERY_STRING’]
  • agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
Returns:

Fresh WSGI environment.

swift.common.wsgi.make_pre_authed_request(env, method=None, path=None, body=None, headers=None, agent='Swift')

Makes a new webob.Request based on the current env but with the parameters specified. Note that this request will be preauthorized.

Parameters:
  • env – The WSGI environment to base the new request on.
  • method – HTTP method of new request; default is from the original env.
  • path – HTTP path of new request; default is from the original env. path should be compatible with what you would send to Request.blank. path should be quoted and it can include a query string. for example: ‘/a%20space?unicode_str%E8%AA%9E=y%20es’
  • body – HTTP body of new request; empty by default.
  • headers – Extra HTTP headers of new request; None by default.
  • agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
Returns:

Fresh webob.Request object.

swift.common.wsgi.monkey_patch_mimetools()

mimetools.Message defaults content-type to “text/plain” This changes it to default to None, so we can detect missing headers.

swift.common.wsgi.run_wsgi(conf_file, app_section, *args, **kwargs)

Loads common settings from conf, then instantiates app and runs the server using the specified number of workers.

Parameters:
  • conf_file – Path to paste.deploy style configuration file
  • app_section – App name from conf file to load config from

Direct Client

Internal client library for making calls directly to the servers rather than through the proxy.

swift.common.direct_client.direct_delete_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers={})
swift.common.direct_client.direct_delete_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers={})

Delete object directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

response from server

swift.common.direct_client.direct_get_account(node, part, account, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15)

Get listings directly from the account server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the account is on
  • account – account name
  • marker – marker query
  • limit – query limit
  • prefix – prefix query
  • delimeter – delimeter for the query
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_get_container(node, part, account, container, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15)

Get container listings directly from the container server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • marker – marker query
  • limit – query limit
  • prefix – prefix query
  • delimeter – delimeter for the query
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_get_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, resp_chunk_size=None, headers={})

Get object directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
  • resp_chunk_size – if defined, chunk size of data to read.
  • headers – dict to be passed into HTTPConnection headers
Returns:

a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_head_container(node, part, account, container, conn_timeout=5, response_timeout=15)

Request container information directly from the container server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

a dict containing the response’s headers (all header names will be lowercase)

swift.common.direct_client.direct_head_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15)

Request object information directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • obj – object name
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Returns:

a dict containing the response’s headers (all header names will be lowercase)

swift.common.direct_client.direct_post_object(node, part, account, container, name, headers, conn_timeout=5, response_timeout=15)

Direct update to object metadata on object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • name – object name
  • headers – headers to store as metadata
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
Raises ClientException:
 

HTTP POST request failed

swift.common.direct_client.direct_put_object(node, part, account, container, name, contents, content_length=None, etag=None, content_type=None, headers=None, conn_timeout=5, response_timeout=15, resp_chunk_size=None)

Put object directly from the object server.

Parameters:
  • node – node dictionary from the ring
  • part – partition the container is on
  • account – account name
  • container – container name
  • name – object name
  • contents – a string to read object data from
  • content_length – value to send as content-length header
  • etag – etag of contents
  • content_type – value to send as content-type header
  • headers – additional headers to include in the request
  • conn_timeout – timeout in seconds for establishing the connection
  • response_timeout – timeout in seconds for getting the response
  • chunk_size – if defined, chunk size of data to send.
Returns:

etag from the server response

swift.common.direct_client.quote(value, safe='/')
swift.common.direct_client.retry(func, *args, **kwargs)

Helper function to retry a given function a number of times.

Parameters:
  • func – callable to be called
  • retries – number of retries
  • error_log – logger for errors
  • args – arguments to send to func
  • kwargs – keyward arguments to send to func (if retries or error_log are sent, they will be deleted from kwargs before sending on to func)
Returns:

restult of func

Internal Client

class swift.common.internal_client.CompressingFileReader(file_obj, compresslevel=9)

Bases: object

Wrapper for file object to compress object while reading.

Can be used to wrap file objects passed to InternalClient.upload_object().

Used in testing of InternalClient.

Parameters:
  • file_obj – File object to wrap.
  • compresslevel – Compression level, defaults to 9.
next()
read(*a, **kw)

Reads a chunk from the file object.

Params are passed directly to the underlying file object’s read().

Returns:Compressed chunk from file object.
class swift.common.internal_client.InternalClient(conf_path, user_agent, request_tries)

Bases: object

An internal client that uses a swift proxy app to make requests to Swift.

This client will exponentially slow down for retries.

Parameters:
  • conf_path – Full path to proxy config.
  • user_agent – User agent to be sent to requests to Swift.
  • request_tries – Number of tries before InternalClient.make_request() gives up.
container_exists(account, container)

Checks to see if a container exists.

Parameters:
  • account – The container’s account.
  • container – Container to check.

:returns : True if container exists, false otherwise.

Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
create_container(account, container, headers=None, acceptable_statuses=(2, ))

Creates container.

Parameters:
  • account – The container’s account.
  • container – Container to create.
  • headers – Defaults to empty dict.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
delete_container(account, container, acceptable_statuses=(2, 404))

Deletes a container.

Parameters:
  • account – The container’s account.
  • container – Container to delete.
  • acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
delete_object(account, container, object_name, acceptable_statuses=(2, 404))

Deletes an object.

Parameters:
  • account – The object’s account.
  • container – The object’s container.
  • object_name – The object.
  • acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
get_account_info(account, acceptable_statuses=(2, 404))

Returns (container_count, object_count) for an account.

Parameters:
  • account – Account on which to get the information.
  • acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
get_account_metadata(account, metadata_prefix='', acceptable_statuses=(2, ))

Gets account metadata.

Parameters:
  • account – Account on which to get the metadata.
  • metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Returns dict of account metadata.

Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
get_container_metadata(account, container, metadata_prefix='', acceptable_statuses=(2, ))

Gets container metadata.

Parameters:
  • account – The container’s account.
  • container – Container to get metadata on.
  • metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Returns dict of container metadata.

Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
get_object_metadata(account, container, object_name, metadata_prefix='', acceptable_statuses=(2, ))

Gets object metadata.

Parameters:
  • account – The object’s account.
  • container – The object’s container.
  • object_name – The object.
  • metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Dict of object metadata.

Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
iter_containers(account, marker='', end_marker='', acceptable_statuses=(2, 404))

Returns an iterator of containers dicts from an account.

Parameters:
  • account – Account on which to do the container listing.
  • marker – Prefix of first desired item, defaults to ‘’.
  • end_marker – Last item returned will be ‘less’ than this, defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
iter_object_lines(account, container, object_name, headers=None, acceptable_statuses=(2, ))

Returns an iterator of object lines from an uncompressed or compressed text object.

Uncompress object as it is read if the object’s name ends with ‘.gz’.

Parameters:
  • account – The object’s account.
  • container – The object’s container.
  • objec_namet – The object.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
iter_objects(account, container, marker='', end_marker='', acceptable_statuses=(2, 404))

Returns an iterator of object dicts from a container.

Parameters:
  • account – The container’s account.
  • container – Container to iterate objects on.
  • marker – Prefix of first desired item, defaults to ‘’.
  • end_marker – Last item returned will be ‘less’ than this, defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
make_request(method, path, headers, acceptable_statuses, body_file=None)

Makes a request to Swift with retries.

Parameters:
  • method – HTTP method of request.
  • path – Path of request.
  • headers – Headers to be sent with request.
  • acceptable_statuses – List of acceptable statuses for request.
  • body_file – Body file to be passed along with request, defaults to None.

:returns : Response object on success.

Raises:
  • UnexpectedResponse – Exception raised when make_request() fails to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
set_account_metadata(account, metadata, metadata_prefix='', acceptable_statuses=(2, ))

Sets account metadata. A call to this will add to the account metadata and not overwrite all of it with values in the metadata dict. To clear an account metadata value, pass an empty string as the value for the key in the metadata dict.

Parameters:
  • account – Account on which to get the metadata.
  • metadata – Dict of metadata to set.
  • metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
set_container_metadata(account, container, metadata, metadata_prefix='', acceptable_statuses=(2, ))

Sets container metadata. A call to this will add to the container metadata and not overwrite all of it with values in the metadata dict. To clear a container metadata value, pass an empty string as the value for the key in the metadata dict.

Parameters:
  • account – The container’s account.
  • container – Container to set metadata on.
  • metadata – Dict of metadata to set.
  • metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
set_object_metadata(account, container, object_name, metadata, metadata_prefix='', acceptable_statuses=(2, ))

Sets an object’s metadata. The object’s metadata will be overwritten by the values in the metadata dict.

Parameters:
  • account – The object’s account.
  • container – The object’s container.
  • object_name – The object.
  • metadata – Dict of metadata to set.
  • metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
  • acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
upload_object(fobj, account, container, object_name, headers=None)
Parameters:
  • fobj – File object to read object’s content from.
  • account – The object’s account.
  • container – The object’s container.
  • object_name – The object.
  • headers – Headers to send with request, defaults ot empty dict.
Raises:
  • UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
  • Exception – Exception is raised when code fails in an unexpected way.
exception swift.common.internal_client.UnexpectedResponse(message, resp)

Bases: exceptions.Exception

Exception raised on invalid responses to InternalClient.make_request().

Parameters:
  • message – Exception message.
  • resp – The unexpected response.

Buffered HTTP

Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.

Warning

If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.

class swift.common.bufferedhttp.BufferedHTTPConnection(host, port=None, strict=None, timeout=<object object at 0x557d2668>, source_address=None)

Bases: httplib.HTTPConnection

HTTPConnection class that uses BufferedHTTPResponse

response_class

alias of BufferedHTTPResponse

class swift.common.bufferedhttp.BufferedHTTPResponse(sock, debuglevel=0, strict=0, method=None)

Bases: httplib.HTTPResponse

HTTPResponse class that buffers reading of headers

swift.common.bufferedhttp.http_connect(ipaddr, port, device, partition, method, path, headers=None, query_string=None, ssl=False)

Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.

Parameters:
  • ipaddr – IPv4 address to connect to
  • port – port to connect to
  • device – device of the node to query
  • partition – partition on the device
  • method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
  • path – request path
  • headers – dictionary of headers
  • query_string – request query string
  • ssl – set True if SSL should be used (default: False)
Returns:

HTTPConnection object

swift.common.bufferedhttp.http_connect_raw(ipaddr, port, method, path, headers=None, query_string=None, ssl=False)

Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.

Parameters:
  • ipaddr – IPv4 address to connect to
  • port – port to connect to
  • method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
  • path – request path
  • headers – dictionary of headers
  • query_string – request query string
  • ssl – set True if SSL should be used (default: False)
Returns:

HTTPConnection object

Healthcheck

class swift.common.middleware.healthcheck.HealthCheckMiddleware(app, *args, **kwargs)

Bases: object

Healthcheck middleware used for monitoring.

If the path is /healthcheck, it will respond with “OK” in the body

GET(req)

Returns a 200 response with “OK” in the body.

Recon

class swift.common.middleware.recon.ReconMiddleware(app, conf, *args, **kwargs)

Bases: object

Recon middleware used for monitoring.

/recon/load|mem|async... will return various system metrics.

Needs to be added to the pipeline and a requires a filter declaration in the object-server.conf:

[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift

get_async_info()

get # of async pendings

get_auditor_info(recon_type)

get auditor info

get_device_info()

get devices

get_diskusage()

get disk utilization statistics

get_expirer_info(recon_type)

get expirer info

get_load(openr=<built-in function open>)

get info from /proc/loadavg

get_mem(openr=<built-in function open>)

get info from /proc/meminfo

get_mounted(openr=<built-in function open>)

get ALL mounted fs from /proc/mounts

get_quarantine_count()

get obj/container/account quarantine counts

get_replication_info(recon_type)

get replication info

get_ring_md5(openr=<built-in function open>)

get all ring md5sum’s

get_socket_info(openr=<built-in function open>)

get info from /proc/net/sockstat and sockstat6

Note: The mem value is actually kernel pages, but we return bytes allocated based on the systems page size.

get_unmounted()

list unmounted (failed?) devices

get_updater_info(recon_type)

get updater info

MemCacheD

Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:

http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt

class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.3, io_timeout=2.0, tries=3, allow_pickle=False, allow_unpickle=False)

Bases: object

Simple, consistent-hashed memcache client.

decr(key, delta=1, timeout=0)

Decrements a key which has a numeric value by delta. Calls incr with -delta.

Parameters:
  • key – key
  • delta – amount to subtract to the value of key (or set the value to 0 if the key is not found) will be cast to an int
  • timeout – ttl in memcache
Raises MemcacheConnectionError:
 

delete(key)

Deletes a key/value pair from memcache.

Parameters:key – key to be deleted
get(key)

Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.

Parameters:key – key
Returns:value of the key in memcache
get_multi(keys, server_key)

Gets multiple values from memcache for the given keys.

Parameters:
  • keys – keys for values to be retrieved from memcache
  • servery_key – key to use in determining which server in the ring is used
Returns:

list of values

incr(key, delta=1, timeout=0)

Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.

Parameters:
  • key – key
  • delta – amount to add to the value of key (or set as the value if the key is not found) will be cast to an int
  • timeout – ttl in memcache
Raises MemcacheConnectionError:
 

set(key, value, serialize=True, timeout=0)

Set a key/value pair in memcache

Parameters:
  • key – key
  • value – value
  • serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
  • timeout – ttl in memcache
set_multi(mapping, server_key, serialize=True, timeout=0)

Sets multiple key/value pairs in memcache.

Parameters:
  • mapping – dictonary of keys and values to be set in memcache
  • servery_key – key to use in determining which server in the ring is used
  • serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
  • timeout – ttl for memcache

Manager

class swift.common.manager.Manager(servers)

Main class for performing commands on groups of servers.

Parameters:servers – list of server names as strings
force_reload(*a, **kw)

alias for reload

get_command(cmd)

Find and return the decorated method named like cmd

Parameters:cmd – the command to get, a string, if not found raises UnknownCommandError
classmethod list_commands()

Get all publicly accessible commands

Returns:a list of string tuples (cmd, help), the method names who are decorated as commands
no_daemon(*a, **kw)

start a server interactively

no_wait(*a, **kw)

spawn server and return immediately

once(*a, **kw)

start server and run one pass on supporting daemons

reload(*a, **kw)

graceful shutdown then restart on supporting servers

restart(*a, **kw)

stops then restarts server

run_command(cmd, **kwargs)

Find the named command and run it

Parameters:cmd – the command name to run
shutdown(*a, **kw)

allow current requests to finish on supporting servers

start(*a, **kw)

starts a server

status(*a, **kw)

display status of tracked pids for server

stop(*a, **kw)

stops a server

class swift.common.manager.Server(server)

Manage operations on a server or group of servers of similar type

Parameters:server – name of server
conf_files(**kwargs)

Get conf files for this server

Param :number, if supplied will only lookup the nth server
Returns:list of conf files
get_conf_file_name(pid_file)

Translate pid_file to a corresponding conf_file

Parameters:pid_file – a pid_file for this server, a string
Returns:the conf_file for this pid_file
get_pid_file_name(conf_file)

Translate conf_file to a corresponding pid_file

Parameters:conf_file – an conf_file for this server, a string
Returns:the pid_file for this conf_file
get_running_pids(**kwargs)

Get running pids

Returns:a dict mapping pids (ints) to pid_files (paths)
interact(**kwargs)

wait on spawned procs to terminate

iter_pid_files(**kwargs)

Generator, yields (pid_file, pids)

kill_running_pids(**kwargs)

Kill running pids

Parameters:graceful – if True, attempt SIGHUP on supporting servers
Returns:a dict mapping pids (ints) to pid_files (paths)
launch(**kwargs)

Collect conf files and attempt to spawn the processes for this server

pid_files(**kwargs)

Get pid files for this server

Param :number, if supplied will only lookup the nth server
Returns:list of pid files
signal_pids(sig, **kwargs)

Send a signal to pids for this server

Parameters:sig – signal to send
Returns:a dict mapping pids (ints) to pid_files (paths)
spawn(conf_file, once=False, wait=True, daemon=True, **kwargs)

Launch a subprocess for this server.

Parameters:
  • conf_file – path to conf_file to use as first arg
  • once – boolean, add once argument to command
  • wait – boolean, if true capture stdout with a pipe
  • daemon – boolean, if true ask server to log to console

:returns : the pid of the spawned process

status(pids=None, **kwargs)

Display status of server

Param :pids, if not supplied pids will be populated automatically
Param :number, if supplied will only lookup the nth server
Returns:1 if server is not running, 0 otherwise
stop(**kwargs)

Send stop signals to pids for this server

Returns:a dict mapping pids (ints) to pid_files (paths)
wait(**kwargs)

wait on spawned procs to start

swift.common.manager.command(func)

Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.

Parameters:func – function to make public
swift.common.manager.setup_env()

Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp

swift.common.manager.watch_server_pids(server_pids, interval=1, **kwargs)

Monitor a collection of server pids yeilding back those pids that aren’t responding to signals.

Parameters:server_pids – a dict, lists of pids [int,...] keyed on Server objects

Ratelimit

class swift.common.middleware.ratelimit.RateLimitMiddleware(app, conf, logger=None)

Bases: object

Rate limiting middleware

Rate limits requests on both an Account and Container level. Limits are configurable.

get_container_maxrate(container_size)

Returns number of requests allowed per second for given container size.

get_ratelimitable_key_tuples(req_method, account_name, container_name=None, obj_name=None)

Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.

Parameters:
  • req_method – HTTP method
  • account_name – account name from path
  • container_name – container name from path
  • obj_name – object name from path
handle_ratelimit(req, account_name, container_name, obj_name)

Performs rate limiting and account white/black listing. Sleeps if necessary.

Parameters:
  • account_name – account name from path
  • container_name – container name from path
  • obj_name – object name from path
swift.common.middleware.ratelimit.filter_factory(global_conf, **local_conf)

paste.deploy app factory for creating WSGI proxy apps.

StaticWeb

This StaticWeb WSGI middleware will serve container data as a static web site with index file and error file resolution and optional file listings. This mode is normally only active for anonymous requests. If you want to use it with authenticated requests, set the X-Web-Mode: true header on the request.

The staticweb filter should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. Also, the configuration section for the staticweb middleware itself needs to be added. For example:

[DEFAULT]
...

[pipeline:main]
pipeline = healthcheck cache tempauth staticweb proxy-server

...

[filter:staticweb]
use = egg:swift#staticweb
# Seconds to cache container x-container-meta-web-* header values.
# cache_timeout = 300
# You can override the default log routing for this filter here:
# set log_name = staticweb
# set log_facility = LOG_LOCAL0
# set log_level = INFO
# set access_log_name = staticweb
# set access_log_facility = LOG_LOCAL0
# set access_log_level = INFO
# set log_headers = False

Any publicly readable containers (for example, X-Container-Read: .r:*, see acls for more information on this) will be checked for X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values:

X-Container-Meta-Web-Index  <index.name>
X-Container-Meta-Web-Error  <error.name.suffix>

If X-Container-Meta-Web-Index is set, any <index.name> files will be served without having to specify the <index.name> part. For instance, setting X-Container-Meta-Web-Index: index.html will be able to serve the object .../pseudo/path/index.html with just .../pseudo/path or .../pseudo/path/

If X-Container-Meta-Web-Error is set, any errors (currently just 401 Unauthorized and 404 Not Found) will instead serve the .../<status.code><error.name.suffix> object. For instance, setting X-Container-Meta-Web-Error: error.html will serve .../404error.html for requests for paths not found.

For psuedo paths that have no <index.name>, this middleware can serve HTML file listings if you set the X-Container-Meta-Web-Listings: true metadata item on the container.

If listings are enabled, the listings can have a custom style sheet by setting the X-Container-Meta-Web-Listings-CSS header. For instance, setting X-Container-Meta-Web-Listings-CSS: listing.css will make listings link to the .../listing.css style sheet. If you “view source” in your browser on a listing page, you will see the well defined document structure that can be styled.

Example usage of this middleware via swift:

Make the container publicly readable:

swift post -r '.r:*' container

You should be able to get objects directly, but no index.html resolution or listings.

Set an index file directive:

swift post -m 'web-index:index.html' container

You should be able to hit paths that have an index.html without needing to type the index.html part.

Turn on listings:

swift post -m 'web-listings: true' container

Now you should see object listings for paths and pseudo paths that have no index.html.

Enable a custom listings style sheet:

swift post -m 'web-listings-css:listings.css' container

Set an error file:

swift post -m 'web-error:error.html' container

Now 401’s should load 401error.html, 404’s should load 404error.html, etc.

class swift.common.middleware.staticweb.StaticWeb(app, conf)

Bases: object

The Static Web WSGI middleware filter; serves container data as a static web site. See staticweb for an overview.

Parameters:
  • app – The next WSGI application/filter in the paste.deploy pipeline.
  • conf – The filter configuration dict.
access_logger = None

Web access logger for this filter.

app = None

The next WSGI application/filter in the paste.deploy pipeline.

cache_timeout = None

The seconds to cache the x-container-meta-web-* headers.,

conf = None

The filter configuration dict.

log_headers = None

Indicates whether full HTTP headers should be logged or not.

logger = None

Logger for this filter.

swift.common.middleware.staticweb.filter_factory(global_conf, **local_conf)

Returns a Static Web WSGI filter for use with paste.deploy.

swift.common.middleware.staticweb.quote(value, safe='/')

Patched version of urllib.quote that encodes utf-8 strings before quoting

TempURL

TempURL Middleware

Allows the creation of URLs to provide temporary access to objects.

For example, a website may wish to provide a link to download a large object in Swift, but the Swift account has no public access. The website can generate a URL that will provide GET access for a limited time to the resource. When the web browser user clicks on the link, the browser will download the object directly from Swift, obviating the need for the website to act as a proxy for the request.

If the user were to share the link with all his friends, or accidentally post it on a forum, etc. the direct access would be limited to the expiration time set when the website created the link.

To create such temporary URLs, first an X-Account-Meta-Temp-URL-Key header must be set on the Swift account. Then, an HMAC-SHA1 (RFC 2104) signature is generated using the HTTP method to allow (GET or PUT), the Unix timestamp the access should be allowed until, the full path to the object, and the key set on the account.

For example, here is code generating the signature for a GET for 60 seconds on /v1/AUTH_account/container/object:

import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + 60)
path = '/v1/AUTH_account/container/object'
key = 'mykey'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()

Be certain to use the full path, from the /v1/ onward.

Let’s say the sig ends up equaling da39a3ee5e6b4b0d3255bfef95601890afd80709 and expires ends up 1323479485. Then, for example, the website could provide a link to:

https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485

Any alteration of the resource path or query arguments would result in 401 Unauthorized. Similary, a PUT where GET was the allowed method would 401. HEAD is allowed if GET or PUT is allowed.

Using this in combination with browser form post translation middleware could also allow direct-from-browser uploads to specific locations in Swift.

Note that changing the X-Account-Meta-Temp-URL-Key will invalidate any previously generated temporary URLs within 60 seconds (the memcache time for the key).

class swift.common.middleware.tempurl.TempURL(app, conf)

Bases: object

WSGI Middleware to grant temporary URLs specific access to Swift resources. See the overview for more information.

This middleware understands the following configuration settings:

incoming_remove_headers
    The headers to remove from incoming requests. Simply a
    whitespace delimited list of header names and names can
    optionally end with '*' to indicate a prefix match.
    incoming_allow_headers is a list of exceptions to these
    removals.
    Default: x-timestamp

incoming_allow_headers
    The headers allowed as exceptions to
    incoming_remove_headers. Simply a whitespace delimited
    list of header names and names can optionally end with
    '*' to indicate a prefix match.
    Default: None

outgoing_remove_headers
    The headers to remove from outgoing responses. Simply a
    whitespace delimited list of header names and names can
    optionally end with '*' to indicate a prefix match.
    outgoing_allow_headers is a list of exceptions to these
    removals.
    Default: x-object-meta-*

outgoing_allow_headers
    The headers allowed as exceptions to
    outgoing_remove_headers. Simply a whitespace delimited
    list of header names and names can optionally end with
    '*' to indicate a prefix match.
    Default: x-object-meta-public-*
Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.
agent = None

HTTP user agent to use for subrequests.

app = None

The next WSGI application/filter in the paste.deploy pipeline.

conf = None

The filter configuration dict.

incoming_allow_headers = None

Headers to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY.

incoming_allow_headers_startswith = None

Header with match prefixes to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY_*.

incoming_remove_headers = None

Headers to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_PRIVATE.

incoming_remove_headers_startswith = None

Header with match prefixes to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_SENSITIVE_*.

logger = None

The logger to use with this middleware.

outgoing_allow_headers = None

Headers to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay.

outgoing_allow_headers_startswith = None

Header with match prefixes to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay-*.

outgoing_remove_headers = None

Headers to remove from outgoing responses. Lowercase, like x-account-meta-temp-url-key.

outgoing_remove_headers_startswith = None

Header with match prefixes to remove from outgoing responses. Lowercase, like x-account-meta-private-*.

swift.common.middleware.tempurl.filter_factory(global_conf, **local_conf)

Returns the WSGI filter for use with paste.deploy.

swift.common.middleware.tempurl.DEFAULT_INCOMING_REMOVE_HEADERS = 'x-timestamp'

Default headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_INCOMING_ALLOW_HEADERS is a list of exceptions to these removals.

swift.common.middleware.tempurl.DEFAULT_INCOMING_ALLOW_HEADERS = ''

Default headers as exceptions to DEFAULT_INCOMING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.

swift.common.middleware.tempurl.DEFAULT_OUTGOING_REMOVE_HEADERS = 'x-object-meta-*'

Default headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_OUTGOING_ALLOW_HEADERS is a list of exceptions to these removals.

swift.common.middleware.tempurl.DEFAULT_OUTGOING_ALLOW_HEADERS = 'x-object-meta-public-*'

Default headers as exceptions to DEFAULT_OUTGOING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.

FormPost

FormPost Middleware

Translates a browser form post into a regular Swift object PUT.

The format of the form is:

<form action="<swift-url>" method="POST"
      enctype="multipart/form-data">
  <input type="hidden" name="redirect" value="<redirect-url>" />
  <input type="hidden" name="max_file_size" value="<bytes>" />
  <input type="hidden" name="max_file_count" value="<count>" />
  <input type="hidden" name="expires" value="<unix-timestamp>" />
  <input type="hidden" name="signature" value="<hmac>" />
  <input type="file" name="file1" /><br />
  <input type="submit" />
</form>

The <swift-url> is the URL to the Swift desination, such as:

https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix

The name of each file uploaded will be appended to the <swift-url> given. So, you can upload directly to the root of container with a url like:

https://swift-cluster.example.com/v1/AUTH_account/container/

Optionally, you can include an object prefix to better separate different users’ uploads, such as:

https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix

Note the form method must be POST and the enctype must be set as “multipart/form-data”.

The redirect attribute is the URL to redirect the browser to after the upload completes. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”).

The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes.

The max_file_count attribute must be included and indicates the maximum number of files that can be uploaded with the form. Include additional <input type="file" name="filexx" /> attributes if desired.

The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated.

The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature:

import hmac
from hashlib import sha1
from time import time
path = '/v1/account/container/object_prefix'
redirect = 'https://myserver.com/some-page'
max_file_size = 104857600
max_file_count = 10
expires = int(time() + 600)
key = 'mykey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
    max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()

The key is the value of the X-Account-Meta-Temp-URL-Key header on the account.

Be certain to use the full path, from the /v1/ onward.

The command line tool swift-form-signature may be used (mostly just when testing) to compute expires and signature.

Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).

class swift.common.middleware.formpost.FormPost(app, conf)

Bases: object

FormPost Middleware

See above for a full description.

Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.
agent = None

The HTTP user agent to use with subrequests.

app = None

The next WSGI application/filter in the paste.deploy pipeline.

conf = None

The filter configuration dict.

logger = None

The logger to use with this middleware.

swift.common.middleware.formpost.filter_factory(global_conf, **local_conf)

Returns the WSGI filter for use with paste.deploy.

swift.common.middleware.formpost.READ_CHUNK_SIZE = 4096

The size of data to read from the form at any given time.

swift.common.middleware.formpost.MAX_VALUE_LENGTH = 4096

The maximum size of any attribute’s value. Any additional data will be truncated.

Domain Remap

Domain Remap Middleware

Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.

container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object

account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object

Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. The reseller_prefixes list is exclusive. If defined, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.

Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).

Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.

class swift.common.middleware.domain_remap.DomainRemapMiddleware(app, conf)

Bases: object

Domain Remap Middleware

See above for a full description.

Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.

CNAME Lookup

CNAME Lookup Middleware

Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.

This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.

class swift.common.middleware.cname_lookup.CNAMELookupMiddleware(app, conf)

Bases: object

CNAME Lookup Middleware

See above for a full description.

Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.
swift.common.middleware.cname_lookup.lookup_cname(domain)

Given a domain, returns its DNS CNAME mapping and DNS ttl.

Parameters:domain – domain to query on
Returns:(ttl, result)

Proxy Logging

Logging middleware for the Swift proxy.

This serves as both the default logging implementation and an example of how to plug in your own logging format/method.

The logging format implemented below is as follows:

client_ip remote_addr datetime request_method request_path protocol
status_int referer user_agent auth_token bytes_recvd bytes_sent client_etag transaction_id headers request_time source

These values are space-separated, and each is url-encoded, so that they can be separated with a simple .split()

  • remote_addr is the contents of the REMOTE_ADDR environment variable, while client_ip is swift’s best guess at the end-user IP, extracted variously from the X-Forwarded-For header, X-Cluster-Ip header, or the REMOTE_ADDR environment variable.
  • Values that are missing (e.g. due to a header not being present) or zero are generally represented by a single hyphen (‘-‘).
class swift.common.middleware.proxy_logging.InputProxy(wsgi_input)

Bases: object

File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.

read(*args, **kwargs)

Pass read request to the underlying file-like object and add bytes read to total.

readline(*args, **kwargs)

Pass readline request to the underlying file-like object and add bytes read to total.

class swift.common.middleware.proxy_logging.ProxyLoggingMiddleware(app, conf)

Bases: object

Middleware that logs Swift proxy requests in the swift log format.

log_request(env, status_int, bytes_received, bytes_sent, request_time, client_disconnect)

Log a request.

Parameters:
  • env – WSGI environment
  • status_int – integer code for the response status
  • bytes_received – bytes successfully read from the request body
  • bytes_sent – bytes yielded to the WSGI server
  • request_time – time taken to satisfy the request, in seconds