Bases: eventlet.timeout.Timeout
Bases: eventlet.timeout.Timeout
Bases: eventlet.timeout.Timeout
Bases: eventlet.timeout.Timeout
Bases: exceptions.Exception
Max container list length of a get request for an account
Max object list length of a get request for a container
Query string format= values to their corresponding content-type values
Max file size allowed for objects
Max number of metadata items
Max length of the name of a key for metadata
Max overall size of metadata
Max length of the value of a key for metadata
Max object name length
Helper function for checking if a string can be converted to a float.
Parameters: | string – string to be verified as a float |
---|---|
Returns: | True if the string can be converted to a float, False otherwise |
Check metadata sent in the request headers.
Parameters: |
|
---|---|
Raises HTTPBadRequest: | |
bad metadata |
Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidently filling up the root partition.
Parameters: |
|
---|---|
Returns: | True if it is a valid mounted device, False otherwise |
Check to ensure that everything is alright about an object to be created.
Parameters: |
|
---|---|
Raises: |
|
Validate if a string is valid UTF-8 str or unicode
Parameters: | string – string to be validated |
---|---|
Returns: | True if the string is valid utf-8 str or unicode, False otherwise |
Miscellaneous utility functions for use with Swift.
Bases: eventlet.greenpool.GreenPool
GreenPool subclassed to kill its coros when it gets gc’ed
Bases: logging.LoggerAdapter, object
A Logger like object which performs some reformatting on calls to exception(). Can be used to store a threadlocal transaction id and client ip.
Convenience function for syslog priority LOG_NOTICE. The python logging lvl is set to 25, just above info. SysLogHandler is monkey patched to map this log lvl to the LOG_NOTICE syslog priority.
Add extra info to message
The StatsD client prefix defaults to the “name” of the logger. This method may override that default with a specific value. Currently used in the proxy-server to differentiate the Account, Container, and Object controllers.
Factory which creates methods which delegate to methods on self.logger.statsd_client (an instance of StatsdClient). The created methods conditionally delegate to a method whose name is given in ‘statsd_func_name’. The created delegate methods are a no-op when StatsD logging is not configured. The created delegate methods also handle the defaulting of sample_rate (to either the default specified in the config with ‘log_statsd_default_sample_rate’ or the value passed into delegate function).
Parameters: | statsd_func_name – the name of a method on StatsdClient. |
---|
Bases: urlparse.ParseResult
Parse results class for urlparse.
A no-op logger for eventlet wsgi.
Bases: logging.Formatter
Custom logging.Formatter will append txn_id to a log message if the record has one and the message does not.
Given a devices path and a data directory, yield (path, device, partition) for all files in that directory
Parameters: |
|
---|
Get memcache connection pool from the environment (which had been previously set by the memcache middleware
Parameters: | env – wsgi environment dict |
---|---|
Returns: | swift.common.memcached.MemcacheRing from environment |
Log unhandled exceptions, close stdio, capture stdout and stderr.
param logger: Logger object to use
Compute an ETA. Now only if we could also have a progress bar...
Parameters: |
|
---|---|
Returns: | ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
Drop ‘buffer’ cache for the given range of the given file.
Parameters: |
|
---|
Sets the userid/groupid of the current process, get session leader, etc.
Parameters: | user – User name to change privileges to |
---|
Update recon cache values
Parameters: |
|
---|
Pre-allocate disk space for a file.
Parameters: |
|
---|
Get the current system logger using config settings.
Log config and defaults:
log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
log_udp_host = (disabled)
log_udp_port = logging.handlers.SYSLOG_UDP_PORT
log_address = /dev/log
log_statsd_host = (disabled)
log_statsd_port = 8125
log_statsd_default_sample_rate = 1
log_statsd_metric_prefix = (empty-string)
Parameters: |
|
---|
Get parameters from an HTTP request ensuring proper handling UTF-8 encoding.
Parameters: |
|
---|---|
Returns: | HTTP request parameter value |
Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)
Parameters: | time_amount – length of time in seconds |
---|---|
Returns: | A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
Get valid parts of utf-8 str from str, unicode and even invalid utf-8 str
Parameters: | str_or_unicode – a string or an unicode which can be invalid utf-8 |
---|
Get the connonical hash for an account/container/object
Parameters: |
|
---|---|
Returns: | hash string |
Returns the number in a human readable format; for example 1048576 = “1Mi”.
Get a value from the wsgi environment
Parameters: |
|
---|---|
Returns: | the value from the environment |
Iterate over partitions accross all devices.
Parameters: |
|
---|---|
Returns: | Each iteration returns a tuple of (device, partition) |
Attempt to find the function in libc, otherwise return a no-op func.
Parameters: | func_name – name of the function to pull from libc. |
---|
Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.
Parameters: |
|
---|
Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.
Parameters: | path – path to create |
---|
Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx format.
Parameters: | timestamp – unix timestamp |
---|---|
Returns: | normalized timestamp as a string |
Parse standard swift server/daemon options with optparse.OptionParser.
Parameters: |
|
---|
Raises SystemExit: | |
---|---|
First arg (CONFIG) is required, file must exist |
Decorator to declare which methods are publicly accessible as HTTP requests
Parameters: | func – function to make public |
---|
Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.
Parameters: |
|
---|
Read config file and return config items as a dict
Parameters: |
|
---|---|
Returns: | dict of config items |
Quiet wrapper for os.unlink, OSErrors are suppressed
Parameters: | path – first and only argument passed to os.unlink |
---|
Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
Parameters: |
|
---|
Transform ip string to an rsync-compatible form
Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.
Parameters: | ip – an ip string (ipv4 or ipv6) |
---|---|
Returns: | a string ip address |
Look in root, for any files/dirs matching glob, recurively traversing any found directories looking for files ending with ext
Parameters: |
|
---|---|
Returns: | list of full paths to matching files, sorted |
Validate and split the given HTTP request path.
Examples:
['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters: |
|
---|---|
Returns: | list of segments with a length of maxsegs (non-existant segments will return as None) |
Raises : | ValueError if given an invalid path |
Get the storage directory
Parameters: |
|
---|---|
Returns: | Storage directory |
Constant-time string comparison.
Params s1: | the first string |
---|---|
Params s2: | the second string |
Returns: | True if the strings are equal. |
This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.
Remove any file in a given path that that was last modified before mtime.
Parameters: | path – path to remove file from |
---|---|
Mtime : | timestamp of oldest file to keep |
urlparse augmentation. This is necessary because urlparse can’t handle RFC 2732 URLs.
Parameters: | url – URL to parse. |
---|
Validate that a device and a partition are valid and won’t lead to directory traversal when used.
Parameters: |
|
---|---|
Raises : | ValueError if given an invalid device or partition |
Get the machine’s ip addresses
Returns: | list of Strings of ip addresses |
---|
Write contents to file at path
Parameters: |
|
---|
Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location
Parameters: |
|
---|
Bases: object
Test authentication and authorization system.
Add to your pipeline in proxy-server.conf, such as:
[pipeline:main]
pipeline = catch_errors cache tempauth proxy-server
Set account auto creation to true in proxy-server.conf:
[app:proxy-server]
account_autocreate = true
And add a tempauth filter section, such as:
[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
See the proxy-server.conf-sample for more information.
Parameters: |
|
---|
Returns None if the request is authorized to continue or a standard WSGI response callable if not.
Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
Get groups for the given token.
Parameters: |
|
---|---|
Returns: | None if the token is invalid or a string containing a comma separated list of groups the authenticated user is a member of. The first group in the list is also considered a unique identifier for that user. |
WSGI entry point for auth requests (ones that match the self.auth_prefix). Wraps env in webob.Request object and passes it down.
Parameters: |
|
---|
Handles the various request for token and service end point(s) calls. There are various formats to support the various auth servers in the past. Examples:
GET <auth-prefix>/v1/<act>/auth
X-Auth-User: <act>:<usr> or X-Storage-User: <usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
GET <auth-prefix>/auth
X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
GET <auth-prefix>/v1.0
X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
On successful authentication, the response will have X-Auth-Token and X-Storage-Token set to the token to use with Swift and X-Storage-URL set to the URL to the default Swift cluster to use.
Parameters: | req – The webob.Request to process. |
---|---|
Returns: | webob.Response, 2xx on success with data set as explained above. |
Entry point for auth requests (ones that match the self.auth_prefix). Should return a WSGI-style callable (such as webob.Response).
Parameters: | req – webob.Request object |
---|
Returns a WSGI filter app for use with paste.deploy.
Returns a cleaned ACL header value, validating that it meets the formatting requirements for standard Swift ACL strings.
The ACL format is:
[item[,item...]]
Each item can be a group name to give access to or a referrer designation to grant or deny based on the HTTP Referer header.
The referrer designation format is:
.r:[-]value
The .r can also be .ref, .referer, or .referrer; though it will be shortened to just .r for decreased character count usage.
The value can be * to specify any referrer host is allowed access, a specific host name like www.example.com, or if it has a leading period . or leading *. it is a domain name specification, like .example.com or *.example.com. The leading minus sign - indicates referrer hosts that should be denied access.
Referrer access is applied in the order they are specified. For example, .r:.example.com,.r:-thief.example.com would allow all hosts ending with .example.com except for the specific host thief.example.com.
Example valid ACLs:
.r:*
.r:*,.r:-.thief.com
.r:*,.r:.example.com,.r:-thief.example.com
.r:*,.r:-.thief.com,bobs_account,sues_account:sue
bobs_account,sues_account:sue
Example invalid ACLs:
.r:
.r:-
By default, allowing read access via .r will not allow listing objects in the container – just retrieving objects from the container. To turn on listings, use the .rlistings directive.
Also, .r designations aren’t allowed in headers whose names include the word ‘write’.
ACLs that are “messy” will be cleaned up. Examples:
Original | Cleaned |
bob, sue | bob,sue |
bob , sue | bob,sue |
bob,,,sue | bob,sue |
.referrer : * | .r:* |
.ref:*.example.com | .r:.example.com |
.r:*, .rlistings | .r:*,.rlistings |
Parameters: |
|
---|---|
Returns: | The value, cleaned of extraneous formatting. |
Raises ValueError: | |
If the value does not meet the ACL formatting requirements; the error message will indicate why. |
Parses a standard Swift ACL string into a referrers list and groups list.
See clean_acl() for documentation of the standard Swift ACL format.
Parameters: | acl_string – The standard Swift ACL string to parse. |
---|---|
Returns: | A tuple of (referrers, groups) where referrers is a list of referrer designations (without the leading .r:) and groups is a list of groups to allow access. |
Returns True if the referrer should be allowed based on the referrer_acl list (as returned by parse_acl()).
See clean_acl() for documentation of the standard Swift ACL format.
Parameters: |
|
---|---|
Returns: | True if the referrer should be allowed; False if not. |
WSGI tools for use with swift.
Bases: object
This class provides a means to provide context (scope) for a middleware filter to have access to the wsgi start_response results like the request status and headers.
Bind socket to bind ip:port in conf
Parameters: |
|
---|
Returns a new fresh WSGI environment with escalated privileges to do backend checks, listings, etc. that the remote user wouldn’t be able to accomplish directly.
Parameters: |
|
---|---|
Returns: | Fresh WSGI environment. |
Makes a new webob.Request based on the current env but with the parameters specified. Note that this request will be preauthorized.
Parameters: |
|
---|---|
Returns: | Fresh webob.Request object. |
mimetools.Message defaults content-type to “text/plain” This changes it to default to None, so we can detect missing headers.
Loads common settings from conf, then instantiates app and runs the server using the specified number of workers.
Parameters: |
|
---|
Internal client library for making calls directly to the servers rather than through the proxy.
Delete object directly from the object server.
Parameters: |
|
---|---|
Returns: | response from server |
Get listings directly from the account server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase. |
Get container listings directly from the container server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase. |
Get object directly from the object server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase. |
Request container information directly from the container server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Request object information directly from the object server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Direct update to object metadata on object server.
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP POST request failed |
Put object directly from the object server.
Parameters: |
|
---|---|
Returns: | etag from the server response |
Helper function to retry a given function a number of times.
Parameters: |
|
---|---|
Returns: | restult of func |
Bases: object
Wrapper for file object to compress object while reading.
Can be used to wrap file objects passed to InternalClient.upload_object().
Used in testing of InternalClient.
Parameters: |
|
---|
Reads a chunk from the file object.
Params are passed directly to the underlying file object’s read().
Returns: | Compressed chunk from file object. |
---|
Bases: object
An internal client that uses a swift proxy app to make requests to Swift.
This client will exponentially slow down for retries.
Parameters: |
|
---|
Checks to see if a container exists.
Parameters: |
|
---|
:returns : True if container exists, false otherwise.
Raises: |
|
---|
Creates container.
Parameters: |
|
---|---|
Raises: |
|
Deletes a container.
Parameters: |
|
---|---|
Raises: |
|
Deletes an object.
Parameters: |
|
---|---|
Raises: |
|
Returns (container_count, object_count) for an account.
Parameters: |
|
---|---|
Raises: |
|
Gets account metadata.
Parameters: |
|
---|
:returns : Returns dict of account metadata.
Raises: |
|
---|
Gets container metadata.
Parameters: |
|
---|
:returns : Returns dict of container metadata.
Raises: |
|
---|
Gets object metadata.
Parameters: |
|
---|
:returns : Dict of object metadata.
Raises: |
|
---|
Returns an iterator of containers dicts from an account.
Parameters: |
|
---|---|
Raises: |
|
Returns an iterator of object lines from an uncompressed or compressed text object.
Uncompress object as it is read if the object’s name ends with ‘.gz’.
Parameters: |
|
---|---|
Raises: |
|
Returns an iterator of object dicts from a container.
Parameters: |
|
---|---|
Raises: |
|
Makes a request to Swift with retries.
Parameters: |
|
---|
:returns : Response object on success.
Raises: |
|
---|
Sets account metadata. A call to this will add to the account metadata and not overwrite all of it with values in the metadata dict. To clear an account metadata value, pass an empty string as the value for the key in the metadata dict.
Parameters: |
|
---|---|
Raises: |
|
Sets container metadata. A call to this will add to the container metadata and not overwrite all of it with values in the metadata dict. To clear a container metadata value, pass an empty string as the value for the key in the metadata dict.
Parameters: |
|
---|---|
Raises: |
|
Sets an object’s metadata. The object’s metadata will be overwritten by the values in the metadata dict.
Parameters: |
|
---|---|
Raises: |
|
Parameters: |
|
---|---|
Raises: |
|
Bases: exceptions.Exception
Exception raised on invalid responses to InternalClient.make_request().
Parameters: |
|
---|
Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.
Warning
If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.
Bases: httplib.HTTPConnection
HTTPConnection class that uses BufferedHTTPResponse
alias of BufferedHTTPResponse
Bases: httplib.HTTPResponse
HTTPResponse class that buffers reading of headers
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
Bases: object
Recon middleware used for monitoring.
/recon/load|mem|async... will return various system metrics.
Needs to be added to the pipeline and a requires a filter declaration in the object-server.conf:
[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift
get # of async pendings
get auditor info
get devices
get disk utilization statistics
get expirer info
get info from /proc/loadavg
get info from /proc/meminfo
get ALL mounted fs from /proc/mounts
get obj/container/account quarantine counts
get replication info
get all ring md5sum’s
get info from /proc/net/sockstat and sockstat6
Note: The mem value is actually kernel pages, but we return bytes allocated based on the systems page size.
list unmounted (failed?) devices
get updater info
Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:
http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt
Bases: object
Simple, consistent-hashed memcache client.
Decrements a key which has a numeric value by delta. Calls incr with -delta.
Parameters: |
|
---|---|
Raises MemcacheConnectionError: | |
Deletes a key/value pair from memcache.
Parameters: | key – key to be deleted |
---|
Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.
Parameters: | key – key |
---|---|
Returns: | value of the key in memcache |
Gets multiple values from memcache for the given keys.
Parameters: |
|
---|---|
Returns: | list of values |
Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.
Parameters: |
|
---|---|
Raises MemcacheConnectionError: | |
Set a key/value pair in memcache
Parameters: |
|
---|
Sets multiple key/value pairs in memcache.
Parameters: |
|
---|
Main class for performing commands on groups of servers.
Parameters: | servers – list of server names as strings |
---|
alias for reload
Find and return the decorated method named like cmd
Parameters: | cmd – the command to get, a string, if not found raises UnknownCommandError |
---|
Get all publicly accessible commands
Returns: | a list of string tuples (cmd, help), the method names who are decorated as commands |
---|
start a server interactively
spawn server and return immediately
start server and run one pass on supporting daemons
graceful shutdown then restart on supporting servers
stops then restarts server
Find the named command and run it
Parameters: | cmd – the command name to run |
---|
allow current requests to finish on supporting servers
starts a server
display status of tracked pids for server
stops a server
Manage operations on a server or group of servers of similar type
Parameters: | server – name of server |
---|
Get conf files for this server
Param : | number, if supplied will only lookup the nth server |
---|---|
Returns: | list of conf files |
Translate pid_file to a corresponding conf_file
Parameters: | pid_file – a pid_file for this server, a string |
---|---|
Returns: | the conf_file for this pid_file |
Translate conf_file to a corresponding pid_file
Parameters: | conf_file – an conf_file for this server, a string |
---|---|
Returns: | the pid_file for this conf_file |
Get running pids
Returns: | a dict mapping pids (ints) to pid_files (paths) |
---|
wait on spawned procs to terminate
Generator, yields (pid_file, pids)
Kill running pids
Parameters: | graceful – if True, attempt SIGHUP on supporting servers |
---|---|
Returns: | a dict mapping pids (ints) to pid_files (paths) |
Collect conf files and attempt to spawn the processes for this server
Get pid files for this server
Param : | number, if supplied will only lookup the nth server |
---|---|
Returns: | list of pid files |
Send a signal to pids for this server
Parameters: | sig – signal to send |
---|---|
Returns: | a dict mapping pids (ints) to pid_files (paths) |
Launch a subprocess for this server.
Parameters: |
|
---|
:returns : the pid of the spawned process
Display status of server
Param : | pids, if not supplied pids will be populated automatically |
---|---|
Param : | number, if supplied will only lookup the nth server |
Returns: | 1 if server is not running, 0 otherwise |
Send stop signals to pids for this server
Returns: | a dict mapping pids (ints) to pid_files (paths) |
---|
wait on spawned procs to start
Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.
Parameters: | func – function to make public |
---|
Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp
Monitor a collection of server pids yeilding back those pids that aren’t responding to signals.
Parameters: | server_pids – a dict, lists of pids [int,...] keyed on Server objects |
---|
Bases: object
Rate limiting middleware
Rate limits requests on both an Account and Container level. Limits are configurable.
Returns number of requests allowed per second for given container size.
Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.
Parameters: |
|
---|
Performs rate limiting and account white/black listing. Sleeps if necessary.
Parameters: |
|
---|
paste.deploy app factory for creating WSGI proxy apps.
This StaticWeb WSGI middleware will serve container data as a static web site with index file and error file resolution and optional file listings. This mode is normally only active for anonymous requests. If you want to use it with authenticated requests, set the X-Web-Mode: true header on the request.
The staticweb filter should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. Also, the configuration section for the staticweb middleware itself needs to be added. For example:
[DEFAULT]
...
[pipeline:main]
pipeline = healthcheck cache tempauth staticweb proxy-server
...
[filter:staticweb]
use = egg:swift#staticweb
# Seconds to cache container x-container-meta-web-* header values.
# cache_timeout = 300
# You can override the default log routing for this filter here:
# set log_name = staticweb
# set log_facility = LOG_LOCAL0
# set log_level = INFO
# set access_log_name = staticweb
# set access_log_facility = LOG_LOCAL0
# set access_log_level = INFO
# set log_headers = False
Any publicly readable containers (for example, X-Container-Read: .r:*, see acls for more information on this) will be checked for X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values:
X-Container-Meta-Web-Index <index.name>
X-Container-Meta-Web-Error <error.name.suffix>
If X-Container-Meta-Web-Index is set, any <index.name> files will be served without having to specify the <index.name> part. For instance, setting X-Container-Meta-Web-Index: index.html will be able to serve the object .../pseudo/path/index.html with just .../pseudo/path or .../pseudo/path/
If X-Container-Meta-Web-Error is set, any errors (currently just 401 Unauthorized and 404 Not Found) will instead serve the .../<status.code><error.name.suffix> object. For instance, setting X-Container-Meta-Web-Error: error.html will serve .../404error.html for requests for paths not found.
For psuedo paths that have no <index.name>, this middleware can serve HTML file listings if you set the X-Container-Meta-Web-Listings: true metadata item on the container.
If listings are enabled, the listings can have a custom style sheet by setting the X-Container-Meta-Web-Listings-CSS header. For instance, setting X-Container-Meta-Web-Listings-CSS: listing.css will make listings link to the .../listing.css style sheet. If you “view source” in your browser on a listing page, you will see the well defined document structure that can be styled.
Example usage of this middleware via swift:
Make the container publicly readable:
swift post -r '.r:*' containerYou should be able to get objects directly, but no index.html resolution or listings.
Set an index file directive:
swift post -m 'web-index:index.html' containerYou should be able to hit paths that have an index.html without needing to type the index.html part.
Turn on listings:
swift post -m 'web-listings: true' containerNow you should see object listings for paths and pseudo paths that have no index.html.
Enable a custom listings style sheet:
swift post -m 'web-listings-css:listings.css' containerSet an error file:
swift post -m 'web-error:error.html' containerNow 401’s should load 401error.html, 404’s should load 404error.html, etc.
Bases: object
The Static Web WSGI middleware filter; serves container data as a static web site. See staticweb for an overview.
Parameters: |
|
---|
Web access logger for this filter.
The next WSGI application/filter in the paste.deploy pipeline.
The seconds to cache the x-container-meta-web-* headers.,
The filter configuration dict.
Indicates whether full HTTP headers should be logged or not.
Logger for this filter.
Returns a Static Web WSGI filter for use with paste.deploy.
Patched version of urllib.quote that encodes utf-8 strings before quoting
TempURL Middleware
Allows the creation of URLs to provide temporary access to objects.
For example, a website may wish to provide a link to download a large object in Swift, but the Swift account has no public access. The website can generate a URL that will provide GET access for a limited time to the resource. When the web browser user clicks on the link, the browser will download the object directly from Swift, obviating the need for the website to act as a proxy for the request.
If the user were to share the link with all his friends, or accidentally post it on a forum, etc. the direct access would be limited to the expiration time set when the website created the link.
To create such temporary URLs, first an X-Account-Meta-Temp-URL-Key header must be set on the Swift account. Then, an HMAC-SHA1 (RFC 2104) signature is generated using the HTTP method to allow (GET or PUT), the Unix timestamp the access should be allowed until, the full path to the object, and the key set on the account.
For example, here is code generating the signature for a GET for 60 seconds on /v1/AUTH_account/container/object:
import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + 60)
path = '/v1/AUTH_account/container/object'
key = 'mykey'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
Be certain to use the full path, from the /v1/ onward.
Let’s say the sig ends up equaling da39a3ee5e6b4b0d3255bfef95601890afd80709 and expires ends up 1323479485. Then, for example, the website could provide a link to:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485
Any alteration of the resource path or query arguments would result in 401 Unauthorized. Similary, a PUT where GET was the allowed method would 401. HEAD is allowed if GET or PUT is allowed.
Using this in combination with browser form post translation middleware could also allow direct-from-browser uploads to specific locations in Swift.
Note that changing the X-Account-Meta-Temp-URL-Key will invalidate any previously generated temporary URLs within 60 seconds (the memcache time for the key).
Bases: object
WSGI Middleware to grant temporary URLs specific access to Swift resources. See the overview for more information.
This middleware understands the following configuration settings:
incoming_remove_headers
The headers to remove from incoming requests. Simply a
whitespace delimited list of header names and names can
optionally end with '*' to indicate a prefix match.
incoming_allow_headers is a list of exceptions to these
removals.
Default: x-timestamp
incoming_allow_headers
The headers allowed as exceptions to
incoming_remove_headers. Simply a whitespace delimited
list of header names and names can optionally end with
'*' to indicate a prefix match.
Default: None
outgoing_remove_headers
The headers to remove from outgoing responses. Simply a
whitespace delimited list of header names and names can
optionally end with '*' to indicate a prefix match.
outgoing_allow_headers is a list of exceptions to these
removals.
Default: x-object-meta-*
outgoing_allow_headers
The headers allowed as exceptions to
outgoing_remove_headers. Simply a whitespace delimited
list of header names and names can optionally end with
'*' to indicate a prefix match.
Default: x-object-meta-public-*
Parameters: |
|
---|
HTTP user agent to use for subrequests.
The next WSGI application/filter in the paste.deploy pipeline.
The filter configuration dict.
Headers to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY.
Header with match prefixes to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY_*.
Headers to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_PRIVATE.
Header with match prefixes to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_SENSITIVE_*.
The logger to use with this middleware.
Headers to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay.
Header with match prefixes to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay-*.
Headers to remove from outgoing responses. Lowercase, like x-account-meta-temp-url-key.
Header with match prefixes to remove from outgoing responses. Lowercase, like x-account-meta-private-*.
Returns the WSGI filter for use with paste.deploy.
Default headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_INCOMING_ALLOW_HEADERS is a list of exceptions to these removals.
Default headers as exceptions to DEFAULT_INCOMING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.
Default headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_OUTGOING_ALLOW_HEADERS is a list of exceptions to these removals.
Default headers as exceptions to DEFAULT_OUTGOING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.
FormPost Middleware
Translates a browser form post into a regular Swift object PUT.
The format of the form is:
<form action="<swift-url>" method="POST"
enctype="multipart/form-data">
<input type="hidden" name="redirect" value="<redirect-url>" />
<input type="hidden" name="max_file_size" value="<bytes>" />
<input type="hidden" name="max_file_count" value="<count>" />
<input type="hidden" name="expires" value="<unix-timestamp>" />
<input type="hidden" name="signature" value="<hmac>" />
<input type="file" name="file1" /><br />
<input type="submit" />
</form>
The <swift-url> is the URL to the Swift desination, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
The name of each file uploaded will be appended to the <swift-url> given. So, you can upload directly to the root of container with a url like:
https://swift-cluster.example.com/v1/AUTH_account/container/
Optionally, you can include an object prefix to better separate different users’ uploads, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
Note the form method must be POST and the enctype must be set as “multipart/form-data”.
The redirect attribute is the URL to redirect the browser to after the upload completes. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”).
The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes.
The max_file_count attribute must be included and indicates the maximum number of files that can be uploaded with the form. Include additional <input type="file" name="filexx" /> attributes if desired.
The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated.
The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature:
import hmac
from hashlib import sha1
from time import time
path = '/v1/account/container/object_prefix'
redirect = 'https://myserver.com/some-page'
max_file_size = 104857600
max_file_count = 10
expires = int(time() + 600)
key = 'mykey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()
The key is the value of the X-Account-Meta-Temp-URL-Key header on the account.
Be certain to use the full path, from the /v1/ onward.
The command line tool swift-form-signature may be used (mostly just when testing) to compute expires and signature.
Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).
Bases: object
FormPost Middleware
See above for a full description.
Parameters: |
|
---|
The HTTP user agent to use with subrequests.
The next WSGI application/filter in the paste.deploy pipeline.
The filter configuration dict.
The logger to use with this middleware.
Returns the WSGI filter for use with paste.deploy.
The size of data to read from the form at any given time.
The maximum size of any attribute’s value. Any additional data will be truncated.
Domain Remap Middleware
Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.
container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object
account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object
Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. The reseller_prefixes list is exclusive. If defined, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.
Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).
Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.
Bases: object
Domain Remap Middleware
See above for a full description.
Parameters: |
|
---|
CNAME Lookup Middleware
Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.
This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.
Bases: object
CNAME Lookup Middleware
See above for a full description.
Parameters: |
|
---|
Given a domain, returns its DNS CNAME mapping and DNS ttl.
Parameters: | domain – domain to query on |
---|---|
Returns: | (ttl, result) |
Logging middleware for the Swift proxy.
This serves as both the default logging implementation and an example of how to plug in your own logging format/method.
The logging format implemented below is as follows:
These values are space-separated, and each is url-encoded, so that they can be separated with a simple .split()
Bases: object
File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.
Pass read request to the underlying file-like object and add bytes read to total.
Pass readline request to the underlying file-like object and add bytes read to total.
Bases: object
Middleware that logs Swift proxy requests in the swift log format.
Log a request.
Parameters: |
|
---|