Table Of Contents

Previous topic

Container

Next topic

Object

This Page

Psst... hey. Did you know you can read about Swift at docs.openstack.org also? Plus you can get to past versions atSwift 1.3 docs and Swift 1.2 docs.

Account DB and Container DB

DB

Database code for Swift

class swift.common.db.AccountBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a account database.

can_delete_db(cutoff)

Check if the accont DB can be deleted.

Returns:True if the account can be deleted, False otherwise
create_account_stat_table(conn, put_timestamp)

Create account_stat table which is specific to the account DB.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
create_container_table(conn)

Create container table which is specific to the account DB.

Parameters:conn – DB connection object
db_contains_type = 'container'
db_type = 'account'
empty()

Check if the account DB is empty.

Returns:True if the database has no active containers.
get_container_timestamp(container_name)

Get the put_timestamp of a container.

Parameters:container_name – container name
Returns:put_timestamp of the container
get_db_version(conn)
get_info()

Get global data for the account.

Returns:dict with keys: account, created_at, put_timestamp, delete_timestamp, container_count, object_count, bytes_used, hash, id
is_deleted()

Check if the account DB is considered to be deleted.

Returns:True if the account DB is considered to be deleted, False otherwise
is_status_deleted()

Only returns true if the status field is set to DELETED.

list_containers_iter(limit, marker, end_marker, prefix, delimiter)

Get a list of containerss sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • end_marker – end marker query
  • prefix – prefix query
  • delimeter – delimeter for query
Returns:

list of tuples of (name, object_count, bytes_used, 0)

merge_items(item_list, source=None)

Merge items into the container table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘put_timestamp’, ‘delete_timestamp’, ‘object_count’, ‘bytes_used’, ‘deleted’}
  • source – if defined, update incoming_sync with the source
put_container(name, put_timestamp, delete_timestamp, object_count, bytes_used)

Create a container with the given attributes.

Parameters:
  • name – name of the container to create
  • put_timestamp – put_timestamp of the container to create
  • delete_timestamp – delete_timestamp of the container to create
  • object_count – number of objects in the container
  • bytes_used – number of bytes used by the container
reclaim(container_timestamp, sync_timestamp)

Delete rows from the container table that are marked deleted and whose created_at timestamp is < container_timestamp. Also deletes rows from incoming_sync and outgoing_sync where the updated_at timestamp is < sync_timestamp.

In addition, this calls the DatabaseBroker’s :func:_reclaim method.

Parameters:
  • container_timestamp – max created_at timestamp of container rows to delete
  • sync_timestamp – max update_at timestamp of sync rows to delete
update_put_timestamp(timestamp)

Update the put_timestamp. Only modifies it if it is greater than the current timestamp.

Parameters:timestamp – put timestamp
swift.common.db.BROKER_TIMEOUT = 25

Timeout for trying to connect to a DB

class swift.common.db.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a container database.

create_container_stat_table(conn, put_timestamp=None)

Create the container_stat table which is specific to the container DB.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
create_object_table(conn)

Create the object table which is specifc to the container DB.

Parameters:conn – DB connection object
db_contains_type = 'object'
db_type = 'container'
delete_object(name, timestamp)

Mark an object deleted.

Parameters:
  • name – object name to be deleted
  • timestamp – timestamp when the object was marked as deleted
empty()

Check if the DB is empty.

Returns:True if the database has no active objects, False otherwise
get_db_version(conn)
get_info(include_metadata=False)

Get global data for the container.

Returns:dict with keys: account, container, created_at, put_timestamp, delete_timestamp, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, and x_container_sync_point2. If include_metadata is set, metadata is included as a key pointing to a dict of tuples of the metadata
is_deleted(timestamp=None)

Check if the DB is considered to be deleted.

Returns:True if the DB is considered to be deleted, False otherwise
list_objects_iter(limit, marker, end_marker, prefix, delimiter, path=None)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • end_marker – end marker query
  • prefix – prefix query
  • delimeter – delimeter for query
  • path – if defined, will set the prefix and delimter based on the path
Returns:

list of tuples of (name, created_at, size, content_type, etag)

merge_items(item_list, source=None)

Merge items into the object table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’}
  • source – if defined, update incoming_sync with the source
put_object(name, timestamp, size, content_type, etag, deleted=0)

Creates an object in the DB with its metadata.

Parameters:
  • name – object name to be created
  • timestamp – timestamp of when the object was created
  • size – object size
  • content_type – object content-type
  • etag – object etag
  • deleted – if True, marks the object as deleted and sets the deteleted_at timestamp to timestamp
reclaim(object_timestamp, sync_timestamp)

Delete rows from the object table that are marked deleted and whose created_at timestamp is < object_timestamp. Also deletes rows from incoming_sync and outgoing_sync where the updated_at timestamp is < sync_timestamp.

In addition, this calls the DatabaseBroker’s :func:_reclaim method.

Parameters:
  • object_timestamp – max created_at timestamp of object rows to delete
  • sync_timestamp – max update_at timestamp of sync rows to delete
reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats.

Parameters:
  • put_timestamp – put_timestamp to update
  • delete_timestamp – delete_timestamp to update
  • object_count – object_count to update
  • bytes_used – bytes_used to update
set_x_container_sync_points(sync_point1, sync_point2)
update_put_timestamp(timestamp)

Update the put_timestamp. Only modifies it if it is greater than the current timestamp.

Parameters:timestamp – put timestamp
swift.common.db.DB_PREALLOCATION = True

Whether calls will be made to preallocate disk space for database files.

class swift.common.db.DatabaseBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=10, stale_reads_ok=False)

Bases: object

Encapsulates working with a database.

delete_db(timestamp)

Mark the DB as deleted

Parameters:timestamp – delete timestamp
get(*args, **kwds)

Use with the “with” statement; returns a database connection.

get_items_since(start, count)

Get a list of objects in the database between start and end.

Parameters:
  • start – start ROWID
  • count – number to get
Returns:

list of objects between start and end

get_replication_info()

Get information about the DB required for replication.

Returns:dict containing keys: hash, id, created_at, put_timestamp, delete_timestamp, count, max_row, and metadata
get_sync(id, incoming=True)

Gets the most recent sync point for a server from the sync table.

Parameters:
  • id – remote ID to get the sync_point for
  • incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
Returns:

the sync point, or -1 if the id doesn’t exist.

get_syncs(incoming=True)

Get a serialized copy of the sync table.

Parameters:incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
Returns:list of {‘remote_id’, ‘sync_point’}
initialize(put_timestamp=None)

Create the DB

Parameters:put_timestamp – timestamp of initial PUT request
lock(*args, **kwds)

Use with the “with” statement; locks a database.

merge_syncs(sync_points, incoming=True)

Merge a list of sync points with the incoming sync table.

Parameters:
  • sync_points – list of sync points where a sync point is a dict of {‘sync_point’, ‘remote_id’}
  • incoming – if True, get the last incoming sync, otherwise get the last outgoing sync
merge_timestamps(created_at, put_timestamp, delete_timestamp)

Used in replication to handle updating timestamps.

Parameters:
  • created_at – create timestamp
  • put_timestamp – put timestamp
  • delete_timestamp – delete timestamp
metadata

Returns the metadata dict for the database. The metadata dict values are tuples of (value, timestamp) where the timestamp indicates when that key was set to that value.

newid(remote_id)

Re-id the database. This should be called after an rsync.

Parameters:remote_id – the ID of the remote database being rsynced in
possibly_quarantine(exc_type, exc_value, exc_traceback)

Checks the exception info to see if it indicates a quarantine situation (malformed or corrupted database). If not, the original exception will be reraised. If so, the database will be quarantined and a new sqlite3.DatabaseError will be raised indicating the action taken.

reclaim(timestamp)

Removes any empty metadata values older than the timestamp

update_metadata(metadata_updates)

Updates the metadata dict for the database. The metadata dict values are tuples of (value, timestamp) where the timestamp indicates when that key was set to that value. Key/values will only be overwritten if the timestamp is newer. To delete a key, set its value to (‘’, timestamp). These empty keys will eventually be removed by :func:reclaim

exception swift.common.db.DatabaseConnectionError(path, msg, timeout=0)

Bases: sqlite3.DatabaseError

More friendly error messages for DB Errors.

class swift.common.db.GreenDBConnection(*args, **kwargs)

Bases: sqlite3.Connection

SQLite DB Connection handler that plays well with eventlet.

commit()
execute(*args, **kwargs)
swift.common.db.PENDING_CAP = 131072

Max number of pending entries

swift.common.db.PICKLE_PROTOCOL = 2

Pickle protocol to use

swift.common.db.chexor(old, name, timestamp)

Each entry in the account and container databases is XORed by the 128-bit hash on insert or delete. This serves as a rolling, order-independent hash of the contents. (check + XOR)

Parameters:
  • old – hex representation of the current DB hash
  • name – name of the object or container being inserted
  • timestamp – timestamp of the new record
Returns:

a hex representation of the new hash value

swift.common.db.dict_factory(crs, row)

This should only be used when you need a real dict, i.e. when you’re going to serialize the results.

swift.common.db.get_db_connection(path, timeout=30, okay_to_create=False)

Returns a properly configured SQLite database connection.

Parameters:
  • path – path to DB
  • timeout – timeout for connection
  • okay_to_create – if True, create the DB if it doesn’t exist
Returns:

DB connection object

swift.common.db.utf8encode(*args)

DB replicator

class swift.common.db_replicator.ReplConnection(node, partition, hash_, logger)

Bases: swift.common.bufferedhttp.BufferedHTTPConnection

Helper to simplify REPLICATEing to a remote server.

replicate(*args)

Make an HTTP REPLICATE request

Parameters:args – list of json-encodable objects
Returns:httplib response object
class swift.common.db_replicator.Replicator(conf)

Bases: swift.common.daemon.Daemon

Implements the logic for directing db replication.

delete_db(object_file)
extract_device(object_file)

Extract the device name from an object path. Returns “UNKNOWN” if the path could not be extracted successfully for some reason.

Parameters:object_file – the path to a database file.
report_up_to_date(full_info)
roundrobin_datadirs(datadirs)

Generator to walk the data dirs in a round robin manner, evenly hitting each device on the system.

Parameters:datadirs – a list of paths to walk
run_forever(*args, **kwargs)

Replicate dbs under the given root in an infinite loop.

run_once(*args, **kwargs)

Run a replication pass once.

class swift.common.db_replicator.ReplicatorRpc(root, datadir, broker_class, mount_check=True, logger=None)

Bases: object

Handle Replication RPC calls. TODO(redbo): document please :)

complete_rsync(drive, db_file, args)
dispatch(replicate_args, args)
merge_items(broker, args)
merge_syncs(broker, args)
rsync_then_merge(drive, db_file, args)
sync(broker, args)
swift.common.db_replicator.quarantine_db(object_file, server_type)

In the case that a corrupt file is found, move it to a quarantined area to allow replication to fix it.

Parameters:
  • object_file – path to corrupt file
  • server_type – type of file that is corrupt (‘container’ or ‘account’)