DRS Download Tools

Module for downloading and listing JSON DRS manifest and DRS objects. The main classes in this module for downloading DRS objects are DownloadManager and Manifest.

Examples:

This generates the Gen3Jobs class pointed at the sandbox commons while using the credentials.json downloaded from the commons profile page.

>>> datafiles = Manifest.load('sample/manifest_1.json')
    downloadManager = DownloadManager("source.my_commons.org",
                      Gen3Auth(refresh_file="~.gen3/my_credentials.json"), datafiles)
    for i in datafiles:
        print(i)
    downloadManager.download(datafiles, ".")

See docs/howto/drsDownloading.md for more details

class gen3.tools.download.drs_download.DownloadManager(hostname: str, auth: Gen3Auth, download_list: List[Downloadable], show_progress: bool = False, endpoint: str | None = None)

Class to assist in downloading a list of Downloadable object which at a minimum is a json manifest of DRS object ids. The methods of interest are download and user_access.

cache_hosts_wts_tokens(object_list)

Using the list of DRS host obtain a WTS token for all DRS hosts in the list. It’s is possible

download(object_list: List[Downloadable], save_directory: str = '.', show_progress: bool = False, unpack_packages: bool = True, delete_unpacked_packages: bool = False) → Dict[str, Any]

Downloads objects to the directory or current working directory. The input is an list of Downloadable object created by loading a manifest using the Manifest class or a call to Manifest.load(…

The download manager will download each file in the manifest, in the case of errors they are logged and it continues.

The return value is a list of DownloadStatus object, detailing the results of the download.

Parameters:

• object_list (List[Downloadable])

• save_directory (str) – directory to save to (will be created)

• show_progress (bool) – show a download progress bar

• unpack_packages (bool) – set to False to disable the unpacking of downloaded packages

• delete_unpacked_packages (bool) – set to True to delete package files after unpacking them

Returns:

List of DownloadStatus objects for each object id in object_list

get_fresh_token(drs_hostname: str) → str | None

Will return and/or refresh and return a WTS token if hostname is known otherwise returns None. :param drs_hostname: hostname to get token for :type drs_hostname: str

Returns:

access token if successful otherwise None

resolve_objects(object_list: List[Downloadable], show_progress: bool)

Given an Downloadable object list, resolve the DRS hostnames and update each Downloadable

Parameters:

object_list (List[Downloadable]) – list of Downloadable objects to resolve

user_access()

List the user’s access permissions on each host needed to download DRS objects in the manifest. A useful way to determine if access permissions are one reason a download failed.

Returns:

list of authz for each DRS host

class gen3.tools.download.drs_download.DownloadStatus(filename: str, status: str = 'pending', start_time: datetime | None = None, end_time: datetime | None = None)

Stores the download status of objectIDs.

The DataManager will return a list of DownloadStatus as a result of calling the download method

Status is “pending” until it is downloaded or an error occurs. .. attribute:: filename

the name of the file to download

type:

str

status

status of file download initially “pending”

Type:

str

start_time

start time of download as datetime initially None

Type:

Optional[datetime]

end_time

end time of download as datetime initially None

Type:

Optional[datetime]

class gen3.tools.download.drs_download.Downloadable(object_id: str, object_type: ~gen3.tools.download.drs_download.DRSObjectType | None = DRSObjectType.unknown, hostname: str | None = None, file_size: int | None = -1, file_name: str | None = None, updated_time: ~datetime.datetime | None = None, created_time: ~datetime.datetime | None = None, access_methods: ~typing.List[~typing.Dict[str, ~typing.Any]] = <factory>, children: ~typing.List[~gen3.tools.download.drs_download.Downloadable] = <factory>)

Class handling the information for a DRS object. The information is populated from the manifest or by retrieving the information from a DRS server.

object_id

DRS object id (REQUIRED)

Type:

str

object_type

type of DRS object

Type:

DRSObjectType

hostname

hostname of DRS object

Type:

str

file_size

size in bytes

Type:

int

file_name

name of file

Type:

str

updated_time

timestamp of last update to file

Type:

datetime

created_time

timestamp when file is created

Type:

datetime

access_methods

list of access methods (e.g. s3) for DRS object

Type:

List[Dict[str, Any]]

children

list of child objects (in the case of DRS bundles)

Type:

List[Downloadable]

_manager

manager for this Downloadable

Type:

DownloadManager

download()

calls the manager to download this object. Allows Downloadables to be self downloading

pprint(indent: str = '')

Pretty prints the object information. This is used for listing an object. In the case of a DRS bundle the child objects are listed similar to the linux tree command

class gen3.tools.download.drs_download.Manifest(object_id: str, file_size: int | None = -1, file_name: str | None = None, md5sum: str | None = None, commons_url: str | None = None)

Data class representing a Gen3 JSON manifest typically exported from a Gen3 discovery page.

The class is passed to the DownloadManager to download or list all of the files in the manifest. The Download manager will cache additional information (if available)

object_id

the DRS object id. This is the only attribute that needs to be defined

Type:

str

file_size

the filesize of the object, if contained in the manifest

Type:

Optional[int]

file_name

the name of the file pointed to by the DRS object id

Type:

Optional[str]

md5sum

the checksum of the object

Type:

Optional[str]

commons_url

url of the indexd server to retrieve file/bundle from

Type:

Optional[str]

static create_object_list(manifest) → List[Downloadable]

Create a list of Downloadable instances from the manifest

Parameters:

manifest (list) – list of manifest objects

Returns:

List of Downloadable instances

static load(filename: Path) → List[Downloadable] | None

Method to load a json manifest and return a list of Bownloadable object. This list is passed to the DownloadManager methods of download, and list

Parameters:

filename (Path) – path to manifest file

Returns:

list of Downloadable objects if successfully opened/parsed None otherwise

static load_manifest(path: Path)

Loads a json manifest

gen3.tools.download.drs_download.download_files_in_drs_manifest(hostname, auth, infile, output_dir, show_progress=True, unpack_packages=True, delete_unpacked_packages=False) → None

A convenience function used to download a json manifest. :param hostname: hostname of Gen3 commons to use for access and WTS :type hostname: str :param auth: Gen3 Auth instance :type auth: str :param infile: manifest file :type infile: str :param output_dir: directory to save downloaded files to :param unpack_packages: set to False to disable the unpacking of downloaded packages :type unpack_packages: bool :param delete_unpacked_packages: set to True to delete package files after unpacking them :type delete_unpacked_packages: bool

Returns:

gen3.tools.download.drs_download.list_access_in_drs_manifest(hostname, auth, infile) → bool

A convenience function to list a users access for all DRS hostname in a manifest. :param hostname: hostname of Gen3 commons to use for access and WTS :type hostname: str :param auth: Gen3 Auth instance :param infile: manifest file :type infile: str

Returns:

True if successfully listed

gen3.tools.download.drs_download.list_drs_object(hostname, auth, object_id: str) → bool

A convenience function used to list a DRS object. :param hostname: hostname of Gen3 commons to use for access and WTS :type hostname: str :param auth: Gen3 Auth instance :param object_id: DRS object :type object_id: str

Returns:

True if successfully listed

gen3.tools.download.drs_download.list_files_in_drs_manifest(hostname, auth, infile: str) → bool

A wrapper function used by the cli to list files in a manifest. :param hostname: hostname of Gen3 commons to use for access and WTS :type hostname: str :param auth: Gen3 Auth instance :param infile: manifest file :type infile: str

Returns:

True if successfully listed