metakb.harvesters.base#

A module for the Harvester base class

class metakb.harvesters.base.FetchMode(value, names=_not_given, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Define options for fetching data

CHECK_STALE = 'check_stale'[source]#

FORCE_REFRESH = 'force_refresh'[source]#

USE_LOCAL = 'use_local'[source]#

class metakb.harvesters.base.Harvester(src_data_dir)[source]#

A base class for content harvesters.

__init__(src_data_dir)[source]#

Initialize harvester class

Parameters:: src_data_dir (SourceDataStore) – container for MetaKB-managed data for this source

abstract harvest(fetch_mode=FetchMode.CHECK_STALE)[source]#

Grab data from a source and stash a copy locally, returning the stashed location

Parameters:: fetch_mode (FetchMode) – set data caching/fetching behavior. Not evenly used across sources.
Return type:: Path
Returns:: Location of performed data harvest