ahvn.klengine package

class ahvn.klengine.BaseKLEngine(storage=None, inplace=False, name=None, condition=None, *args, **kwargs)

基类：ToolRegistry, ABC

An abstract base class for KLEngine implementations.

This class provides a protocol for indexing/searching Knowledge items. Each class should provide a list of retrieval methods with prefix _search_, which will be viewed together as a toolkit of this engine.

Subclasses must implement the abstract methods for upsert, and may override batch and existence-checking methods for performance optimization.

Search Methods:

Subclasses can implement multiple search methods: - _search(include, *args, **kwargs): Default search method (required). - _search_xxx(include, *args, **kwargs): Named search method for mode "xxx". Use search(mode="xxx") to route to the corresponding _search_xxx method.

Abstract Methods:

_search(include, *args, **kwargs): Perform a search for KLs in the engine. _upsert(kl): Insert or update a KL in the engine. _remove(key): Remove a KL from the engine by its key (id). If not applicable, override it with an empty function or an exception. _clear(): Clear all KLs from the store.

Optional Methods:

_get(kl): Retrieve a KL from the engine.

Though not required, leaving _get unimplemented may lead to unexpected behavior. This is recommended if kl should be returned by search and there is no KLStore attached.

_post_search(results: List[BaseUKF]) -> List[BaseUKF]:

Postprocessing for search results. By default, it returns the results unchanged.

参数:

• storage (BaseKLStore | None)

• inplace (bool)

• name (str | None)

• condition (Callable | None)

__init__(storage=None, inplace=False, name=None, condition=None, *args, **kwargs)

Initialization.

参数:

• storage (Optional[BaseKLStore]) -- attach KLEngine to a KLStore. When attached, the engine will try to retrieve KLs from the store if not found in the engine itself. Notice that though attached, operations on the engine will never change the actual KLs in the store. Notice that one KLEngine can be attached to only one KLStore at a time, to support multiple KLStores, use RouterKLStore or CascadeKLStore.

• inplace (Optional[bool]) -- Whether the engine is in-place. When inplace is True, the engine does not hold KLs itself, but only provides get capabilities over an attached KLStore. When inplace is True, storage must be a DatabaseKLStore instance.

• name (Optional[str]) -- Name of the KLEngine instance. If None, defaults to "default".

• condition (Optional[Callable]) -- Optional upsert/insert condition to apply to the KLEngine. KLs that do not satisfy the condition will be ignored. If None, all KLs are accepted.

• *args -- Additional positional arguments.

• **kwargs -- Additional keyword arguments.

inplace: bool = False

attach(storage)

Attach KLEngine to a KLStore. When attached, the engine will try to retrieve KLs from the store if not found in the engine itself. Notice that though attached, operations on the engine will never change the actual KLs in the store. Notice that one KLEngine can be attached to only one KLStore at a time, to support multiple KLStores, use RouterKLStore or CascadeKLStore.

参数:

storage (BaseKLStore) -- The KLStore to attach.

detach()

Detach KLEngine from its attached KLStore.

get(key, default=..., storage=None)

Retrieves a KL by its key. By default it tries to get the KL from the engine itself. When a KLStore is attached, it will try to get the KL from the store if not found in the engine.

参数:

• key (Union[int, str, BaseUKF]) -- The key or BaseUKF instance to retrieve.

• default (Any) -- The default value to return if the KL is not found.

• storage (bool) -- Whether to retrieve the KL from the storage. Default is None. If storage is True, it will always try to get the KL from the store, and bypass the engine. If storage is False, it will only try to get the KL from the engine, and if not found, it will not try to get the KL from the store. If storage is None, it will prioritize the engine over the store.

返回:

The retrieved KL instance if found. Otherwise default.

返回类型:

BaseUKF

__contains__(key)

Check if a key exists in the engine.

upsert(kl, **kwargs)

Upsert a KL.

参数:

• kl (BaseUKF) -- The KL to upsert.

• kwargs -- Additional keyword arguments.

insert(kl, **kwargs)

Insert a KL.

参数:

• kl (BaseUKF) -- The KL to insert.

• kwargs -- Additional keyword arguments.

batch_upsert(kls, progress=None, **kwargs)

Upsert multiple KLs. The default batch upsert is not optimized nor parallelized. It is recommended to override this method for better performance.

参数:

• kls (Iterable[BaseUKF]) -- The KLs to upsert.

• progress (Type[Progress]) -- Progress class for reporting. None for silent, TqdmProgress for terminal.

• kwargs -- Additional keyword arguments.

batch_insert(kls, progress=None, **kwargs)

Insert multiple KLs. The default batch insert first checks for existing keys and then batch upserts. When overriding batch_upsert, batch insert is automatically optimized. Nevertheless, the existence check uses _has, which is by default not optimized. It is recommended to override batch_insert or _has for better performance.

参数:

• kls (Iterable[BaseUKF]) -- The KLs to insert.

• progress (Type[Progress]) -- Progress class for reporting. None for silent, TqdmProgress for terminal.

• kwargs -- Additional keyword arguments.

__delitem__(key, **kwargs)

Removes a KL from the engine.

参数:

• key (Union[int, str, BaseUKF]) -- The key or BaseUKF instance to remove.

• **kwargs -- Additional keyword arguments.

remove(key, conditioned=True, **kwargs)

Removes a KL from the engine.

参数:

• key (Union[int, str, BaseUKF]) -- The key or BaseUKF instance to remove.

• conditioned (bool) -- Remove only if the KL satisfies the engine's condition. Default is True. Notice that conditioned removal only applies when passing a BaseUKF instance as key.

• **kwargs -- Additional keyword arguments.

batch_remove(keys, conditioned=True, progress=None, **kwargs)

Removes multiple KLs from the engine.

参数:

• keys (Iterable[Union[int, str, BaseUKF]]) -- The keys or BaseUKF instances to remove.

• conditioned (bool) -- Remove only if the KLs satisfy the engine's condition. Default is True. Notice that conditioned removal only applies when passing BaseUKF instances in keys.

• progress (Type[Progress]) -- Progress class for reporting. None for silent, TqdmProgress for terminal.

• **kwargs -- Additional keyword arguments.

clear()

Clears the engine.

close()

Closes the engine, if applicable.

flush()

Flushes the engine, if applicable.

sync(batch_size=None, progress=None, **kwargs)

Synchronize KLEngine with its attached KLStore, if applicable. Notice that a whole synchronization can often lead to large data upload/download. This could result in performance issues and even errors for particular backends. Therefore, parameters like batch_size are provided to control the synchronization process. It is recommended to override this method for better performance.

参数:

• batch_size (Optional[int]) -- The batch size for synchronization. If None, use the default batch size from configuration (512). If <= 0, yields all KLs in a single batch.

• progress (Type[Progress]) -- Progress class for reporting. None for silent, TqdmProgress for terminal.

• **kwargs -- Additional keyword arguments.

list_search()

List all available search methods.

返回:

A list of search method names. None represents the default search method.

返回类型:

List[Optional[str]]

search(*args, include=None, mode=None, **kwargs)

Perform a search operation on the engine, return the KLs with keys limited to include. Conventionally, it ir recommended use id to return BaseUKF.id, and kl to return BaseUKF itself.

参数:

• include (Optional[Iterable[str]]) -- The keys to include in the search results. Defaults to None, which includes at least 'id' and 'kl'.

• mode (Optional[str]) -- The search method mode to use. None uses the default _search method.

• *args -- The positional arguments to pass to the search.

• **kwargs -- The keyword arguments to pass to the search.

返回:

The search results.

返回类型:

List[Dict[str, Any]]

Submodules

• ahvn.klengine.base module
  • BaseKLEngine
    • BaseKLEngine.__init__()
    • BaseKLEngine.inplace
    • BaseKLEngine.attach()
    • BaseKLEngine.detach()
    • BaseKLEngine.get()
    • BaseKLEngine.__contains__()
    • BaseKLEngine.upsert()
    • BaseKLEngine.insert()
    • BaseKLEngine.batch_upsert()
    • BaseKLEngine.batch_insert()
    • BaseKLEngine.__delitem__()
    • BaseKLEngine.remove()
    • BaseKLEngine.batch_remove()
    • BaseKLEngine.clear()
    • BaseKLEngine.close()
    • BaseKLEngine.flush()
    • BaseKLEngine.sync()
    • BaseKLEngine.list_search()
    • BaseKLEngine.search()
• ahvn.klengine.daac_engine module
  • DAACKLEngine
    • DAACKLEngine.inplace
    • DAACKLEngine.recoverable
    • DAACKLEngine.__init__()
    • DAACKLEngine.__len__()
    • DAACKLEngine.clear()
    • DAACKLEngine.flush()
    • DAACKLEngine.sync()
    • DAACKLEngine.save()
    • DAACKLEngine.load()
    • DAACKLEngine.close()
• ahvn.klengine.facet_engine module
  • FacetKLEngine
    • FacetKLEngine.inplace
    • FacetKLEngine.__init__()
    • FacetKLEngine.recoverable
    • FacetKLEngine.__len__()
    • FacetKLEngine.close()
• ahvn.klengine.mongo_engine module
  • MongoKLEngine
    • MongoKLEngine.inplace
    • MongoKLEngine.__init__()
    • MongoKLEngine.recoverable
    • MongoKLEngine.k_encode()
    • MongoKLEngine.k_embed()
    • MongoKLEngine.batch_k_encode()
    • MongoKLEngine.batch_k_embed()
    • MongoKLEngine.q_encode()
    • MongoKLEngine.batch_q_encode()
    • MongoKLEngine.q_embed()
    • MongoKLEngine.batch_q_embed()
    • MongoKLEngine.k_encode_embed()
    • MongoKLEngine.batch_k_encode_embed()
    • MongoKLEngine.q_encode_embed()
    • MongoKLEngine.batch_q_encode_embed()
    • MongoKLEngine.close()
• ahvn.klengine.scan_engine module
  • ScanKLEngine
    • ScanKLEngine.inplace
    • ScanKLEngine.recoverable
    • ScanKLEngine.__init__()
    • ScanKLEngine.__len__()
    • ScanKLEngine.__iter__()
• ahvn.klengine.vector_engine module
  • VectorKLEngine
    • VectorKLEngine.inplace
    • VectorKLEngine.__init__()
    • VectorKLEngine.recoverable
    • VectorKLEngine.close()
    • VectorKLEngine.k_encode()
    • VectorKLEngine.k_embed()
    • VectorKLEngine.batch_k_encode()
    • VectorKLEngine.batch_k_embed()
    • VectorKLEngine.q_encode()
    • VectorKLEngine.q_embed()
    • VectorKLEngine.batch_q_encode()
    • VectorKLEngine.batch_q_embed()
    • VectorKLEngine.k_encode_embed()
    • VectorKLEngine.batch_k_encode_embed()
    • VectorKLEngine.q_encode_embed()
    • VectorKLEngine.batch_q_encode_embed()
    • VectorKLEngine.embedding_field
    • VectorKLEngine.adapter