Modules

Find the module that fits your needs!

ETL Modules

Currently, about 50 functions are registered as the ETL process, which means they are eagerly awaiting your use!

By clicking module name, you can directly go to corresponding API reference.

Type	Package/Modules	Description
Extract	data_ingestion	Loading data from any source to the preferred format
Transform	bias	(WIP) Reduce skewed or prejudiced data, particularly data that reinforce stereotypes.
	cleaning	Remove irrelevant, redundant, or noisy information, such as stop words or special characters.
	decontamination	(WIP) Remove contaminated data including benchmark.
	deduplication	Remove duplicated data, targeting not only identical matches but also similar data.
	pii	PII stands for Personally Identifiable Information. Removing sensitive information from data.
	quality	Improving the data quality, in the perspective of accuracy, consistency, and reliability of data.
	toxicity	(WIP) Removing harmful, offensive, or inappropriate content within the data.
Load	data_load	Saving the processed data to a preferred source like data lake, database, etc.
Utils	utils	Essential tools for data processing, including sampling, logging, statistics, etc.
Pipeline	pipeline	Executing the given ETL configuration.
Register	registry	Register custom functions to be used in Dataverse.