Pipeline is a object containg a list of transformation steps.
Let's create a pipeline using Python interface:
from frictionless import Pipeline, transform, steps
pipeline = Pipeline(steps=[steps.table_normalize(), steps.table_melt(field_name='name')])
print(pipeline)
{'steps': [{'type': 'table-normalize'},
{'type': 'table-melt', 'fieldName': 'name'}]}To run a pipeline you need to use a transform function or method:
from frictionless import Pipeline, transform, steps
pipeline = Pipeline(steps=[steps.table_normalize(), steps.table_melt(field_name='name')])
resource = transform('table.csv', pipeline=pipeline)
print(resource.schema)
print(resource.read_rows())
{'fields': [{'name': 'name', 'type': 'string'},
{'name': 'variable', 'type': 'string'},
{'name': 'value', 'type': 'any'}]}
[{'name': 'english', 'variable': 'id', 'value': 1}, {'name': '中国人', 'variable': 'id', 'value': 2}]The Step concept is a part of the Transform API. You can create a custom Step to be used as part of resource or package transformation.
This step uses PETL under the hood.
from frictionless import Step
class cell_set(Step):
code = "cell-set"
def __init__(self, descriptor=None, *, value=None, field_name=None):
self.setinitial("value", value)
self.setinitial("fieldName", field_name)
super().__init__(descriptor)
def transform_resource(self, resource):
value = self.get("value")
field_name = self.get("fieldName")
yield from resource.to_petl().update(field_name, value)
Pipeline representation
(*, name: Optional[str] = None, title: Optional[str] = None, description: Optional[str] = None, steps: List[Step] = NOTHING) -> None
A short url-usable (and preferably human-readable) name. This MUST be lower-case and contain only alphanumeric characters along with “_” or “-” characters.
Optional[str]
Type of the package
ClassVar[Union[str, None]]
A human-oriented title for the Pipeline.
Optional[str]
A brief description of the Pipeline.
Optional[str]
List of transformation steps to apply.
List[Step]
Return type list of the steps
List[str]
Add new step to the schema
(step: Step) -> None
Remove all the steps
() -> None
Get step by type
(type: str) -> Step
Check if a step is present
(type: str) -> bool
Remove step by type
(type: str) -> Step
Set step by type
(step: Step) -> Optional[Step]
Step representation. A base class for all the step subclasses.
(*, name: Optional[str] = None, title: Optional[str] = None, description: Optional[str] = None) -> None
A short url-usable (and preferably human-readable) name. This MUST be lower-case and contain only alphanumeric characters along with “_” or “-” characters.
Optional[str]
A short url-usable (and preferably human-readable) name/type. This MUST be lower-case and contain only alphanumeric characters along with “_” or “-” characters. For example: "cell-fill".
ClassVar[str]
A human-oriented title for the Step.
Optional[str]
A brief description of the Step.
Optional[str]
Transform package
(package: Package)
Transform resource
(resource: Resource)