Edit page in Livemark
(2023-01-25 11:55)

Field Steps

The Field steps are responsible for managing a Table Schema's fields. You can add or remove them along with more complex operations like unpacking.

Add Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_add(name="note", value="eu", descriptor={"type": "string"}),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'integer'},
            {'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'note', 'type': 'string'}]}
+----+-----------+------------+------+
| id | name      | population | note |
+====+===========+============+======+
|  1 | 'germany' |         83 | 'eu' |
+----+-----------+------------+------+
|  2 | 'france'  |         66 | 'eu' |
+----+-----------+------------+------+
|  3 | 'spain'   |         47 | 'eu' |
+----+-----------+------------+------+

Reference

steps.field_add (class)

steps.field_add (class)

Add field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, value: Optional[Any] = None, formula: Optional[Any] = None, function: Optional[Any] = None, position: Optional[int] = None, descriptor: Optional[IDescriptor] = None, incremental: bool = False) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • value (Optional[Any])
  • formula (Optional[Any])
  • function (Optional[Any])
  • position (Optional[int])
  • descriptor (Optional[IDescriptor])
  • incremental (bool)

steps.field_add.name (property)

A human-oriented name for the field.

Signature

str

steps.field_add.value (property)

Specifies value for the field.

Signature

Optional[Any]

steps.field_add.formula (property)

Evaluatable expressions to set the value for the field. The expressions are processed using simpleeval library.

Signature

Optional[Any]

steps.field_add.function (property)

Python function to set the value for the field.

Signature

Optional[Any]

steps.field_add.position (property)

Position index where to add the field. For example, to add the field in second position, we need to set it as 'position=2'.

Signature

Optional[int]

steps.field_add.descriptor (property)

A dictionary, which contains metadata for the field which describes the properties of the field.

Signature

Optional[IDescriptor]

steps.field_add.incremental (property)

Indicates if it is an incremental value. If True, the sequential value is set to the new field. The default value is false.

Signature

bool

Filter Fields

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_filter(names=["id", "name"]),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'integer'},
            {'name': 'name', 'type': 'string'}]}
+----+-----------+
| id | name      |
+====+===========+
|  1 | 'germany' |
+----+-----------+
|  2 | 'france'  |
+----+-----------+
|  3 | 'spain'   |
+----+-----------+

Reference

steps.field_filter (class)

steps.field_filter (class)

Filter fields. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, names: List[str]) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • names (List[str])

steps.field_filter.names (property)

Names of the field to be read. Other fields will be ignored.

Signature

List[str]

Merge Fields

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
     source,
     steps=[
     	 # seperator argument can be used to set delimeter. Default value is '-'
    	 # preserve argument keeps the original fields
         steps.field_merge(name="details", from_names=["name", "population"], preserve=True)
     ],
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'integer'},
            {'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'details', 'type': 'string'}]}
+----+-----------+------------+--------------+
| id | name      | population | details      |
+====+===========+============+==============+
|  1 | 'germany' |         83 | 'germany-83' |
+----+-----------+------------+--------------+
|  2 | 'france'  |         66 | 'france-66'  |
+----+-----------+------------+--------------+
|  3 | 'spain'   |         47 | 'spain-47'   |
+----+-----------+------------+--------------+

Reference

steps.field_merge (class)

steps.field_merge (class)

Merge fields. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, from_names: List[str], separator: str = -, preserve: bool = False) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • from_names (List[str])
  • separator (str)
  • preserve (bool)

steps.field_merge.name (property)

Name of the new field that will be created after merge.

Signature

str

steps.field_merge.from_names (property)

List of field names to merge.

Signature

List[str]

steps.field_merge.separator (property)

Separator to use while merging values of the two fields.

Signature

str

steps.field_merge.preserve (property)

It indicates if the fields are preserved or not after merging. If True, fields will not be removed and vice versa.

Signature

bool

Move Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_move(name="id", position=3),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'id', 'type': 'integer'}]}
+-----------+------------+----+
| name      | population | id |
+===========+============+====+
| 'germany' |         83 |  1 |
+-----------+------------+----+
| 'france'  |         66 |  2 |
+-----------+------------+----+
| 'spain'   |         47 |  3 |
+-----------+------------+----+

Reference

steps.field_move (class)

steps.field_move (class)

Move field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, position: int) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • position (int)

steps.field_move.name (property)

Field name to move.

Signature

str

steps.field_move.position (property)

New position for the field being moved.

Signature

int

Pack Fields

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
    	# field_type returns packed fields as JSON Object. Default value for field_type is 'array'
    	# preserve argument keeps the original fields
        steps.field_pack(name="details", from_names=["name", "population"], as_object=True, preserve=True)
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'integer'},
            {'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'details', 'type': 'object'}]}
+----+-----------+------------+-----------------------------------------+
| id | name      | population | details                                 |
+====+===========+============+=========================================+
|  1 | 'germany' |         83 | {'name': 'germany', 'population': '83'} |
+----+-----------+------------+-----------------------------------------+
|  2 | 'france'  |         66 | {'name': 'france', 'population': '66'}  |
+----+-----------+------------+-----------------------------------------+
|  3 | 'spain'   |         47 | {'name': 'spain', 'population': '47'}   |
+----+-----------+------------+-----------------------------------------+

Reference

steps.field_pack (class)

steps.field_pack (class)

Pack fields. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, from_names: List[str], as_object: bool = False, preserve: bool = False) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • from_names (List[str])
  • as_object (bool)
  • preserve (bool)

steps.field_pack.name (property)

Name of the new field.

Signature

str

steps.field_pack.from_names (property)

List of fields to be packed.

Signature

List[str]

steps.field_pack.as_object (property)

The packed value of the field will be stored as object if set to True.

Signature

bool

steps.field_pack.preserve (property)

Specifies if the field should be preserved or not. If True, fields part of packing process will be preserved.

Signature

bool

Remove Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_remove(names=["id"]),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'}]}
+-----------+------------+
| name      | population |
+===========+============+
| 'germany' |         83 |
+-----------+------------+
| 'france'  |         66 |
+-----------+------------+
| 'spain'   |         47 |
+-----------+------------+

Reference

steps.field_remove (class)

steps.field_remove (class)

Remove field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, names: List[str]) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • names (List[str])

steps.field_remove.names (property)

List of fields to remove.

Signature

List[str]

Split Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_split(name="name", to_names=["name1", "name2"], pattern="a"),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'integer'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'name1', 'type': 'string'},
            {'name': 'name2', 'type': 'string'}]}
+----+------------+--------+-------+
| id | population | name1  | name2 |
+====+============+========+=======+
|  1 |         83 | 'germ' | 'ny'  |
+----+------------+--------+-------+
|  2 |         66 | 'fr'   | 'nce' |
+----+------------+--------+-------+
|  3 |         47 | 'sp'   | 'in'  |
+----+------------+--------+-------+

Reference

steps.field_split (class)

steps.field_split (class)

Split field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, to_names: List[str], pattern: str, preserve: bool = False) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • to_names (List[str])
  • pattern (str)
  • preserve (bool)

steps.field_split.name (property)

Name of the field to split.

Signature

str

steps.field_split.to_names (property)

List of names of new fields.

Signature

List[str]

steps.field_split.pattern (property)

Pattern to split the field value, for example: "a".

Signature

str

steps.field_split.preserve (property)

Whether to preserve the fields after the split. If True, the fields are not removed after split.

Signature

bool

Unpack Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_update(name="id", value=[1, 1], descriptor={"type": "string"}),
        steps.field_unpack(name="id", to_names=["id2", "id3"]),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'},
            {'name': 'id2', 'type': 'any'},
            {'name': 'id3', 'type': 'any'}]}
+-----------+------------+-----+-----+
| name      | population | id2 | id3 |
+===========+============+=====+=====+
| 'germany' |         83 |   1 |   1 |
+-----------+------------+-----+-----+
| 'france'  |         66 |   1 |   1 |
+-----------+------------+-----+-----+
| 'spain'   |         47 |   1 |   1 |
+-----------+------------+-----+-----+

Reference

steps.field_unpack (class)

steps.field_unpack (class)

Unpack field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, to_names: List[str], preserve: bool = False) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • to_names (List[str])
  • preserve (bool)

steps.field_unpack.name (property)

Name of the field to unpack.

Signature

str

steps.field_unpack.to_names (property)

List of names for new fields that will be created after unpacking.

Signature

List[str]

steps.field_unpack.preserve (property)

Whether to preserve the source fields after unpacking.

Signature

bool

Update Field

Example

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
    source,
    steps=[
        steps.field_update(name="id", value=str, descriptor={"type": "string"}),
    ]
)
print(target.schema)
print(target.to_view())
{'fields': [{'name': 'id', 'type': 'string'},
            {'name': 'name', 'type': 'string'},
            {'name': 'population', 'type': 'integer'}]}
+------+-----------+------------+
| id   | name      | population |
+======+===========+============+
| None | 'germany' |         83 |
+------+-----------+------------+
| None | 'france'  |         66 |
+------+-----------+------------+
| None | 'spain'   |         47 |
+------+-----------+------------+

Reference

steps.field_update (class)

steps.field_update (class)

Update field. This step can be added using the `steps` parameter for the `transform` function.

Signature

(*, title: Optional[str] = None, description: Optional[str] = None, name: str, value: Optional[Any] = None, formula: Optional[Any] = None, function: Optional[Any] = None, descriptor: Optional[IDescriptor] = None) -> None

Parameters
  • title (Optional[str])
  • description (Optional[str])
  • name (str)
  • value (Optional[Any])
  • formula (Optional[Any])
  • function (Optional[Any])
  • descriptor (Optional[IDescriptor])

steps.field_update.name (property)

Name of the field to update.

Signature

str

steps.field_update.value (property)

Cell value to set for the field.

Signature

Optional[Any]

steps.field_update.formula (property)

Evaluatable expressions to set the value for the field. The expressions are processed using simpleeval library.

Signature

Optional[Any]

steps.field_update.function (property)

Python function to set the value for the field.

Signature

Optional[Any]

steps.field_update.descriptor (property)

A descriptor for the field to set the metadata.

Signature

Optional[IDescriptor]