Edit page in Livemark
(2023-01-25 11:55)

System

System Object

The most important undelaying object in the Frictionless Framework is system. It's an singleton object avaialble as frictionless.system.

System Context

Using the system object a user can alter the execution context. It uses a Python context manager so it can be used in anyway that it's possible in Python, for example, it can be nested or combined.

trusted

If data or metadata comes from a trusted origin, it's possible to disable safety checks for paths:

with system.use_context(trusted=True):
    extract('/path/to/file/is/absolute.csv')

onerror

To raise warning or errors on data problems, it's possible to use onerror context value. It's default to ignore and can be set to warn or error:

with system.use_context(onerror='error'):
    extract('table-with-error-will-raise-an-exeption.csv')

standards

By default, the framework uses coming v2 version of the standards for outputing metadata. It's possible to alter this behaviour:

with system.use_context(standards='v1'):
    describe('metadata-will-be-in-v1.csv')

http_session

It's possible to provide a custom requests.Session:

session = requests.Session()
with system.use_context(http_session=session):
    with Resource(BASEURL % "data/table.csv") as resource:
        assert resource.header == ["id", "name"]

System methods

This object can be used to instantiate different kind of lower-level as though Check, Step, or Field. Here is a quick example of using the system object:

from frictionless import Resource, system

# Create

adapter = system.create_adapter(source, control=control)
loader = system.create_loader(resource)
parser = system.create_parser(resource)

# Detect

system.detect_resource(resource)
field_candidates = system.detect_field_candidates()

# Select

Check = system.selectCheck('type')
Control = system.selectControl('type')
Error = system.selectError('type')
Field = system.selectField('type')
Step = system.selectStep('type')

As an extension author you might use the system object in various cases. For example, take a look at this MultipartLoader excerpts:

def read_line_stream(self):
    for number, path in enumerate(self.__path, start=1):
        resource = Resource(path=path)
        resource.infer(sample=False)
        with system.create_loader(resource) as loader:
            for line_number, line in enumerate(loader.byte_stream, start=1):
                if not self.__headless and number > 1 and line_number == 1:
                    continue
                yield line

It's important to understand that creating low-level objects in general is more corect using the system object than just classes because it will include all the available plugins in the process.

Plugin API

The Plugin API almost fully follows the system object's API. So as a plugin author you need to hook into the same methods. For example, let's take a look at a builtin Csv Plugin:

class CsvPlugin(Plugin):
    """Plugin for CSV"""

    # Hooks

    def create_parser(self, resource: Resource):
        if resource.format in ["csv", "tsv"]:
            return CsvParser(resource)

    def detect_resource(self, resource: Resource):
        if resource.format in ["csv", "tsv"]:
            resource.type = "table"
            resource.mediatype = f"text/{resource.format}"

    def select_Control(self, type: str):
        if type == "csv":
            return CsvControl

Reference

Adapter (class)

Loader (class)

Mapper (class)

Parser (class)

Plugin (class)

System (class)

Adapter (class)

Loader (class)

Loader representation

Signature

(resource: Resource)

Parameters

  • resource (Resource): resource

loader.remote (property)

Specifies if the resource is remote.

Signature

bool

loader.buffer (property)

Signature

IBuffer

loader.byte_stream (property)

Resource byte stream The stream is available after opening the loader

Signature

IByteStream

loader.closed (property)

Whether the loader is closed

Signature

bool

loader.resource (property)

Signature

Resource

loader.text_stream (property)

Resource text stream The stream is available after opening the loader

Signature

ITextStream

loader.close (method)

Close the loader as "filelike.close" does

Signature

() -> None

loader.open (method)

Open the loader as "io.open" does

loader.read_byte_stream (method)

Read bytes stream

Signature

() -> IByteStream

loader.read_byte_stream_analyze (method)

Detect metadta using sample

Signature

(buffer)

Parameters

  • buffer : byte buffer

loader.read_byte_stream_buffer (method)

Buffer byte stream

Signature

(byte_stream)

Parameters

  • byte_stream : resource byte stream

loader.read_byte_stream_create (method)

Create bytes stream

Signature

() -> IByteStream

loader.read_byte_stream_decompress (method)

Decompress byte stream

Signature

(byte_stream: IByteStream) -> IByteStream

Parameters

  • byte_stream (IByteStream): resource byte stream

loader.read_byte_stream_process (method)

Process byte stream

Signature

(byte_stream: IByteStream) -> ByteStreamWithStatsHandling

Parameters

  • byte_stream (IByteStream): resource byte stream

loader.read_text_stream (method)

Read text stream

loader.write_byte_stream (method)

Write from a temporary file

Signature

(path) -> Any

Parameters

  • path : path to a temporary file

loader.write_byte_stream_create (method)

Create byte stream for writing

Signature

(path) -> IByteStream

Parameters

  • path : path to a temporary file

loader.write_byte_stream_save (method)

Store byte stream

Signature

(byte_stream) -> Any

Parameters

  • byte_stream

Mapper (class)

Parser (class)

Parser representation

Signature

(resource: Resource)

Parameters

  • resource (Resource): resource

parser.requires_loader (property)

Specifies if parser requires the loader to load the data.

Signature

ClassVar[bool]

parser.supported_types (property)

Data types supported by the parser.

Signature

ClassVar[List[str]]

parser.cell_stream (property)

Signature

ICellStream

parser.closed (property)

Whether the parser is closed

Signature

bool

parser.loader (property)

Signature

Loader

parser.resource (property)

Signature

Resource

parser.sample (property)

Signature

ISample

parser.close (method)

Close the parser as "filelike.close" does

Signature

() -> None

parser.open (method)

Open the parser as "io.open" does

parser.read_cell_stream (method)

Read list stream

Signature

() -> ICellStream

parser.read_cell_stream_create (method)

Create list stream from loader

Signature

() -> ICellStream

parser.read_cell_stream_handle_errors (method)

Wrap list stream into error handler

Signature

(cell_stream: ICellStream) -> CellStreamWithErrorHandling

Parameters

  • cell_stream (ICellStream)

parser.read_loader (method)

Create and open loader

Signature

() -> Optional[Loader]

parser.write_row_stream (method)

Write row stream from the source resource

Signature

(source: Resource) -> None

Parameters

  • source (Resource): source resource

Plugin (class)

Plugin representation It's an interface for writing Frictionless plugins. You can implement one or more methods to hook into Frictionless system.

plugin.create_adapter (method)

Create adapter

Signature

(source: Any, *, control: Optional[Control] = None) -> Optional[Adapter]

Parameters

  • source (Any): source
  • control (Optional[Control]): control

plugin.create_loader (method)

Create loader

Signature

(resource: Resource) -> Optional[Loader]

Parameters

  • resource (Resource): loader resource

plugin.create_parser (method)

Create parser

Signature

(resource: Resource) -> Optional[Parser]

Parameters

  • resource (Resource): parser resource

plugin.detect_field_candidates (method)

Detect field candidates

Signature

(candidates: List[dict]) -> Optional[List[dict]]

Parameters

  • candidates (List[dict])

plugin.detect_resource (method)

Hook into resource detection

Signature

(resource: Resource) -> None

Parameters

  • resource (Resource): resource

System (class)

System representation This class provides an ability to make system Frictionless calls. It's available as `frictionless.system` singletone.

Signature

system.supported_hooks (property)

A flag that indicates if resource, path or package is trusted.

Signature

ClassVar[List[str]]

system.trusted (property)

A flag that indicates if resource, path or package is trusted.

Signature

bool

system.onerror (property)

Type of action to take on Error such as "warn", "raise" or "ignore".

Signature

IOnerror

system.standards (property)

Setting this value user can use feature of the specific version. The default value is v2.

Signature

IStandards

system.http_session (property)

Return a HTTP session This method will return a new session or the session from `system.use_http_session` context manager

system.create_adapter (method)

Create adapter

Signature

(source: Any, *, control: Optional[Control] = None) -> Optional[Adapter]

Parameters

  • source (Any)
  • control (Optional[Control])

system.create_loader (method)

Create loader

Signature

(resource: Resource) -> Loader

Parameters

  • resource (Resource): loader resource

system.create_parser (method)

Create parser

Signature

(resource: Resource) -> Parser

Parameters

  • resource (Resource): parser resource

system.deregister (method)

Deregister a plugin

Signature

(name)

Parameters

  • name : plugin name

system.detect_field_candidates (method)

Create candidates

Signature

() -> List[dict]

system.detect_resource (method)

Hook into resource detection

Signature

(resource: Resource) -> None

Parameters

  • resource (Resource): resource

system.register (method)

Register a plugin

Signature

(name, plugin)

Parameters

  • name : plugin name
  • plugin : plugin to register