Validation Guide
This guide assumes basic familiarity with the Frictionless Framework. To learn more, please read the Introduction and Quick Start.
Tabular data validation is a process of identifying problems that have occured in your data so you can correct them. Let's explore how Frictionless helps to achieve this task using an invalid data table example:
Download
capital-invalid.csvto reproduce the examples (right-click and "Save link as")..
We can validate this file by using the command-line interface. Frictionless provides comprehensive error details so that errors can be understood by the user. Continue reading to learn the validation process in detail.
Validate Functions#
The high-level interface for validating data provided by Frictionless is a set of validate functions:
validate: detects the source type and validates data accordinglyvalidate_schema: validates a schema's metadatavalidate_resource: validates a resource's data and metadatavalidate_package: validates a package's data and metadatavalidate_inquiry: validates a specialInquiryobject which represents a validation task instruction
On the command-line, there is only one command but there is a flag to adjust the behavior. It's useful when you have a file which has a ambiguous type, for example, a json file containing a data instead of metadata:
As a reminder, in the Frictionless ecosystem, a resource is a single file, such as a data file, and a package is a set of files, such as a data file and a schema. This concept is described in more detail in the Introduction.
Validating a Schema#
The validate_schema function is the only function validating solely metadata. To see this work, let's create an invalid table schema:
And let's validate this schema using the command-line interface:
We see that the schema is invalid and the error is displayed. Schema validation can be very useful when you work with different classes of tables and create schemas for them. Using this function will ensure that the metadata is valid.
Validating a Resource#
As was shown in the "Describing Data" guide, a resource is a container having both metadata and data. We need to create a resource descriptor and then we can validate it:
Note: this example uses JSON for the resource descriptor format, but Frictionless also supports YAML format as shown in examples below.
Let's now use the command-line interface to ensure that we are getting the same result that we got without using a resource:
Okay, why do we need to use a resource descriptor if the result is the same? The reason is metadata + data packaging. Let's extend our resource descriptor to show how you can edit and validate metadata:
We have added a few incorrect, made up attributes to our resource descriptor as an example. Now, the validation below reports these errors in addition to all the errors we had before. This example shows how concepts like Data Resource can be extremely useful when working with data.
Validating a Package#
A package is a set of resources + additional metadata. To showcase a package validation we need to use one more tabular file:
Download
capital-valid.csvto reproduce the examples (right-click and "Save link as").
Now let's describe and validate a package which contains the data files we have seen so far:
As we can see, the result is in a similar format to what we have already seen, and shows errors as we expected: we have one invalid resource and one valid resource.
Validating an Inquiry#
The Inquiry is an advanced concept mostly used by software integrators. For example, under the hood, Frictionless Framework uses inquiries to implement client-server validation within the built-in API. Please skip this section if this information feels unnecessary for you.
Inquiry is a declarative representation of a validation job. It gives you an ability to create, export, ans share arbitrary validation jobs containing a set of individual validation tasks. Tasks in the Inquiry accept the same arguments written in camelCase as the corresponding validate functions.
Let's create an Inquiry that includes an individual file validation and a resource validation. In this example we will use the data file, capital-valid.csv and the resource, capital.resource.json which describes the invalid data file we have already seen:
As usual, let's run validation:
At first sight, it might not be clear why such a construct exists, but when your validation workflow gets complex, the Inquiry can provide a lot of flexibility and power.
The Inquiry will use multiprocessing if there is the
parallelflag provided. It might speed up your validation dramatically especially on a 4+ cores processor.
Validation Report#
All the validate functions return a Validation Report. This is a unified object containing information about a validation: source details, the error, etc. Let's explore a report:
As we can see, there is a lot of information; you can find a detailed description of the Validation Report in the API Reference. Errors are grouped by tasks (i.e. data files); for some validation there can be dozens of tasks. Let's use the report.flatten function to simplify the representation of errors. This function helps to represent a report as a list of errors:
In some situations, an error can't be associated with a task; then it goes to the top-level report.errors property:
Validation Errors#
The Error object is at the heart of the validation process. The Report has report.errors and report.tasks[].errors, properties that can contain the Error object. Let's explore it by taking a deeper look at the duplicate-label error:
Above, we have listed universal error properties. Depending on the type of an error there can be additional ones. For example, for our duplicate-label error:
Please explore the Errors Reference to learn about all the available errors and their properties.
Available Checks#
There are various validation checks included in the core Frictionless Framework along with an ability to create custom checks. You can provide a list of checks where individual checks are in the form of:
- a dict:
{'code': 'code', 'option1': 'value1'} - an object:
checks.code(option1='value1')
See Validation Checks for a list of available checks.
Note that only the Baseline Check is enabled by default. Other built-in checks need to be activated as shown below.
Custom Checks#
There are many cases when built-in Frictionless checks are not enough. For instance, you might want to create a business logic rule or specific quality requirement for the data. With Frictionless it's very easy to use your own custom checks. Let's see with an example:
Usually, it also makes sense to create a custom error for your custom check. The Check class provides other useful methods like validate_header etc. Please read the API Reference for more details.
Learn more about custom checks in the Check Guide.
Pick/Skip Errors#
We can pick or skip errors by providing a list of error codes. This is useful when you already know your data has some errors, but you want to ignore them for now. For instance, if you have a data table with repeating header names. Let's see an example of how to pick and skip errors:
It's also possible to use error tags (for more information please consult the Errors Reference):
Limit Errors#
This option allows you to limit the amount of errors, and can be used when you need to do a quick check or want to "fail fast". For instance, here we use limit_errors to find just the 1st error and add it to our report:
Limit Memory#
Frictionless is a streaming engine; usually it is possible to validate terrabytes of data with basically O(1) memory consumption. It means that memory usage doesn't depend on the size of your data making the validation infinitely scalable. For some validation, this is not the case because Frctionless needs to buffer some cells e.g. to check uniqueness. Here memory management can be handy.
The default memory limit is 1000MB. You can adjust this based on your exact use case. For example, if you're running Frictionless as an API server you might reduce the memory usage. If a validation hits the limit it will not raise a failure - it will return a report with a task error: