Edit page in Livemark
(2023-06-06 13:09)

Contributing

We welcome contributions from anyone! Please read the following guidelines, and feel free to reach out to us if you have questions. Thanks for your interest in helping make Frictionless awesome!

General Guidelines

We use Github as a code and issues hosting platform. To report a bug or propose a new feature, please open an issue. For pull requests, we would ask you initially create an issue and then create a pull requests linked to this issue.

Docs Contribution

To contribute to the documentation, please find an article in the docs folder and update its contents. We write our documentation using Livemark. Livemark provides an ability to provide examples without providing an output as it's generated automatically.

It's possible to run this documentation portal locally:

livemark start

Code Contribution

Frictionless is a Python3.8+ framework, and it uses some common Python tools for the development process:

You also need git to work on the project, and make is recommended.

After cloning the repository, you can set up the development environment either by creating a virtual environment or a docker container.

Using a virtual environment

Create a virtual environment and install the dependencies by following this code:

this will install a git commit hook running the tests

python3.8 -m venv .python
source .python/bin/activate
pip install wheel
make install
alias "frictionless=python -m frictionless"

Note: You may need to run sudo apt-get install postgresql libpq-dev on a Debian-based system, because the python Postgres module depends on some postgres CLI tools.

Using a Docker container

Use the following command to build the container:

make docker-setup

This should take care of setting up everything. If the container is built without errors, you can then run commands like make inside the container to accomplish various tasks (see the next section for details).

To make things easier, we can create an alias:

alias "frictionless-dev=docker run --rm -v $PWD:/home/frictionless -it frictionless-dev"

Then, for example, to run the tests, we can use:

frictionless-dev make test

Using make

Then you can run various make commands:

We also recommend running underlying commands like pytest or pylama to speed up the development process, though this is optional.

Release Process

To release a new version:

Running tests offline(HTTP requests) with VCR

VCR library records the response from HTTP requests locally as cassette in its first run. All subsequent calls are run using recorded metadata from previous HTTP request, so it speeds up the testing process. To record a unit test(as cassette), we mark it with a decorator:

@pytest.mark.vcr
def test_connect_with_server():
	pass

Cassettee will be recorded as "test_connect_with_server.yaml". A new call is made when params change. To skip sensitive data, we can use filters:

@pytest.fixture(scope="module")
def vcr_config():
    return {"filter_headers": ["authorization"]}

Regenerating cassettes for CKAN

CKAN_APIKEY=***************************

Regenerating cassettes for Zenodo

Read

ZENODO_ACCESS_TOKEN=***************************

Write

ZENODO_SANDBOX_ACCESS_TOKEN=***************************

base_url='base_url="https://sandbox.zenodo.org/api/'

Regenerating cassettes for Github

GITHUB_NAME=FD
GITHUB_EMAIL=frictionle[email protected]
GITHUB_ACCESS_TOKEN=***************************