Client

The QALITA platform client allows you to communicate with the QALITA platform. It also enables you to launch QALITA agents. qalita agent run These agents are programs that allow you to run packs on sources. They can be thought of as workers.

Having a good understanding of how the QALITA Platform client works is essential for using the platform. As a data engineer, it will be your responsibility to set up QALITA agents on the machines that host the data sources.

tip

For the most up-to-date command-line documentation: Go directly to the PyPi.org website

Using the Command Line

The QALITA Command Line Interface (CLI) is a tool intended for Data Engineers who set up QALITA Platform's agents, sources, and assets.

It provides easy-to-use commands to help them set up a running QALITA platform environment quickly.

• Client
  • Using the Command Line
  • Quick Start
    • Installation
    • Usage
    • Setup
      • Minimal Config
      • Connected Config
      • Make an .env file and export ENV Values:
  • qalita agent
    • qalita agent login
    • qalita agent run
      • Job
      • Worker
    • qalita agent joblist
    • qalita agent info
  • qalita pack
    • qalita pack init
    • qalita pack list
    • qalita pack run
    • qalita pack validate
    • qalita pack push
  • qalita source
    • qalita source add
    • qalita source validate
    • qalita source list
    • qalita source push

Quick Start

Installation

As simple as:

pip install qalita

or there is a Docker Image

Usage

If you want more detailed and contextual help, type:

qalita COMMAND -h

Usage: qalita [OPTIONS] COMMAND [ARGS]...

  QALITA Command Line Interface

Setup

This CLI command communicates with the QALITA Platform API backend.

There are several layers of configuration depending on your needs:

Minimal Config

• QALITA_AGENT_NAME=agent_name

The agent name will help you identify it in the frontend interface; there are no restrictions on the name.

• QALITA_AGENT_MODE=job/worker

The mode of the agent:

Job: In job mode, when you use the command qalita agent run, it will immediately try to run a job in the local current context.

Worker: In worker mode, when you use the command qalita agent run, it will wait for the backend to give it jobs to run. It is similar to a scheduler.

Note that the commandqalita agent run` needs more configuration to run correctly; it will display errors otherwise.

Connected Config

• QALITA_AGENT_ENDPOINT=backend_api_url

Example: http://localhost:3080

The agent URL endpoint allows the agent to communicate with the QALITA platform endpoints. It enables:

• Listing packs
• Running jobs
• Publishing sources
• Publishing packs

• QALITA_AGENT_TOKEN=api_token

The token is provided while doing the quickstart steps in the frontend app. It is associated with your user and your role.

` Note that you need to have at least the [Data Engineer] role to use the QALITA CLI.

Make an .env file and export ENV Values:

You can alternatively make an .env file and export the values to your environment.

.env-local

QALITA_AGENT_NAME=`agent_name`
QALITA_AGENT_MODE=`job/worker`
QALITA_AGENT_ENDPOINT=https://api.company.com
QALITA_AGENT_TOKEN=`api_token`
QALITA_PACK_NAME=`pack_name`

Then export the values of the file to ENV values with:

export $(xargs < .env-local)

qalita agent

The qalita agent command allows you to:

• Register an agent to the platform
• Get information about your local agent
• Run a pack on a source
• List agent jobs (past & future)

qalita agent login

Parameters:

• name: the name of the agent
• mode: the mode of the agent job/worker
• token: the API token you get from the platform
• url: the backend API URL of the platform

qalita agent login registers your local agent to the platform. It enables you to run jobs or create routines (schedules) to run packs programmatically.

You need to have configured your agent with:

• QALITA_AGENT_ENDPOINT=backend_api_url
• QALITA_AGENT_TOKEN=api_token

You can get your token from the frontend or with an OAUTH2 API call to the /users/signin backend's endpoint.

More info on your frontend documentation and on the Connected config of the doc.

qalita agent run

Parameters:

• --name: the name of the agent
• --mode: the mode of the agent job/worker
• --token: the API token you get from the platform
• --url: the backend API URL of the platform

Specific parameters in job mode:

• --source: the source ID you want to run your job against
• --source-version (optional): the source version; by default, it will run the latest source version
• --pack: the pack ID you want to run your job against
• --pack-version (optional): the pack version; by default, it will run the latest version of the pack

qalita agent run

runs in different modes:

Job

The agent will run with the given configuration:

• -p: a pack_id given with the qalita pack list; note that your pack needs to be pushed to the platform to have an ID.
• -s: a source_id given with the qalita source list; note that your source needs to be pushed to the platform to have an ID.

Worker

The agent will wait until it receives an order from the frontend. It will then work the same as in job mode.

` Note that this mode will run indefinitely.

qalita agent joblist

Parameters:

• --name: the name of the agent
• --mode: the mode of the agent job/worker
• --token: the API token you get from the platform
• --url: the backend API URL of the platform

List jobs from the platform backend.

qalita agent info

Parameters:

• --name: the name of the agent
• --mode: the mode of the agent job/worker
• --token: the API token you get from the platform
• --url: the backend API URL of the platform

Get info about your local agent configuration.

qalita pack

The qalita pack command allows you to:

• Initialize a new pack
• List all available packs
• Validate it
• Run a local pack
• Push your pack version to the platform

qalita pack init

Parameters:

• --name: the name of the pack

Initialize a new pack. You need to set a name; it will create a new folder with the name of the pack.

You can set your name by passing a new parameter to the command line or setting a new environment variable: QALITA_PACK_NAME=my-super-pack``.

Here is the directory structure created:

./pack-name_pack/ /run.sh # Entrypoint file that will be run with qalita agent run /README.md # Documentation file /properties.yaml # Properties file that contains properties about the pack /main.py # (pack specific) The main script (you can run your pack with whatever language you choose) /config.json # (pack specific) The config file of your pack; you can use it to set any configurations you like. /requirements.txt # (pack specific) The requirements file that is run inside the run.sh

qalita pack list

Parameters:

• You need to have logged in with qalita agent login

List all the packs that are accessible to you with the QALITA Platform.

qalita pack run

Parameters:

• --name: Pack name

Run your locally configured pack.

qalita pack validate

Parameters:

• --name: Pack name

Validate your locally configured pack.

qalita pack push

Parameters:

• --name: Pack name

Push your locally configured pack.

qalita source

The qalita source command allows you to:

• Add a new source to your local configuration
• List your local sources from your qalita-conf.yml file
• Push your local sources from your qalita-conf.yml file
• Validate your conf file qalita-conf.yml

Note: By default, the qalita-conf.yml file is stored in ~/.qalita/qalita-conf.yml. Set the QALITA_HOME environment variable to customize the default path.

qalita source add

This function will help you add a new source to your configuration file qalita-conf.yaml.

This command does not have parameters; you need to follow the command prompts.

• Prompt 1: Source name
• Prompt 2: Source type
• Prompt 3: Is conditional, depends on the source type.
  • Case: Source Type = file: Source path
  • Case: Source Type = database: host / port / username / password / database
• Prompt 4: Source description
• Prompt 5: Is the source a reference? [bool] (default: false)
• Prompt 6: Is the source sensitive? [bool] (default: false)
• Prompt 7: Visibility of the source (private, internal, public) (default: private)

At the end of the prompt, the CLI will check the reachability of the source depending on the configuration and type. This step is called validate_source.

To complete the process of registering a new source to the platform, you need to push your source with the command: qalita source push.

qalita source validate

Helper function to help you add a new source to your configuration file qalita-conf.yaml.

qalita source list

Parameters:

You need to have a qalita-conf.yaml file that contains your sources configuration.

Example:

version: 1
sources:
  - config:
      path: /home/user/data_dir
    description: Folder containing CSV files
    name: my_csv_files
    owner: user
    reference: false
    visibility: private
    sensitive: false
    type: file

In this example, we have:

General keys

Key	Type	Description
version	int	The version of the configuration
sources	list	The list of sources

Source keys

Key	Type	Description
name	string	The name of the source
description	string	The description of the source
owner	string	The owner of the source
type	string	The type of the source
config	dict	The configuration of the source
visibility	string	The visibility of the source private/internal/public
reference	bool	Is the source a reference source
sensitive	bool	Is the source containing sensitive data

qalita source push

Registers your sources to the platform.

` Note: If you want to run a pack on your source, you will first need to push your source to the platform. It will give you a source_id with which you can run your pack.