Command Line Interface Tools

Core Pbsmrtpipe executable

Pbsmrtpipe workflow engine

usage: pbsmrtpipe [-h] [--version]
                  {pipeline,pipeline-id,task,show-templates,show-template-details,show-tasks,show-task-details,show-workflow-options,run-diagnostic,show-chunk-operators}
                  ...

Named Arguments

--version show program’s version number and exit

Sub-commands:

pipeline

Run a pipeline using a pipeline template or with explict Bindings and EntryPoints.

pbsmrtpipe pipeline [-h] [--debug] -e ENTRY_POINTS [ENTRY_POINTS ...]
                    [-o OUTPUT_DIR] [--preset-xml PRESET_XML]
                    [--preset-json PRESET_JSON]
                    [--preset-rc-xml PRESET_RC_XML]
                    [--service-uri SERVICE_URI]
                    [--force-distributed | --local-only]
                    [--force-chunk-mode | --disable-chunk-mode]
                    pipeline_template_xml
Positional Arguments
pipeline_template_xml
 Path to pipeline template XML file.
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

-e, --entry Entry Points using ‘entry_idX:/path/to/file.txt’ format.
-o, --output-dir
 

Path to job output directory. Directory will be created if it does not exist.

Default: /home/docs/checkouts/readthedocs.org/user_builds/pbsmrtpipe/checkouts/master/docs

--preset-xml

Preset/Option XML file. This option may be repeated if you have multiple preset files.

Default: []

--preset-json

Preset/Option JSON file. This option may be repeated if you have multiple preset files.

Default: []

--preset-rc-xml
 Skipping loading preset from ENV var ‘PB_SMRTPIPE_XML_PRESET’ and Explicitly load the supplied preset.xml
--service-uri Remote Webservice URI to send status updates to.
--force-distributed
 Override XML settings to enable distributed mode (if cluster manager is provided)
--local-only Override XML settings to disable distributed mode. All Task will be submitted to build-7100375-project-9441-pbsmrtpipe
--force-chunk-mode
 Override to enable Chunk mode
--disable-chunk-mode
 Override to disable Chunk mode

pipeline-id

Run a registered pipeline by specifying the pipeline id.

pbsmrtpipe pipeline-id [-h] [--debug] -e ENTRY_POINTS [ENTRY_POINTS ...]
                       [-o OUTPUT_DIR] [--preset-xml PRESET_XML]
                       [--preset-json PRESET_JSON]
                       [--preset-rc-xml PRESET_RC_XML]
                       [--service-uri SERVICE_URI]
                       [--force-distributed | --local-only]
                       [--force-chunk-mode | --disable-chunk-mode]
                       pipeline_id
Positional Arguments
pipeline_id Registered pipeline id (run show-templates) to show a list of the registered pipelines.
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

-e, --entry Entry Points using ‘entry_idX:/path/to/file.txt’ format.
-o, --output-dir
 

Path to job output directory. Directory will be created if it does not exist.

Default: /home/docs/checkouts/readthedocs.org/user_builds/pbsmrtpipe/checkouts/master/docs

--preset-xml

Preset/Option XML file. This option may be repeated if you have multiple preset files.

Default: []

--preset-json

Preset/Option JSON file. This option may be repeated if you have multiple preset files.

Default: []

--preset-rc-xml
 Skipping loading preset from ENV var ‘PB_SMRTPIPE_XML_PRESET’ and Explicitly load the supplied preset.xml
--service-uri Remote Webservice URI to send status updates to.
--force-distributed
 Override XML settings to enable distributed mode (if cluster manager is provided)
--local-only Override XML settings to disable distributed mode. All Task will be submitted to build-7100375-project-9441-pbsmrtpipe
--force-chunk-mode
 Override to enable Chunk mode
--disable-chunk-mode
 Override to disable Chunk mode

task

Run Task (i.e., ToolContract) by id

pbsmrtpipe task [-h] [--debug] -e ENTRY_POINTS [ENTRY_POINTS ...]
                [-o OUTPUT_DIR] [--preset-json PRESET_JSON]
                [--preset-xml PRESET_XML] [--preset-rc-xml PRESET_RC_XML]
                [--service-uri SERVICE_URI]
                [--force-distributed | --local-only]
                [--force-chunk-mode | --disable-chunk-mode]
                task_id
Positional Arguments
task_id Run a single task by Tool Contract Id (e.g., pbcommand.tasks.dev_txt_hello). Use ‘pbsmrtpipe show-tasks’ to get a complete list of registered tasks.
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

-e, --entry Entry Points should be defined as e_{x}:/path/to/file for each positional index of the Task. Example, for a task with 2 inputs, -e e_0:/path/to/f1.txt -e e_1:/path/to/f2.txt
-o, --output-dir
 

Path to job output directory. Directory will be created if it does not exist.

Default: /home/docs/checkouts/readthedocs.org/user_builds/pbsmrtpipe/checkouts/master/docs

--preset-json

Preset/Option JSON file. This option may be repeated if you have multiple preset files.

Default: []

--preset-xml

Preset/Option XML file. This option may be repeated if you have multiple preset files.

Default: []

--preset-rc-xml
 Skipping loading preset from ENV var ‘PB_SMRTPIPE_XML_PRESET’ and Explicitly load the supplied preset.xml
--service-uri Remote Webservice URI to send status updates to.
--force-distributed
 Override XML settings to enable distributed mode (if cluster manager is provided)
--local-only Override XML settings to disable distributed mode. All Task will be submitted to build-7100375-project-9441-pbsmrtpipe
--force-chunk-mode
 Override to enable Chunk mode
--disable-chunk-mode
 Override to disable Chunk mode

show-templates

List all pipeline templates. A pipeline ‘id’ can be referenced in your my_pipeline.xml file using ‘<import-template id=”pbsmrtpipe.pipelines.my_pipeline_id” />. This can replace the explicit listing of EntryPoints and Bindings.

pbsmrtpipe show-templates [-h] [--debug]
                          [--output-templates-avro OUTPUT_TEMPLATES_AVRO]
                          [--output-templates-json OUTPUT_TEMPLATES_JSON]
                          [--show-all]
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

--output-templates-avro
 Resolve, Validate and Output Registered pipeline templates to AVRO files to output-dir
--output-templates-json
 Resolve, Validate and Output Registered pipeline templates to JSON files to output-dir
--show-all

Display developer/internal pipelines

Default: False

show-template-details

Show details about a specific Pipeline template.

pbsmrtpipe show-template-details [-h] [-j OUTPUT_PRESET_JSON]
                                 [-o OUTPUT_PRESET_XML]
                                 template_id
Positional Arguments
template_id Show details of Pipeline Template.
Named Arguments
-j, --output-preset-json
 Write pipeline/task preset.json of options.
-o, --output-preset-xml
 Write pipeline/task preset.xml of options.

show-tasks

Show completed list of Tasks by id. Use ENV PB_TOOL_CONTRACT_DIR to define a custom directory of tool contracts. These TCs will override the installed TCs (e.g., PB_TOOL_CONTRACT_DIR=/path/to/my-tc-dir/)

pbsmrtpipe show-tasks [-h]

show-task-details

Show Details of a particular task by id (e.g., ‘pbsmrtpipe.tasks.filter_report’). Use ‘show-tasks’ to get a completed list of registered tasks.

pbsmrtpipe show-task-details [-h] [-j OUTPUT_PRESET_JSON]
                             [-o OUTPUT_PRESET_XML]
                             task_id
Positional Arguments
task_id Show details of registered Task by id.
Named Arguments
-j, --output-preset-json
 Write pipeline/task preset.json of options.
-o, --output-preset-xml
 Write pipeline/task preset.xml of options.

show-workflow-options

Display all workflow level options that can be set in <options /> for preset.xml

pbsmrtpipe show-workflow-options [-h] [-j OUTPUT_PRESET_JSON]
                                 [-o OUTPUT_PRESET_XML]
Named Arguments
-j, --output-preset-json
 Write pipeline/task preset.json of options.
-o, --output-preset-xml
 Write pipeline/task preset.xml of options.

run-diagnostic

Diagnostic tests of preset.xml and cluster configuration

pbsmrtpipe run-diagnostic [-h] [--debug] [-o OUTPUT_DIR] [--simple] preset_xml
Positional Arguments
preset_xml Path to Preset XML file.
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

-o, --output-dir
 

Path to job output directory. Directory will be created if it does not exist.

Default: /home/docs/checkouts/readthedocs.org/user_builds/pbsmrtpipe/checkouts/master/docs

--simple

Perform full diagnostics tests (e.g., submit test job to cluster).

Default: False

show-chunk-operators

Show a list of loaded chunk operators for Scatter/Gather Tasks. Extend resource loading by exporting ENV var PB_CHUNK_OPERATOR_DIR. Example export PB_CHUNK_OPERATOR_DIR=/path/to/chunk-operators-xml-dir

pbsmrtpipe show-chunk-operators [-h]

pbtools-runner

Runner

General tool used by run task-manifests.json files.

usage: pbtools-runner [-h] [--version] {run,to-cmds,inspect} ...

Named Arguments

--version show program’s version number and exit

Sub-commands:

run

Convert a Pacbio Input.xml file to Movie FOFN

pbtools-runner run [-h] [--debug] [-o OUTPUT_DIR] --task-stdout TASK_STDOUT
                   --task-stderr TASK_STDERR
                   task_manifest
Positional Arguments
task_manifest Path to task-manifest.json
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

-o, --output-dir
 

Output directory.

Default: /home/docs/checkouts/readthedocs.org/user_builds/pbsmrtpipe/checkouts/master/docs

--task-stdout Stdout of exe’ed manifest task commands.
--task-stderr Stderr of exe’ed manifest task commands.

to-cmds

Extract the cmds from manifest.json

pbtools-runner to-cmds [-h] task_manifest
Positional Arguments
task_manifest Path to task-manifest.json

inspect

Pretty-Print a summary of the task-manifestExtract the cmds from manifest.json

pbtools-runner inspect [-h] [--debug] task_manifest
Positional Arguments
task_manifest Path to task-manifest.json
Named Arguments
--debug

Alias for setting log level to DEBUG

Default: False

Testkit Runner

Testkit Tool to run pbsmrtpipe jobs.

usage: pbtestkit-runner [-h] [--version] [--log-file LOG_FILE]
                        [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} | --debug | --quiet | -v]
                        [--output-xml OUTPUT_XML] [--ignore-test-failures]
                        [--only-tests] [--force-distributed | --local-only]
                        [--force-chunk-mode | --disable-chunk-mode]
                        testkit_cfg

Positional Arguments

testkit_cfg Path to testkit.cfg file.

Named Arguments

--version show program’s version number and exit
--log-file Write the log to file. Default(None) will write to stdout.
--log-level

Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

Set log level

Default: “INFO”

--debug

Alias for setting log level to DEBUG

Default: False

--quiet

Alias for setting log level to CRITICAL to suppress output.

Default: False

-v, --verbose Set the verbosity level.
--output-xml Path to output XUnit XML
--ignore-test-failures
 

Exit with code 0 if pbsmrtpipe ran successfully, even if some tests fail

Default: False

--only-tests

Only run the tests.

Default: False

--force-distributed
 Override XML settings to enable distributed mode (if cluster manager is provided)
--local-only Override XML settings to disable distributed mode. All Task will be submitted to build-7100375-project-9441-pbsmrtpipe
--force-chunk-mode
 Override to enable Chunk mode
--disable-chunk-mode
 Override to disable Chunk mode

Testkit Multi-Runner

Run multiple testkit.cfg files in parallel

usage: pbtestkit-multirunner [-h] [--version] [--ignore-test-failures]
                             [--force-distributed | --local-only]
                             [--force-chunk-mode | --disable-chunk-mode]
                             [--debug] [-n NWORKERS] [-j JUNIT_OUT]
                             testkit_cfg_fofn

Positional Arguments

testkit_cfg_fofn
 File of butler.cfg file name relative to the current dir (e.g., RS_Resquencing/testkit.cfg

Named Arguments

--version show program’s version number and exit
--ignore-test-failures
 

Exit with code 0 if pbsmrtpipe ran successfully, even if some tests fail

Default: False

--force-distributed
 Override XML settings to enable distributed mode (if cluster manager is provided)
--local-only Override XML settings to disable distributed mode. All Task will be submitted to build-7100375-project-9441-pbsmrtpipe
--force-chunk-mode
 Override to enable Chunk mode
--disable-chunk-mode
 Override to disable Chunk mode
--debug

Alias for setting log level to DEBUG

Default: False

-n, --nworkers

Number of jobs to concurrently run.

Default: 1

-j, --junit-xml
 

JUnit output file for all tests

Default: “junit_combined_results.xml”

Resource RST Pipeline Generation from Resolved Pipeline JSON

Example:

$> python -m pbsmrtpipe.tools.resources_to_rst /path/to/resolved-pipeline-templates -o pipeline-docs/ --title "PacBio Custom Pipelines" --debug
$> cd pipeline-docs && make clean html
# pipelines docs will be accessible in _build/html

Generate Pipeline documentation from a directory of Resolved Pipeline Templates

usage: python -m pbsmrtpipe.tools.resources_to_rst [-h] [--version]
                                                   [--log-file LOG_FILE]
                                                   [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} | --debug | --quiet | -v]
                                                   [-o OUTPUT_DIR] [-t TITLE]
                                                   [-d DOC_VERSION]
                                                   pipeline_dir

Positional Arguments

pipeline_dir Path to Pipeline Template JSON Dir

Named Arguments

--version show program’s version number and exit
--log-file Write the log to file. Default(None) will write to stdout.
--log-level

Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

Set log level

Default: “INFO”

--debug

Alias for setting log level to DEBUG

Default: False

--quiet

Alias for setting log level to CRITICAL to suppress output.

Default: False

-v, --verbose Set the verbosity level.
-o, --output-dir
 

Path to RST Output Dir

Default: “pipeline-docs”

-t, --title

Title of Pipeline documents

Default: “PacBio Pipelines”

-d, --doc-version
 

Version of Pipeline documents

Default: “0.1.0”