Curlpipe - DSL for making http execution pipelines.

(WARNING - under development as in everything is probably broken!)

Curlpipe is a little DSL making it easy to build up http execution pipelines.

Curlpipe uses libcurl under the covers, curl is a great ~~swiss army knife~~ http client that does much more then just make it easy to work with HTTP, supporting a plethora of URI addressable protocols. Over the years, the curl command line interface has grown - exposing many options with most users only ever invoking a subset of features, learning more advanced features over time.

Getting started

Eventually you will be able to Download the latest release for your platform but today you are on the bleeding edge and must build the software.

To try it out,

echo "[http://www.httpbin.org/get] > [/tmp/output.txt]" | curlpipe

or define a file (example.cp)

[http://www.httpbin.org/get] > [/tmp/output.txt]

and invoke curlpipe, supplying that file as its only argument.

> curlpipe example.cp

You should now observe the output from the URI saved to a file.

It is easy to send data eg.

"name=Jasmine&age=1" | [http://www.httpbin.org/post]

Some more examples.

Usage

To get help run

> curlpipe -h

curlpipe 0.1.0 | ⓒ 2017-2018 James Fuller <jim.fuller@webcomposite.com> | https://github.com/xquery/curlpipe

> curlpipe mycurlpipe.cp
    -h | --help   : Help.
    -d | --debug  : Emit debug info logging.
    -i | --info   : Emit info logging.
    -l | --log    : Enable logging to file.
    -q | --quiet  : Suppress output to stdout (console).
    -a | --auth   : Pass a username:password pair as the argument.
    -A | --auth-type : Specify the auth mechanism (basic|digest).
    -p | --params : Define set of parameters for token replacement (json|xml).
    -P | --param  : Define parameter(s) for token replacement.
    -o | --options : Define set of options controlling curlpipe behavior (default is ~/.curlpiperc).
    -O | --option  : Define option(s) controlling curlpipe behavior.

Curlpipe flags for controlling how much information is emitted during processing.

short	long	description
-h	--help	Obtain help
-d	--debug	Emit debug info logging.
-i	--info	Emit info logging.
-l	--log	Enable logging to file.
-q	--quiet	Suppress output to stdout (console).

Authentication

The following flags set the default auth credentials and auth-type used by curlpipe.

short	long	description
-a	--auth	Pass a username:password pair as the argument.
-A	--auth-type	Specify the auth mechanism (basic

~/.netrc (on windows ~/_netrc) is also supported by curlpipe (by dint of libcurl support):

> cat ~/.netrc
machine example.org
login myuser
password mypassword

Parameters

Parameters maybe passed to curlpipe, replacing tokens in the curlpipe script.

short	long	description
-p	--params	set file containing params (xml
-P	--param	set a param (ex. -Pid=1).

Parameters can be passed in via a file, for example, using data.json:

{"id":1, "name":"Ali G"}

The call to curlpipe would be

> curlpipe -p data.json example.cp

or data.xml looks like

<params>
<param>
<name>id</name>
<value>1</value>
</param>
<param>
<name>name</name>
<value>Ali G</value>
</param>
</params>

and similarly called

> curlpipe -p data.xml example.cp

Paramaters may also be individually defined, overidding params if used in conjunction with above -p flag.

> curlpipe -Pid=1 -Pname=Ali G example.cp

Options

By default, curlpipe looks for a ~/.curlpiperc file (on windows _curlpiperc) containing options that control curlpipe processing behavior. An example ~/.curlpiperc file is here

short	long	description
-o	--options	set file containing curlpipe options (xml
-O	--option	set an option (ex. -Pid=1).

Alternately you can set the location of this file with the -o flag.

Options can be overriden at the command line using the -O flag. The set of options curlpipe exposes are:

option	example	description
		TBA.
		TBA.

The curlpipe language

Curlpipe defines a series of statement(s). The simplest statement defines retrieval of a URI:

[http://www.httpbin.org/get] ;

where the seperator (;) is optional.

When this is run the output of dereferencing the URI is sent to stdout (con).

Data types

In addition to a URI, curlpipe supports boolean, literal, binary, xml, json and null data types.

data type	example	description
URI	`[http://www.example.org]`	used to POST, PUT or DELETE.
literal	`"name=value;name=value"`	literal data value.
binary	ex. zip file	binary data value.
xml	`<person><name>Tommy</name></person>`	well formed xml data.
json	`{id:1,name:"Tommy"}`	json data.
boolean	`? ?`	true or false.
null	`[]`	an empty/null value.

Literal string data can be used to pass in name value pairs (using application/x-www-form-urlencoded content type):

"name=value&name=value" ;

Binary data (for example, a zip file) is also supported.

Support for common formats, like XML and json.

"<person><name>Tommy</name></person>" ;

"{id:1,name:'Tommy;}";

Statements

A single curlpipe statement can be comprised of

(datatype | datatype operator datatype)

either a datatype or a datatype + operator + datatype where operators perform actions (or test conditions) on datatypes.

The following example illustrates how the contents of /temp/data.json is sent to a HTTP endpoint supporting POST method.

[/tmp/data.json] | [http://httpbin.org/post]

If the endpoint URI supports PUT it may opt to use that. If the endpoint does not support these methods then the appropriate http error code is thrown.

"name=Jim&age=21" | [http://httpbin.org/post]

The following would POST xml to the URI.

"<person><name>Tommy</name></person>" | [http://httpbin.org/post]

or now with a json datatype.

"{id:1,name:'Tommy'}" | [http://httpbin.org/post]

The pipe operator can be used to perform an HTTP DELETE

[] | [http://httpbin.org/delete]

Like the PUT the pipe operator will deduce if the endpoint supports DELETE. This built in behavior is enabled by curlpipe options (which can be set when invoking curlpipe).

Otherwise one is always free to force a PUT

"{id:1,name:'Tommy'}" =| [http://httpbin.org/put]

To force a DELETE method

[] =| [http://httpbin.org/delete]

To force a HEAD method use a conditional (explained further down)

[http://httpbin.org/head] != []

Operators

Operators chain together to build execution pipeline of arbitrary length.

[/tmp/data.json] | [http://httpbin.org/post] > [/tmp/output.txt] | [http://httpbin.org/post]

The set of processing operators are:

operator	description
\|	used to POST, PUT or DELETE.
>	redirect output to file.
>>	append output to file.
=\|	used to force PUT or DELETE.

Parameters

Parameters can be passed into curlpipe

> curlpipe -Pname=Tommy example.cp -Pid=1

and used for token replacement (ex. ${token}) in either data or URIs.

{name:"${name}"} | [http://httpbin.org/get/${id}]

Conditional Logic

curlpipe implements boolean and null datatype which can be used with conditional operators to test data values.

[http://www.httpbin.org/get] == "test"

[http://www.httpbin.org/get] ~= "test"

Where conditionals can be composited up using AND(&&) or OR(||) operators.

[http://www.httpbin.org/get] ~= "test" && [http://www.httpbin.org/get] != "not test"

curlpipe supports boolean logic, in the following form.

[http://www.httpbin.org/get] =~ "test" > [/tmp/matches.txt]

where the file is only written if the match is a success.

Additionally, curlpipe supports trinary logic, in the following form.

[http://www.httpbin.org/get] =~ "test"
       ? > [/tmp/success.txt]
       : 2> [/tmp/fail.txt]

The set of conditional operators are:

operator	description
==	equal
!=	does not equal
~=	regex text
&&	AND chain condition
\|\|	OR chain condition

Selectors

You can narrow down data using selectors, with postfix defining a simple xpath like selection.

[http://localhost:81/image/svg].svg.title | [http://localhost:81/post "Content-type":"application/xml"]

Where the above example will retrieve elements under svg/title ... this simple path selection works equally across xml or json.

Options

Curlpipe defines the following options

option	type	default	description
deduce-methods-on-pipe	boolean	true	will detect if URI supports PUT or DELETE when using pipe operator.

Examples

Find more examples here.

Retrieve (GET) and save to file

[http://www.httpbin.org/get] > [/tmp/output.txt]

Retrieve (GET) and save to file

[http://www.httpbin.org/get auth=myser:password auth-type=digest] > [/tmp/output3.txt]

Retrieve (GET) setting headers

[http://www.httpbin.org/get "Accept":"application/json"]

GET and append to file

[http://www.httpbin.org/get] >> [/tmp/response.txt]

[http://www.httpbin.org/get],[http://www.httpbin.org/uuid] >> [/tmp/response.txt]

GET and narrow down result with selector and POST results

[http://www.httpbin.org/image/svg].svg.title | [http://www.httpbin.org "Content-type":"application/xml"]

conditional operation

[http://www.httpbin.org/get] =~ "test" > [/tmp/matches.txt]

Trinary operator

[http://www.httpbin.org/get] =~ "test"
       ? > [/tmp/success.txt]
       : > [/tmp/fail/fail.txt]

POST json

"{'id':1 , "name":'James Fuller' age='${age}'}" | [http://www.httpbin.org/post] ;

[/tmp/data.json] | [http://www.httpbin.org/post "Content-type":"application/json"] ;

POST xml

"<person id="1"><name>John Smith<name><age>${age}</age></person>" | [http://www.httpbin.org/post] ;

[/tmp/data.xml] | [http://www.httpbin.org/post "Content-type":"application/xml"] ;

POST name=value

"id=1&name=James Fuller" | [http://www.httpbin.org/post] ;

POST zip file

[/tmp/mydoc.zip] | [http://www.httpbin.org/post "Content-type":"application/zip"] ;

PUT some json

will deduce if endpoint supports PUT

"{"test":1}" | [http://www.httpbin.org/put]

force an HTTP PUT

"{"test":1}" =| [http://www.httpbin.org/put]

DELETE

will deduce if endpoint supports DELETE

[] | [http://www.httpbin.org/delete]

force an HTTP DELETE

[/dev/null] =| [http://www.httpbin.org/delete]

Transclusion in URI

"${mypayload}" | [http://www.httpbin.org/delete/${myid}]

Transclusion in payload

{"id":${myid}} | [http://www.httpbin.org/delete/${myid}]

Comparison

[http://www.httpbin.org/get].url == 'http://www.httpbin.org/get'

[http://www.httpbin.org/get] .url != 'http://www.example.com'

Points of Interests

curlpipe is by design a 'little language' and most likely missing features from your 'favourite' language. It is intended as an adjunct to your existing script processing or host language.

To provide a framework for design thoughts, here are a few possibly non obvious gaps in the current codebase.

curlpipe is not intended as a drop in replacement for the curl tool (which is already a great CLI).
Intentionally lazy defining internals or worrying too much about performance at this stage.
curlpipe language is defined with an EBNF which is used to produce a strict parser.
Designing a programming language is hard - coherence and an easy to run AST are the first goals, please do raise an issue if you feel strongly where syntax could change.
I find using CMake non intuitive ... its enforced usage on this project is an attempt to learn more (otherwise you would see a Makefile here!).
Currently curlpipe is http centric in initial releases.
Woefully ignorant of windows platform ... looking at appveyor to eventually help solve that (issue #2).

I have blatantly stolen (and deformed) concepts from many places (bash, unix pipes, prolog, etc..). The following projects provide alternate approaches to solving similar problems that curlpipe is trying to address and worth a mention.

httpie
curlrc

Developing curlpipe

Please raise an issue or make a contribution by forking the repository and creating a pr.

Development work happens on the [develop branch])(https://github.com/xquery/curlpipe/tree/develop) and releases are based on master branch.

Compiling

To build software, run cmake:

> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_MANUAL=OFF -DBUILD_TESTING=OFF ..
> make
> make test
> make install

Running cmake requires access to the internet to pull down dependencies. This is problematic approach for those wishing to build on a standalone machine off the network (tracked as issue 1)

The build should complain if any other dependencies are missing (ex. openssl). Please review third party dependencies for any other build requirements.

Testing

Tests require httpbin - to setup run docker-compose from top level dir (which will setup both http/https)

> docker-compose -d up

alternately, run the docker command.

> docker run -p 81:80 kennethreitz/httpbin

After building usually it is just a matter of running make test target.

> make test

or you could run directly

> cd test
> ./runAllTests

Generating the Parser

lib/curlpipe/csparser.cpp is generated using REx Parser Generator with the following flags.

-name csparser -tree -cpp -faster

Release Package

Release packages are built using CPack.

Make a release build.

> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_MANUAL=OFF -DBUILD_TESTING=OFF ..
> make

Then run cpack.

> cpack --config CPackConfig.cmake

Dependencies

This project depends on the following external libs:

curl: libcurl (of course!).
REx Parser Generator: Gunther Rademacher grd@gmx.net excellant parser generator.
loguru: no fuss logging.
cxxopt: parsing command args.
googletest: testing.
rapidjson: json munging.
pugixml: xml dancing.

Please review these individual projects for more details on their own dependencies.

License

curlpipe is provided under the MIT License

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Files

README.md

Latest commit

History

README.md

File metadata and controls

Curlpipe - DSL for making http execution pipelines.

Getting started

Usage

Authentication

Parameters

Options

The curlpipe language

Data types

Statements

Operators

Parameters

Conditional Logic

Selectors

Options

Examples

Retrieve (GET) and save to file

Retrieve (GET) and save to file

Retrieve (GET) setting headers

GET and append to file

GET and narrow down result with selector and POST results

conditional operation

Trinary operator

POST json

POST xml

POST name=value

POST zip file

PUT some json

DELETE

Transclusion in URI

Transclusion in payload

Comparison

Points of Interests

Developing curlpipe

Compiling

Testing

Generating the Parser

Release Package

Dependencies

License