Notes.txt

- Simplestreams
  Simplestreams is a data format, a python client library and a command line
  client to interact with the dataformat.
  - Ubuntu publishes data about downloads and image-ids in simple streams format
  - Juju queries simplestreams data to determine both its downloads and image-ids.

- general overview of simplestreams
  - all json data
  - gpg signatures allow for secure use over insecure transfer (http)
  - designed to be easily mirrored 
    - without race conditions
    - with gpg signatures kept intact
  - generally, all items can have arbitrary "tags" which are key/value pairs
  - some tags have specific meaning ('path')
  - 2 different "format" are defined and used
    - index:1.0
      - indexes other files
      - has 'index' dictionary that is populated with 'content_id's.
    - products:1.0
      - "content_id" : declares where this content is coming from.
         example: "com.ubuntu.cloud:released:download"
      - "products":
         A product is something that acts in an expected way.  Examples of
         a product is "com.ubuntu.cloud:server:11.10:i386"
         or "com.ubuntu.cloud:server:12.04:amd64"
         
         Products have multiple "versions".  Each version should behave
         similarly, essentially promising an ABI of sorts.

         Other examples of a product might be:
           libwhizbang:1.1:ppc64
           python-foolib:1.1

      - "versions"
        Versions allow for updates of a product.  Ubuntu distributes cloud images
        daily or on a 3 week release cadence.
        a "Version" is then something like a YYYYMMDD format, but can be
        anything.  newer versions are those that sort greater in a C-Locale sort.

      - "items"
        Items are things that go with that version.
        Example items for cloud images are -disk1.img and .tar.gz.

- usquery / usmirror
  simple wrapper around sstream-query and sstream-mirror that knows urls

- sstream-mirror
  - filters allow you to filter entries by tags using operators:
     ~, !~, =, !=

  - Example 1:
    u-smirror --no-item-download uc-release ./luc-release --dry-run 
    u-smirror --no-item-download uc-release ./luc-release
    u-smirror uc-release ./luc-release --no-item-download

    find luc-release/ -type f

    # explicitly mirror the .json (unsigned data)
    sstream-mirror --no-item-download \
       http://cloud-images.ubuntu.com/releases/streams/v1/index.json ./luc-release

    find luc-release/ -type f

  - Example 2: mirror cirros data, show --progress
    u-squery cirros --pretty --max=1
    u-squery cirros --pretty --max=1 stream=released

    # use tenv here to use trunk to show '--progress'
    tenv u-smirror cirros stream=released ftype=disk.img ./lcirros --progress

    find lcirros/ -type f

- sstream-query

  - Example 1: just dump a lot of data.  This dumps all data for all items
    in the provided url (released ubuntu cloud images).
     u-squery uc-release --pretty --dry-run
     u-squery uc-release --pretty 

    Since we mirrored that, we can query the local mirror.
     u-squery luc-release --pretty --max=1 --dry-run

  - Example 2: Show me ami-ids on AWS
     u-squery uc-aws --pretty --dry-run
     u-squery luc-aws --pretty --max=1 --dry-run
     u-squery luc-aws --pretty --max=1 arch=amd64 release=trusty
     u-squery luc-aws --max=1  "--output-format=%(region)s %(pubname)s" \
         release=trusty

     u-squery luc-aws --output-format="%(id)s %(pubname)s" | wc -l

     iterate/improve to show

     u-squery luc-aws --max=1 arch=amd64 release=trusty   \
       --output-format="%(id)s %(region)-15s %(virt)-4s %(root_store)-10s %(pubname)"

  - Example 3: Show me ami-ids on Azure
    u-squery luc-release --max=1 | grep windows
    
    u-squery luc-release "--output-format=%(region)s %(id)s %(pubname)s" \
      --max=1 endpoint=https://management.core.windows.net/ "region~Japan"

    This shows a need for '--delimiter'.

  - Example 4: safely download single url with simple shell

    u-squery uc-dl --dry-run
    u-squery uc-dl arch=amd64 ftype=disk1.img release=trusty  --pretty --max=1
    u-squery uc-dl arch=amd64 ftype=disk1.img release=trusty \
       "--output-format=%(sha256)s %(item_url)s"

    out=$(u-squery uc-dl arch=amd64 ftype=disk1.img release=trusty \
            "--output-format=%(sha256)s %(item_url)s" --max=1)
    echo $out
    sum=${out% *}; url=${out#* };
    echo "${sum}  trusty-disk1.img" > sums

    wget "$url" -O trusty-disk1.img
    sha256sum --check sums

- example-sync / GlanceMirror
  - https://code.launchpad.net/~smoser/simplestreams/example-sync
  - GlanceMirror mirrors a cloud-image products data into glance.
  - it can publish simplestreams data to swift, which then can be queried
  - crates data like at https://swift.canonistack.canonical.com/v1/AUTH_a48765cc0e864be980ee21ae26aaaed4/simplestreams/data/streams/v1/index.json

- cstack
  - This is an simple client that queries openstack ids from a simplestreams mirror.
    It knows heuristics and makes launching easier:
     cstack query trusty
     cstack qrun trusty --user-data=my-user-data
  - https://code.launchpad.net/~smoser/simplestreams/cstack

- sstream-sync
  - "hook based sync"
    calls other programs with items to be added or removed.
  - example: http://bazaar.launchpad.net/~smoser/simplestreams/trunk/view/head:/tools/hook-check-downloads