Skip to content

Tools for experimentation and using run:ai. The aim is for these to be small self-contained utilities that are used by multiple people.

License

Notifications You must be signed in to change notification settings

epfml/epfml-utils

Repository files navigation

EPFML Utilities

Internal tools for the MLO lab of EPFL.

Installation

❯ pip install epfml-utils

Add environment variables to your ~/.bashrc or equivalent file:

export EPFML_STORE_S3_ACCESS_KEY=""
export EPFML_STORE_S3_SECRET_KEY=""
export EPFML_STORE_S3_BUCKET=""
export EPFML_LDAP=""
# (Get those values from a friend.)
export EPFML_STORE_S3_ENDPOINT="https://s3.epfl.ch"

and make sure they are loaded:

source ~/.bashrc
echo $EPFML_LDAP  # Check if this prints your username.

Key-value store

This key-value store can help to transfer information between machines. Do not expect this to be fast or high-volume. Don't use this 100's of times in a training script.

Command-line usage

On one machine:

❯ epfml store set my_name "Bob"

On any other machine:

❯ epfml store get my_name
Bob

Python usage

import torch
import epfml.store

epfml.store.set("my_data", {"name": "Bob", "lab": "MLO"})
epfml.store.set("tensor", torch.zeros(4))
print(epfml.store.get("tensor"))
epfml.store.unset("tensor")
print(epfml.store.pop("my_data"))  # get and delete

Transporting code between machines

Packing

Upload a copy of the current working directory:

❯ epfml bundle pack
📦 Packaged and shipped.
⬇️ Unpack with `epfml bundle unpack mlotools_20230202_a205e830 -o .`.

To exclude (large / non-code) files from the package, add a config file to the directory

❯ epfml bundle init
📦 Default config file written to `/Users/vogels/epfl/mlotools/.epfml.bundle.toml`.

and customize it to your needs.

Unpacking

You can download the code into a directory:

❯ epfml bundle unpack mlotools_20230202_a205e830 -o some_directory

Or you can run a training script, or any other shell command, in a temporary check-out of the package:

❯ epfml bundle exec mlotools_20230202_a205e830 -- du -sh
🏃 Running inside a tmp clone of package `mlotools_20230202_a205e830`.
160K    .

Contributing

This repository is meant to be a collection of independent tools that each serve a simple well-defined purpose.

If you want to improve any of the tools in this repo, or contribute new tools. Take the following steps:

  1. Clone this repository.
  2. Initialize pre-commit: cd epfml-utils && pre-commit install (pip install pre-commit). This runs a couple of checks before you commit to keep this repo clean.
  3. Ask someone else to scrutinize your contributions. They can help you to improve the API and catch bugs. You can also check these very nice API design principles for tips.

To release a new version on PyPi, just increase the version number in pyproject.yoml and commit to Github.

About

Tools for experimentation and using run:ai. The aim is for these to be small self-contained utilities that are used by multiple people.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages