Name		Name	Last commit message	Last commit date
parent directory ..
README.md		README.md
layers.py		layers.py
matmul.png		matmul.png
matmul_profiling.py		matmul_profiling.py
matmul_profiling.yaml		matmul_profiling.yaml
mlp_tp.png		mlp_tp.png
plots.ipynb		plots.ipynb
requirements.txt		requirements.txt
test_dot_product_distributed.py		test_dot_product_distributed.py
test_dot_product_local.py		test_dot_product_local.py
test_mlp_tp.py		test_mlp_tp.py
tp_profiling.py		tp_profiling.py
tp_profiling.yaml		tp_profiling.yaml
utils.py		utils.py

README.md

Tensor Parallelism

Code accompanying the deep-dive blog post on Tensor Parallelism.

The MLP and TP MLP layers are in layer.py
Matmul profiling code in matmul_profiling.py
MLP TP profiling code in tp_profiling.py
Tests of the rearranging tensor sums are in test_dot_product_{local,distributed}.py