PyLith and GPUs and accelerators

[chai-debug]

PETSc supports OpenCL.

According to

https://petsc.org/release/install/install/

Run configure with $--download-viennacl$; check config/examples/arch-ci-linux-viennacl.py for example usage.

I think this requires recompiling petsc.Then I found out that Pylith allows for recompiling PETSc, so I followed the tutorial at

https://pylith.readthedocs.io/en/v4.0.0/developer/contributing/rebuilding.html

to recompile it, but it failed.
The error message is

Traceback (most recent call last):
File "/home/ronze/pylith/lib/petsc/conf/reconfigure-arch-pylith.py", line 6, in <module>
import configure
ModuleNotFoundError: No module named 'configure'

I have imported the pylith environment and changed the default python environment, which was originally

 #! /opt/pylith/dist/bin/python3

Now as

#! /home/ronze/pylith/bin/python3

Perhaps you can guide a tutorial on pylith installing petsc external package in detail

[chai-debug]

#! /home/ronze/pylith/bin/python3
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--LIBS=-lz',
    '--download-chaco=1',
    '--download-f2cblaslapack=1',
    '--download-viennacl',
    '--download-ml',
    '--prefix=/opt/pylith/dist',
    '--with-64-bit-points=1',
    '--with-c2html=0',
    '--with-clanguage=C',
    '--with-debugging=0',
    '--with-fc=0',
    '--with-hdf5=1',
    '--with-hwloc=0',
    '--with-large-file-io=1',
    '--with-lgrind=0',
    '--with-mpicompilers=1',
    '--with-shared-libraries=1',
    '--with-ssl=0',
    '--with-x=0',
    '--with-zlib=1',
    'CFLAGS=-g -O2',
    'CPPFLAGS=-I/opt/pylith/dist/include  -I/opt/pylith/dist/include',
    'CXXFLAGS=-g -O2 -DMPICH_IGNORE_CXX_SEEK',
    'FCFLAGS=',
    'LDFLAGS=-L/opt/pylith/dist/lib -L/opt/pylith/dist/lib64  -L/opt/pylith/dist/lib',
    'PETSC_ARCH=arch-pylith',
    'PETSC_DIR=/opt/pylith/build/cig/petsc-pylith',
  ]
  configure.petsc_configure(configure_options)

[baagaard-usgs]

It would help us if you could provide more context. What are your specific reasons for wanting to build PyLith with support for OpenCL? In general, this is not just a matter of building PETSc with OpenCL support.

[knepley]

The justifications for OpenCL have only diminished. It cannot vectorize code for CPUs, so the performance is terrible there. It used to have decent GPU performance, but Nvidia and AMD and Intel decided to pursue their own libraries instead of an open standard and it is no longer competitive there.

[chai-debug]

Thanks for reply

The reason I choose opencl is because our HPC uses a variety of acceleration cards, both GPU and DCU(a accelerator card MADE IN CHINA), which supports HIP, OpenCl and their own protocols. So I chose opencl, the most common protocol.

Moreover, if opencl shows poor results, I may use other accelerators.The main challenge I’m currently facing is getting the hang of using an accelerator card with PyLith.

Thanks again for your reply.

[chai-debug]

If you're having trouble, maybe you can teach me CUDA or HIP acceleration in PyLith.

[knepley]

PyLith does not directly use accelerators (this is in the plan for version 4 .0). Right now, linear algebra only uses accelerators, but this is not a large part of the runtime. Most of the runtime is in the formation of the Jacobian and calculation of the residual. You can try using the GPU with CUDA or HIP by setting the linear algebra types (-dm_vec_type cuda -dm_mat_type cuda), however you will get communication on each Newton iterate because we are forming the values on the CPU.

[chai-debug]

Thanks for your reply, I used the method you introduced.
But it has reported an error,

 -- Initializing timedependent problem with quasistatic formulation.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages
[0]PETSC ERROR: Unknown vector type: cuda

Does this method require petpack to download additional packages?

[CUDA](https://developer.nvidia.com/cuda-toolkit) A parallel computing platform and application programming interface model created by NVIDIA.

[ViennaCL](http://viennacl.sourceforge.net/) Linear algebra library providing matrix and vector operations using OpenMP, CUDA, and OpenCL.

If the method need to additional package,it's inevitable to recompiling petsc.

[knepley]

It means you need to reconfigure PETSc and turn on CUDA if you want to use that. There are instructions here: https://petsc.org/main/install/install/#installing-petsc-to-use-gpus-and-accelerators

A word of warning: This is likely to make very little difference in runtime. Hype has consumed computing with respect to GPUs. The determinant of PyLith runtime is almost always bandwidth, not compute. A high-end CPU and GPU have a very small bandwidth gap, and in order to drive a GPU near peak bandwdith you need 10s of millions of unknowns per GPU. Moreover scalable preconditioners for GPUs are still a research topic rather than settled practice.

[chai-debug]

OK，thank you for your reply.