External backing storage demo

[bennsimon]

Description of what I changed

• Adds a demonstration of deploying a spark thrift server with external backing storage (Postgresql).
  • Uses an intermediary image that provides all the JDBC drivers that will be used by hive.

E2E test

TESTED:

Tested on my local machine by running below:

docker-compose -f compose-controller-spark-sql-external-storage.yaml up

Checklist: I completed these to help reviewers :)

• I have read and will follow the review process.

• I am familiar with Google Style Guides for the language I have coded in.

  No? Please take some time and review Java and Python style guides.

• My IDE is configured to follow the Google code styles.

  No? Unsure? -> configure your IDE.

• I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)

• I ran mvn clean package right before creating this pull request and added all formatting changes to my commit.

• All new and existing tests passed.

• My pull request is based on the latest changes of the master branch.

  No? Unsure? -> execute command git pull --rebase upstream master

[bashir2]

/gcbrun

[bashir2]

Thanks @bennsimon for this change; just added some suggestions/questions.

[bashir2]

We have a github security policy not to check-in binary artifacts as they are not reviewable (details here). Can we find another way of copying PostgreSQL driver, e.g., fetching it from an official docker image during the image build process? (We are violating this in one particular case which should be fixed too but I don't want to add more to it.)

[bennsimon]

Yeah, i can fetch it from the official site during the build process.

[bashir2]

If we are going to build a new image, why not start from bitnami/spark as base then copy the required driver?

[bennsimon]

To reduce the size of the image. When i use bitnami/spark base the final image becomes more than a GB but if i use a minimal image as base the size becomes ~15mb.

I was thinking that since the only purpose of this image is to host all the drivers why not use a minimal image. It frees the need to maintain a parallel spark image.

If we use bitnami/spark as base we wont need to copy over the drivers but then we will need to maintain a parallel sparkimage.

[bashir2]

Can we just reuse/adapt docker/hive-site_example.xml instead of creating a new one?

[bennsimon]

Yes, sure will do.

[bashir2]

Thanks again @bennsimon; I think we discussed this in other forums, can you please remind me if this is still needed and if yes, update it to be merged? It would be great to add a README.md to the docker/drivers-build directory describing that this is for demo purposes and not integrated into our continuous tests.

[bennsimon]

Thanks again @bennsimon; I think we discussed this in other forums, can you please remind me if this is still needed and if yes, update it to be merged? It would be great to add a README.md to the docker/drivers-build directory describing that this is for demo purposes and not integrated into our continuous tests.

Hey @bashir2, yeah i think it will be needed to demonstrate that one can setup spark thriftserver with database other than derby database.

I will add the README.md.