Skip to content

Commit

Permalink
removed solr8 standalone mode
Browse files Browse the repository at this point in the history
  • Loading branch information
@liseli
liseli committed Aug 26, 2024
1 parent 08e3e86 commit b125d30
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 166 deletions.
132 changes: 19 additions & 113 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,17 +73,16 @@ following specifications:
- ${BABEL_HOME}/logs/solr:/opt/solr/server/logs
```

# Overview Solr 8.11.2 in standalone mode
# Overview Solr 8.11.2 in cloud mode

* Solr8.11.2 is the latest version of the 8.x series
*

### Upgrading our index from Solr6 to Solr8.11.2 (the last version)

A Solr configuration for full-text search consists of creating a directory (solrdata) that contains all the
files and directories Solr needs to start up the server in standalone mode and with one core (core-x).

To set up Solr 8, the same logic and resources used with Solr 6 was reused, then, minimal changes were made on JAR files,
To set up Solr 8, the same logic and resources used with Solr 6 have reused, then, minimal changes were made on JAR files,
Solr schemas and config files.

See below the followed steps to upgrade the server from Solr 6 to Solr 8.
Expand Down Expand Up @@ -114,46 +113,15 @@ the /var/solr/data folder. Inside each core directory, should be added:

5) **Create a docker-compose file to start up Solr server and for indexing data**.

### How to start up the Solr server in a standalone mode

* Use the docker-compose file for starting up the Solr server and for indexing data
* `docker compose -f docker-compose_solr8_standalone_mode.yml up`

#### How to integrate it in babel-local-dev

Update _docker-compose.yml_ file inside babel directory replacing the service _solr-lss-dev_. Create a new one with the
following specifications:

````
solr-lss-dev:
container_name: solr-lss-dev
image: solr-lss-dev
build:
context: ./lss_solr_configs
dockerfile: ./solr8.11.2_files/Dockerfile
target: standalone
healthcheck:
test: [ "CMD", "/usr/bin/curl", "-s", "-f", "http://solr-lss-dev:8983/solr/core-x/admin/ping" ]
interval: 5s
timeout: 10s
start_period: 30s
retries: 5
ports:
- "8983:8983"
volumes:
- solr_data:/var/solr/data
volumes:
solr_data:
````

# Overview Solr 8.11.2 in cloud mode:

* In the first iteration, the SolrCloud is a single replica and a single shard
* On docker, the SolrCloud is a single replica and a single shard. If you want to add more you should copy solr and
zookeeper services in the docker-compose file.
* The server was set up combining Solr, ZooKeeper and Docker
* For development, the Solr instance also have an embedded ZooKeeper server
* For production, we should use an external ZooKeeper server
* For development and production, we use an external ZooKeeper server

To start up the Solr server in cloud mode, we mount the script init_files/solr_init.sh in the container
On docker, to start up the Solr server in cloud mode, we mount the script init_files/solr_init.sh in the container
to allow setting up the authentication using a predefined security.json file.

The script specifies the ZK_HOST environment variable to point to the ZooKeeper server. It also copies the
Expand All @@ -166,91 +134,22 @@ will help us to define the dependencies between the services in the docker-compo
If we do not use the health checks, we probably will have to use the scripts wait-for-solr.sh and wait-for-zookeeper.sh
to make sure the authentication is set up correctly.

### Upgrading our index from Solr6 to Solr8.11.2 (the last version)

A solrCloud configuration for full-text search consists on a Solr cluster with a single node and a single Zookeeper. Two different
architecture has tested to set up solr in cloud mode.
On Kubernetes, none script is necessary to set up the authentication, because we use the
default Kubernetes secrets created by the operator.

* Option 1: Version 8.11.2 Solr’s embedded ZooKeeper instance
* Option 2: Version 8.11.2 Solr and an external Zookeeper ensemble

### Solr 8.11.2 and embedded ZooKeeper instance

* Using Solr’s embedded ZooKeeper instance is fine for getting started, development stages
* This set-up is not use in production because, the Solr instance also host Zookeeper, then if Solr shuts down,
Zookeeper is also shut down.
Any shards or Solr instances that rely on it will not be able to communicate with it or each other.
* It is the best alternative if you want to integrate SolrCloud 8 in babel-local-dev repository
* A [ZooKeeper Command Line Interface](https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-utilities.html)
was used to configure our own collections

#### How to start up the Solr server

* Execute the container:
* `docker compose -f docker-compose_embedded_zooKeeper.yml up`

* The docker-compose file, will create images for Solr and to index data, you can push the images to the registry repository
* Solr:
* Associate the local image with the image to register
* `docker image tag full-text-search-embedded_zoo:latest ghcr.io/hathitrust/full-text-search-embedded_zoo:example-8.11`
* Push the image
* `docker image push ghcr.io/hathitrust/full-text-search-embedded_zoo:example-8.11`
* Data loader:
* Associate the local image with the image to register
* `docker image tag lss_solr_configs-data_loader:latest ghcr.io/hathitrust/lss_solr_configs-data_loader:example-8.11`
* Push the image
* `docker image push ghcr.io/hathitrust/lss_solr_configs-data_loader:example-8.11`


#### How to integrate it in babel-local-dev

Update _docker-compose.yml_ file inside babel directory replacing the service _solr-lss-dev_. Create a new one with the
following specifications
### Upgrading our index from Solr6 to Solr8.11.2 (the last version)

```solr-lss-dev:
build:
context: ./lss_solr_configs
dockerfile: ./solr8.11.2_files/Dockerfile
target: embedded_zookeeper
container_name: solr-lss-dev
ports:
- "8983:8983"
volumes:
- solr_data:/var/solr/data
command: solr-foreground -c
healthcheck:
test: [ "CMD-SHELL", "solr healthcheck -c core-x" ]
interval: 5s
timeout: 10s
start_period: 30s
retries: 5
```
#### How to integrate with python full-text search indexer workflow using the images
Different architectures have tested to set up solr in cloud mode.

```solr-lss-dev:
image: ghcr.io/hathitrust/full-text-search-embedded_zoo:example-8.11
container_name: solr-lss-dev
ports:
- "8983:8983"
volumes:
- solr_data:/var/solr/data
command: solr-foreground -c
healthcheck:
test: [ "CMD-SHELL", "solr healthcheck -c core-x" ]
interval: 5s
timeout: 10s
start_period: 30s
retries: 5
```
* Option 1: Version 8.11.2 Solr embedded ZooKeeper instance
* Option 2: Version 8.11.2 Solr and an external Zookeeper ensemble

You might add the volume solr_data to the list of volume.

## Solr 8.11.2 and an external Zookeeper

* This is the recommended architecture for production environment
* In our solution, authentication is applied
* The integration in babel-local-dev repository is more verbose
* No docker image is generated because some services are running inside the docker
* In the docker-compose file, the address (a string) where ZooKeeper is running is defined, this way Solr is able
to connect to ZooKeeper server
* Use [Solr API](https://solr.apache.org/guide/8_11/collections-api.html) for creating and set up the collection
Expand All @@ -267,6 +166,13 @@ docker image tag xt_zoo_8.11.2:latest ghcr.io/hathitrust/full-text-search-extern
docker image push ghcr.io/hathitrust/full-text-search-external_zoo:shards-8.11
```

In Kubernetes, you should use a multiple platform image to run the Solr server. The recommendation is use the [github
actions workflow](https://github.com/hathitrust/lss_solr_configs/actions) to create the image for the different platforms.


lss_solr_configs repository
automatically generates the image for the different platforms on the git actions workflow.

If you are doing changes in the Dockerfile or in the solr_init.sh script it is better to create the image
each time you run the docker-compose file instead of using the image in the repository.

Expand Down
22 changes: 0 additions & 22 deletions docker-compose_solr8_standalone_mode.yml
View file

This file was deleted.

35 changes: 4 additions & 31 deletions solr8.11.2_files/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,37 +12,15 @@ ENV SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:SolrRocks"
ENV SOLR_OPTS="-Denable.packages=true -Dsolr.cloud.client.stallTime=30000 -Dsolr.security.json=/opt/solr/security.json \
-XX:-UseLargePages"

# Copy the required files to the image, lib and conf files
COPY --chown=solr:solr ./solr8.11.2_files/lib /var/solr/lib

#=======================================================
FROM common AS standalone
# any steps specific to building the standalone image

ENV SOLR_JAVA_MEM="-Xms1024m -Xmx1024m"
ENV SOLR_HEAP="1024m"

ENV SOLR_DATA_PATH=solrdata

COPY --chown=solr:solr ./solr8.11.2_files/conf /var/solr/data/core-x/conf

COPY --chown=solr:solr ./solr8.11.2_files/solr_standalone_mode/core.properties /var/solr/data/core-x

#======================DONE=================================
FROM common AS embedded_zookeeper
# any steps specific to building the image with embedded zookeeper

COPY --chown=solr:solr ./solr8.11.2_files/conf /opt/solr/core-x

COPY --chown=solr:solr ./solr8.11.2_files/security.json /opt/solr/security.json

COPY --chown=solr:solr --chmod=0755 ./solr8.11.2_files/solrCloud_embedded_zooKeeper/init_files/solr_init.sh /usr/bin/solr_init.sh

RUN /usr/bin/solr_init.sh

#=======================================================
FROM common AS external_zookeeper
FROM common AS external_zookeeper_docker
# any steps specific to building the image with external zookeeper

# authentication is enabled in the solr container
COPY --chown=solr:solr ./solr8.11.2_files/conf /opt/solr/core-x

COPY --chown=solr:solr ./solr8.11.2_files/security.json /opt/solr/security.json
Expand All @@ -55,12 +33,7 @@ COPY --chown=solr:solr --chmod=0755 ./solr8.11.2_files/solrCloud_external_zooKee
ENTRYPOINT ["/usr/bin/solr_init.sh"]

#=======================================================
FROM common AS external_zookeeper_kubernetes
#FROM common AS external_zookeeper_kubernetes
# any steps specific to building the image with external zookeeper

COPY --chown=solr:solr ./solr8.11.2_files/conf /opt/solr/core-x

COPY --chown=solr:solr --chmod=0755 ./solr8.11.2_files/solrCloud_external_zooKeeper/init_files/solr_init.sh /usr/bin/solr_init.sh

# Run this script to start solr as an entrypoint to configure the container to run as an executable
# ENTRYPOINT ["/usr/bin/solr_init.sh"]

0 comments on commit b125d30

Please sign in to comment.