- Add TLS support for orchestrator server
- Implement Health probes
- Add unit tests
- Add request validation for streaming classification with text generation endpoint
- Tokenization REST API and client will need to be updated for bidirectional streaming if/when available for REST use
- There is currently NO WAY for us to know the prefix ID required by TGIS for inferencing on tuned prompts
- Add configurable request timeouts for all clients