Problem Statement
For model serving with optimized routes we need to be able to send significantly larger volume of concurrent requests. it would be great to be able to control the pooling config.
Proposed Solution
Pass the pooling config as part of the setting the databricks config.
Additional Context
Alternative is to maintain a custom http client implementation with custom pooling manager configurations.
Problem Statement
For model serving with optimized routes we need to be able to send significantly larger volume of concurrent requests. it would be great to be able to control the pooling config.
Proposed Solution
Pass the pooling config as part of the setting the databricks config.
Additional Context
Alternative is to maintain a custom http client implementation with custom pooling manager configurations.