Add concurrent generation and cancellation on the vLLM backends, and a per-request provider mode - #40

Add concurrent generation and cancellation on the vLLM backends, and a per-request provider mode#40

emiehling merged 1 commit into

generative-computing:maingenerative-computing/steerability:mainfrom

emiehling:feat/continuous-batchingemiehling/AISteer360:feat/continuous-batchingCopy head branch name to clipboard