Streaming Predictions
Iterating through results as they become available
Function supports consuming the partial results of a prediction as they are made available by the predictor.
Currently, no predictors support streaming because prediction streams are a work-in-progress. We expect to launch prediction streaming very soon.
Consuming a Prediction Stream
Use the fxn.predictions.stream
function to consume a prediction stream:
You can use prediction streaming to implement text generation user interfaces for LLMs.
Creating vs. Streaming Predictions
Streaming in Function is designed to be highly intuitive to use. We fully separate how a prediction function is implemented (i.e. eager vs generator functions) from how the function might be consumed (i.e. creating vs. streaming). Consider these two predictors:
You can chooose how to consume a prediction function depending on what works best for your user experience.
Here are the reuslts of creating vs. streaming each function at runtime: