Streaming Predictions

Function supports consuming the partial results of a prediction as they are made available by the predictor.

Consuming a Prediction Stream

Use the fxn.predictions.stream function to consume a prediction stream:

You can use prediction streaming to implement text generation user interfaces for LLMs.

Creating vs. Streaming Predictions

Streaming in Function is designed to be highly intuitive to use. We fully separate how a prediction function is implemented (i.e. eager vs generator functions) from how the function might be consumed (i.e. creating vs. streaming). Consider these two predictors:

You can choose how to consume a prediction function depending on what works best for your user experience.

Here are the reuslts of creating vs. streaming each function at runtime:

In this case, the single prediction is returned:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

In this case, the Function client will consume all partial predictions yielded by the predictor then return the very last one:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

In this case, the Function client will return a prediction stream with the single prediction returned by the predictor:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello from Function"

In this case, the Function client will provide a prediction stream containing all partial predictions yielded by the predictor:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello"
// "hello from"
// "hello from Function"

Get Started

Workflows

Insiders

Consuming a Prediction Stream

Creating vs. Streaming Predictions

Get Started

Workflows

Insiders

​Consuming a Prediction Stream

​Creating vs. Streaming Predictions

Consuming a Prediction Stream

Creating vs. Streaming Predictions