Making Streaming Predictions

Function supports consuming the partial results of a prediction as they are made available by the predictor.

Consuming a Prediction Stream

Use the fxn.predictions.stream function to consume a prediction stream:

// Create a prediction stream
const stream = await fxn.predictions.stream({
    tag: "@text-co/split-sentence",
    inputs: { text: "Hello world" }
});

// Consume the stream
for await (const prediction of stream) {
    console.log(prediction.results[0]);
}

You can use prediction streaming to implement text generation user interfaces for LLMs.

Creating vs. Streaming Predictions

Streaming in Function is designed to be highly intuitive to use. We fully separate how a prediction function is implemented (i.e. eager vs generator functions) from how the function might be consumed (i.e. creating vs. streaming). Consider these two predictors:

def predict ():
    return "hello from Function"

You can choose how to consume a prediction function depending on what works best for your user experience.

Here are the reuslts of creating vs. streaming each function at runtime:

Creating Predictions with eager.py

In this case, the single prediction is returned:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

Creating Predictions with generator.py

In this case, the Function client will consume all partial predictions yielded by the predictor then return the very last one:

// Create a prediction with the streaming predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

Streaming Predictions with eager.py

In this case, the Function client will return a prediction stream with the single prediction returned by the predictor:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello from Function"

Streaming Predictions with generator.py

In this case, the Function client will provide a prediction stream containing all partial predictions yielded by the predictor:

// Create a prediction with the streaming predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello"
// "hello from"
// "hello from Function"

On this page

Consuming a Prediction Stream
Creating vs. Streaming Predictions

Function supports consuming the partial results of a prediction as they are made available by the predictor.

Consuming a Prediction Stream

Use the fxn.predictions.stream function to consume a prediction stream:

// Create a prediction stream
const stream = await fxn.predictions.stream({
    tag: "@text-co/split-sentence",
    inputs: { text: "Hello world" }
});

// Consume the stream
for await (const prediction of stream) {
    console.log(prediction.results[0]);
}

You can use prediction streaming to implement text generation user interfaces for LLMs.

Creating vs. Streaming Predictions

def predict ():
    return "hello from Function"

You can choose how to consume a prediction function depending on what works best for your user experience.

Here are the reuslts of creating vs. streaming each function at runtime:

Creating Predictions with eager.py

In this case, the single prediction is returned:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

Creating Predictions with generator.py

In this case, the Function client will consume all partial predictions yielded by the predictor then return the very last one:

// Create a prediction with the streaming predictor
const prediction = await fxn.predictions.create({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
console.log(prediction.results[0]);

// Outputs:
// "hello from Function"

Streaming Predictions with eager.py

In this case, the Function client will return a prediction stream with the single prediction returned by the predictor:

// Create a prediction with the eager predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/eager",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello from Function"

Streaming Predictions with generator.py

In this case, the Function client will provide a prediction stream containing all partial predictions yielded by the predictor:

// Create a prediction with the streaming predictor
const prediction = await fxn.predictions.stream({
  tag: "@fxn/generator",
  inputs: { }
});

// Display the results
for await (const prediction of stream) {
  console.log(prediction.results[0]);
}

// Outputs:
// "hello"
// "hello from"
// "hello from Function"

On this page

Consuming a Prediction Stream
Creating vs. Streaming Predictions

Making Streaming Predictions

Consuming a Prediction Stream

Creating vs. Streaming Predictions

Open Source

Style Guides

Making Streaming Predictions

Consuming a Prediction Stream

Creating vs. Streaming Predictions

​Consuming a Prediction Stream

​Creating vs. Streaming Predictions

Open Source

Style Guides

​Consuming a Prediction Stream

​Creating vs. Streaming Predictions

Consuming a Prediction Stream

Creating vs. Streaming Predictions

Consuming a Prediction Stream

Creating vs. Streaming Predictions