Skip to content

Conversation

TomeHirata
Copy link
Collaborator

@TomeHirata TomeHirata commented Sep 19, 2025

Currently, dspy modules whose output is not dspy.Prediction fail to execute when the usage tracker is enabled. This PR fixes the crash by skipping the usage tracking for this type of modules. While returning objects other than dspy.Prediciton is not expected, this pattern has been found in several places.

Example:

class MyProgram(dspy.Module):
        def __init__(self):
            self.predict = dspy.Predict("question -> answer")

        def forward(self, question: str, **kwargs) -> str:
            return self.predict(question=question).answer

program = MyProgram()
with dspy.context(track_usage=True):
    program(question="What is the capital of France?")

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a crash that occurs when usage tracker is enabled for dspy modules that return non-dspy.Prediction outputs. The fix allows such modules to execute normally by skipping usage tracking when no prediction object is found.

  • Replaces error condition with conditional check to prevent crashes
  • Adds test coverage for modules returning non-prediction outputs with usage tracking enabled

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
dspy/primitives/module.py Removes error condition and adds conditional check for prediction object before setting LM usage
tests/primitives/test_base_module.py Adds test case to verify usage tracker doesn't crash with non-prediction outputs

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link
Collaborator

@chenmoneygithub chenmoneygithub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to discuss a bit about this - I am not sure if we want to change the current behavior to silence the feature without user awareness. track_usage is an opt-in feature, users need to explicitly turn it on to enable the feature. The drawback of this change is users will see no usage being reported, while no error is raised. There are two patterns we can follow:

  • Keep the old behavior, but make the error message better.
  • Ship this PR, but log a warning when skipping setting the usage.

Usually I prefer giving users 100% awareness, but I assume you find this being annoying, which also makes sense to me. Let me know your thoughts!

@TomeHirata
Copy link
Collaborator Author

We shouldn't break program execution due to the failure of observability feature. I've added warning so that users can know why usage is not available.

Copy link
Collaborator

@chenmoneygithub chenmoneygithub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a comment!

@TomeHirata TomeHirata merged commit e433046 into stanfordnlp:main Sep 24, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants