Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inference streaming support #1750

Merged
merged 58 commits into from
May 22, 2024
Merged

Inference streaming support #1750

merged 58 commits into from
May 22, 2024

Commits on May 22, 2024

  1. Add predict_stream with fallback

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    ee25235 View commit details
    Browse the repository at this point in the history

  2. Add predict_stream to handlers

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c799f54 View commit details
    Browse the repository at this point in the history

  3. Add sse-starlette to deps

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    4fa9c7a View commit details
    Browse the repository at this point in the history

  4. Send inference stream back as SSE

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    fadff06 View commit details
    Browse the repository at this point in the history

  5. Add tests for ServerSentEvent encoding

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    8c24122 View commit details
    Browse the repository at this point in the history

  6. Update responses to use \n\n as separator

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    266420e View commit details
    Browse the repository at this point in the history

  7. Add tests for infer-stream endpoint

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    f23c60f View commit details
    Browse the repository at this point in the history

  8. WIP: disable batch when streaming and adding tests for batch

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1451a14 View commit details
    Browse the repository at this point in the history

  9. WIP

    @RobertSamoilescu
    Adrian Gonzalez-Martin authored and RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    f5a5de3 View commit details
    Browse the repository at this point in the history

  10. Included generate and generate_stream endpoints and dataplane impleme… 

    …ntation.
    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    69b3423 View commit details
    Browse the repository at this point in the history

  11. Included generate and generate_stream to MLModel.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    5024f1e View commit details
    Browse the repository at this point in the history

  12. Included adaptive batching hooks for generate and generate_stream.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    9abd39d View commit details
    Browse the repository at this point in the history

  13. Included test for batch hooks.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1a7a2b3 View commit details
    Browse the repository at this point in the history

  14. Corrected type in batch hooks comment.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    43c983b View commit details
    Browse the repository at this point in the history

  15. Included inputs to the text-model.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    ab324bf View commit details
    Browse the repository at this point in the history

  16. Included dataplane generate and generate_stream tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    3de2216 View commit details
    Browse the repository at this point in the history

  17. Included endpoint tests for generate and generate_stream.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    55cf350 View commit details
    Browse the repository at this point in the history

  18. Split text model in TextModel and TextStreamModel.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    7f2140e View commit details
    Browse the repository at this point in the history

  19. Included model tests (generate_stream fallback and generate_stream).

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    482a499 View commit details
    Browse the repository at this point in the history

  20. Refactored benchmarking text-model.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    124e6be View commit details
    Browse the repository at this point in the history

  21. Included gRPC support for generate and generate stream.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1eefd22 View commit details
    Browse the repository at this point in the history

  22. Included grpc servicers tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d9119ae View commit details
    Browse the repository at this point in the history

  23. Included infer_stream as stream stream method.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    245b477 View commit details
    Browse the repository at this point in the history

  24. Back to the initially proposed batching hook.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    91b23c2 View commit details
    Browse the repository at this point in the history

  25. Fixed rest tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    795dd26 View commit details
    Browse the repository at this point in the history

  26. Fixed grpc tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    3cf01cd View commit details
    Browse the repository at this point in the history

  27. Fixed adaptive batching hooks.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c19fa47 View commit details
    Browse the repository at this point in the history

  28. Included grpc stream proto.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    6a01fa5 View commit details
    Browse the repository at this point in the history

  29. Included enable streaming flag for rest.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    14a6b25 View commit details
    Browse the repository at this point in the history

  30. Fixed rest stream tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    322fc3e View commit details
    Browse the repository at this point in the history

  31. Included logging warning for disabling gzip when streaming.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    dd56030 View commit details
    Browse the repository at this point in the history

  32. Disabled grpc interceptor while streaming.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    107266c View commit details
    Browse the repository at this point in the history

  33. Fixed predict_stream fallback.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1ce1ab4 View commit details
    Browse the repository at this point in the history

  34. Settings streaming validator.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1f7da51 View commit details
    Browse the repository at this point in the history

  35. Checkout sum-model model-settings.json

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    37614c2 View commit details
    Browse the repository at this point in the history

  36. Fixed inclusion of infer_stream endpoint.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c143a50 View commit details
    Browse the repository at this point in the history

  37. Checkout runtimes poetry lock.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    6a9d134 View commit details
    Browse the repository at this point in the history

  38. Updated dependencies.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    37ef1e3 View commit details
    Browse the repository at this point in the history

  39. Removed commented outdate batching test hook.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c291d7d View commit details
    Browse the repository at this point in the history

  40. Removed sse_starlette dependency.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    26a0812 View commit details
    Browse the repository at this point in the history

  41. Fixed benchmarking text-model settings.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    5435d48 View commit details
    Browse the repository at this point in the history

  42. Fixed linting issues.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d7d3c74 View commit details
    Browse the repository at this point in the history

  43. Renamed benchmarking text model.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    447e658 View commit details
    Browse the repository at this point in the history

  44. Included lazy fixtures as depenedency.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    12cbccd View commit details
    Browse the repository at this point in the history

  45. Removed streaming_enabled flag and introduced gzip_enabled flag.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d52aca0 View commit details
    Browse the repository at this point in the history

  46. Fixed linting issues.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    2302d99 View commit details
    Browse the repository at this point in the history

  47. Fixed handlers tests.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    ccbf945 View commit details
    Browse the repository at this point in the history

  48. Checkout runtimes poetry.lock

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    2cf3cb6 View commit details
    Browse the repository at this point in the history

  49. Updated poetry.lock

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    04a2aa8 View commit details
    Browse the repository at this point in the history

  50. Regenerated lock after rebasing.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    b632250 View commit details
    Browse the repository at this point in the history

  51. Fixed merge conflicts.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    37af739 View commit details
    Browse the repository at this point in the history

  52. Removed proto dummy hello.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    0841790 View commit details
    Browse the repository at this point in the history

  53. Fixed grpc comments.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    852cfc1 View commit details
    Browse the repository at this point in the history

  54. Adressed review comments.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    bf49a3a View commit details
    Browse the repository at this point in the history

  55. Fixed dataplane test.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    499bc9a View commit details
    Browse the repository at this point in the history

  56. Included grpc stream error handling.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d900100 View commit details
    Browse the repository at this point in the history

  57. Minor fix stream infer error.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    93e3503 View commit details
    Browse the repository at this point in the history

  58. Included new line at the end of json file.

    @RobertSamoilescu
    RobertSamoilescu committed May 22, 2024
    Configuration menu
    Copy the full SHA
    0ec59c7 View commit details
    Browse the repository at this point in the history