Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: feed wrapping #4677

Merged
merged 44 commits into from
Sep 19, 2024
Merged

feat: feed wrapping #4677

merged 44 commits into from
Sep 19, 2024

Conversation

nugaon
Copy link
Member

@nugaon nugaon commented May 15, 2024

This feature drops the JSON structure with CAC reference and timestamp in feed payload in order to be capable of wrapping a whole content addressed chunk instead. The feed payload resolution is backward compatible by checking the span + ts + swarmHash size in content and trying to retrieve the content under swarmHash.

Because the SOC content needs to be resolved in one request and there is no ts attribute of a feed anymore, the Feed API must be updated and it has breaking change on its GET endpoint. Additionally, at parameter is no longer in use at sequential feeds.

Since now feed endpoint can return content bigger than one chunk and that can be erasure coded, erasure-code related request headers have been added.
Some application may use this endpoint to simply get the latest index of a sequential feed, for that new header is introduced swarm-only-root-chunk which is a boolean type and if it is true then Bee does not retrieve the whole chunk tree.

GET /soc/{owner}/{id} method is introduced with required path parameters owner and id which are hex encoded strings.
This new method handles root chunk as a Single Owner Chunk by default and resolves and returns content similarly as GET /feed. Both endpoints return newly introduced header swarm-soc-signature which encodes the signature of the Single Owner Chunk in question.

  • init joiner by root chunk
  • update bootstrap
  • update bzz and feeds api

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

@nugaon nugaon force-pushed the feat/feed-wrapping branch from 2e082ee to abe6896 Compare May 22, 2024 09:37
@nugaon nugaon marked this pull request as ready for review June 28, 2024 16:42
Copy link
Member

@zelig zelig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must go through a SWIP
is direct resolutiion into content really needed? wouldnt it be much simpler if this was done in 2 steps on cllient side?

extensive documentation, examples should accompany release of this feature
thorough test of backward compatibility

pkg/file/joiner/joiner.go Outdated Show resolved Hide resolved
pkg/file/loadsave/loadsave.go Outdated Show resolved Hide resolved
@nugaon
Copy link
Member Author

nugaon commented Sep 17, 2024

This must go through a SWIP

Wrote SWIP and created PR for that

is direct resolutiion into content really needed? wouldnt it be much simpler if this was done in 2 steps on cllient side?

This is desired to eliminate; the content can be resolved by only 1 request.
Moreover, with bigger content there is no API functionality to pass a whole chunk to resolve (and that way would be more complex)

@nugaon nugaon requested a review from zelig September 17, 2024 12:28
@martinconic martinconic requested a review from janos September 17, 2024 12:54
pkg/api/bzz.go Outdated Show resolved Hide resolved
@@ -64,6 +64,14 @@ func (s *Service) feedGetHandler(w http.ResponseWriter, r *http.Request) {
queries.At = time.Now().Unix()
}

headers := struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep the queries above with At and After ? Also bellow there is a lookup using this queries.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it may be needed for epoch based feeds.
I created an issue about that #4830

@Cafe137
Copy link

Cafe137 commented Sep 19, 2024

Please add the new headers to corsHandler allowedHeaders

@nugaon nugaon requested a review from martinconic September 19, 2024 09:43
@nugaon nugaon merged commit 198d41f into master Sep 19, 2024
14 checks passed
@nugaon nugaon deleted the feat/feed-wrapping branch September 19, 2024 12:45
istae added a commit that referenced this pull request Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants