Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support PPE profiling for HTTPCache / HTTPCachedFS #356

Merged
merged 5 commits into from
Oct 21, 2024

Conversation

KantaTamura
Copy link
Collaborator

This PR resolves the CachedHttpFs / HttpCache task in #258

Sample program to check tracing

import json
import tempfile
import pytorch_pfn_extras as ppe

from pfio.v2 import HTTPCachedFS, Local
from pfio.cache import HTTPCache
from pfio.testing import make_http_server


def cache_httpcache():
    filename = "cache_httpcache.json"
    tracer = ppe.profiler.get_tracer()

    with make_http_server() as (httpd, port):
        http_cache = f"http://localhost:{port}/"
        cache = HTTPCache(1, http_cache,
                          do_pickle=True, trace=True)

        cache.put(0, b"foo")
        assert b"foo" == cache.get(0)

    dict = tracer.state_dict()
    keys = [event["name"] for event in json.loads(dict['_event_list'])]
    print(keys)

    writer = ppe.writing.SimpleWriter(out_dir="")
    tracer.initialize_writer(filename, writer)
    tracer.flush(filename, writer)


def fs_httpcache():
    filename = "fs_httpcache.json"
    tracer = ppe.profiler.get_tracer()

    local_filename = "testfile"
    content = b"abcdabcd"

    with make_http_server() as (httpd, port), tempfile.TemporaryDirectory() as tmpdir:
        http_cache = f"http://localhost:{port}/"

        with Local(tmpdir, trace=True) as local_fs:
            fs = HTTPCachedFS(http_cache, local_fs)

            with fs.open(local_filename, mode="wb") as fp:
                fp.write(content)
            with fs.open(local_filename, mode="rb") as fp:
                assert fp.read(-1) == content
            with fs.open(local_filename, mode="rb") as fp:
                assert fp.read(-1) == content

    dict = tracer.state_dict()
    keys = [event["name"] for event in json.loads(dict['_event_list'])]
    print(keys)

    writer = ppe.writing.SimpleWriter(out_dir="")
    tracer.initialize_writer(filename, writer)
    tracer.flush(filename, writer)


cache_httpcache()
fs_httpcache()
  • HTTPCache output
['pfio.cache.http.conn:put:request', 'pfio.cache.http.conn:put', 'pfio.cache.http:put', 'pfio.cache.http.conn:get:request', 'pfio.cache.http.conn:get', 'pfio.cache.http:get']

rendering of the output json (cache_httpcache.json) file with chrome://tracing is shown below
image

  • HTTPCachedFS output
['pfio.v2.Local:open', 'pfio.v2.http_cache:open', 'pfio.v2.Local:write', 'pfio.v2.Local:exit-context', 'pfio.v2.http_cache:open', 'pfio.cache.http.conn:get:request', 'pfio.cache.http.conn:get', 'pfio.v2.Local:stat', 'pfio.v2.Local:open', 'pfio.v2.Local:read', 'pfio.v2.Local:exit-context', 'pfio.cache.http.conn:put:request', 'pfio.cache.http.conn:put', 'pfio.v2.http_cache:load_file', 'pfio.v2.http_cache:read', 'pfio.v2.http_cache:open', 'pfio.cache.http.conn:get:request', 'pfio.cache.http.conn:get', 'pfio.v2.http_cache:load_file', 'pfio.v2.http_cache:read']

rendering of the output json (fs_httpcache.json) file with chrome://tracing is shown below
image

Signed-off-by: KantaTamura <[email protected]>
Signed-off-by: KantaTamura <[email protected]>
Copy link
Member

@k5342 k5342 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor comment

pfio/v2/fs.py Outdated
@@ -154,6 +155,11 @@ def is_forked(self):
assert hasattr(self, 'pid')
return self.pid != os.getpid()

@property
def is_trace(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps is_traced is better grammatically; what do you think?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so

@KantaTamura KantaTamura requested a review from k5342 October 21, 2024 06:00
@k5342 k5342 merged commit 3078de4 into pfnet:master Oct 21, 2024
7 checks passed
@k5342 k5342 mentioned this pull request Oct 21, 2024
7 tasks
@kuenishi kuenishi added this to the 2.9.0 milestone Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants