Skip to content

Commit

Permalink
Set remote cache version and backend type once in compilation metrics
Browse files Browse the repository at this point in the history
Summary:
X-link: pytorch/pytorch#141707

This is causing FbFxGraphRemoteCache.init to no longer be idempotent, i.e. only safe to call once per compile. AOTAutogradCache initializes a new remote cache for the forward and the backward.
Technically, we could make AOTAutogradCache smart and globally thread through a single FbFxGraphRemoteCache everywhere. But there's no reason to do so, as this class is just the handle to access the cache. Plus, it's very brittle for FbFxGraphRemoteCache to not be safe to call multiple times.

(Same problem, different fix of D66502138)
ghstack-source-id: 256106154
exported-using-ghexport

Reviewed By: ezyang

Differential Revision: D66508492

fbshipit-source-id: 014722ba2cfbf9d30359e1eaa39062ad35cd907d
  • Loading branch information
jamesjwu authored and facebook-github-bot committed Dec 3, 2024
1 parent 82a8a60 commit c07f33b
Showing 1 changed file with 20 additions and 0 deletions.
20 changes: 20 additions & 0 deletions userbenchmark/dynamo/dynamobench/_dynamo/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -996,6 +996,24 @@ def safe_str(item: Any) -> str:
return ",".join(safe_str(item) for item in metric)

structured_logging_overhead_s = torch._logging.get_structured_logging_overhead()

if torch._inductor.utils.should_use_remote_fx_graph_cache():
try:
from torch._inductor.fb.remote_cache import (
FbRemoteFxGraphCache,
REMOTE_CACHE_VERSION,
)
except ModuleNotFoundError:
REMOTE_CACHE_VERSION = None
inductor_fx_remote_cache_backend_type = None

remote_cache_version = REMOTE_CACHE_VERSION
backend = FbRemoteFxGraphCache.get_remote_backend()
inductor_fx_remote_cache_backend_type = type(backend).__name__
else:
inductor_fx_remote_cache_backend_type = None
remote_cache_version = None

common_metrics = {
"compile_id": str(torch._guards.CompileContext.current_compile_id()),
"start_time_us": start_time_ns // 1000,
Expand All @@ -1013,6 +1031,8 @@ def safe_str(item: Any) -> str:
"inductor_fx_remote_cache_miss_keys": _convert_collection_to_str(
"inductor_fx_remote_cache_miss_keys"
),
"remote_cache_version": remote_cache_version,
"inductor_fx_remote_cache_backend_type": inductor_fx_remote_cache_backend_type,
}

# TODO: The following are legacy fields, populated from the fields that replace
Expand Down

0 comments on commit c07f33b

Please sign in to comment.