Skip to content

Merge experiments onto main 2025-05-28 #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
May 28, 2025
Merged

Merge experiments onto main 2025-05-28 #35

merged 31 commits into from
May 28, 2025

Conversation

@wavewave
Copy link
Collaborator

@wavewave wavewave commented May 28, 2025 •
edited
Loading

Highlight:

  • Recent single proxy server implementation
  • worker-close gRPC call
  • Refactoring and renaming packages
  • Instrument app with more stats and measurement

sjoerdvisscher and others added 30 commits May 28, 2025 11:22
@sjoerdvisscher @wavewave
Setting extra ghc options in the worker
8c17c26 
Change-Id: Id93c9810571afbcea75c223d2552c4b5ff975c73
@sjoerdvisscher @wavewave
Add Instrument UI for setting extra GHC options
83367d9 
Change-Id: Ife42c62580a8fe8d6c4aa8745044a65ef5cdef96
@sjoerdvisscher @wavewave
Add cabal.project.local to .gitignore
0d05d80 
Change-Id: I5e7c220131adc7c702a0ffed69285c3d80eb0b4e
@sjoerdvisscher @wavewave
Make genproto.sh work in nix
60d72cf 
Change-Id: I23fbcdd7c2257bb62f3e9e46f7da77904a3786b0
@sjoerdvisscher @wavewave
Make it easier to report multiple memory measurements
13a8c2c 
Change-Id: Idbb1fd14857444e09ff20c056631ff4c61f8deb8
@sjoerdvisscher @wavewave
Make Cache available to mkStats
c9b1c9d 
Change-Id: I39df49687cb2a3b89c77a20654547f264ed635c8
@sjoerdvisscher @wavewave
Fix picking up new worker sessions
db2eafa 
Change-Id: I3b7c2aaff96f53f18fbda02062c7a3566e088edf
@sjoerdvisscher @wavewave
Fix import sorting
ac427e8 
Change-Id: I25ec097c2f4edfeb032ba5593b6dc150a35a6da1
@wavewave
Rename worker to ghc-worker
fac7cdd 
Change-Id: I39b644571421d564928a8ddb5d7fb5f027dede66
@wavewave
Remove BUCK file
663856a 
Change-Id: Icdfb6b5e69733d83eeb78f597e510834569511cc
@wavewave
Plumbing WorkerMode to forkCentralGhc
16c51cc 
Change-Id: If9586b346421d7f9bd0f8ed594fd1e50ffbbda0d
@wavewave
Record trace_id and worker_spec id extracted server socket path.
84b53b0 
Change-Id: Id9b33a845272375869a3ccbe790ea102e6181c28
@wavewave
Always use Spawn mode.
c9f073a 
projectSocketDirectory is configured by
Multi = {trace_id}-{worker_spec_id}
Single = {trace_id}

Change-Id: Ia9d1edf6930ee5c14e603b6a7e9c5f5dd54fd75b
@wavewave
Rename ghc-persistent-worker-plugin to buck-worker-internal
b973423 
Change-Id: I3fedc1ea8b84b1a8094eaaedbd2885b08b81f058
@wavewave
Split out buck-worker-types, buck-worker-proto
13ae2f7 
Now we can separate things which does not depend either on GHC or on proto.

Change-Id: I5d22262023d8955ed5050bb99db8e9143b1c8a16
@wavewave
buck-worker binary is now divided to buck-proxy and ghc-worker
e2e4cd4 
- buck-proxy: called by buck2 directly. spawning ghc-worker internally.
This binary should not depend on GHC library.

- ghc-worker: real GHC worker. socket path is passed by buck-proxy.

Change-Id: Ic7fee083209e9b9352291b5b5f39afd48d40fd25
@wavewave
Remove old persistent worker code.
07a938d 
Change-Id: Ic75eba10b96b7681d834f3a2dd586db4fb951fc8
@wavewave
Add missing LICENSE
3d448c6 
Change-Id: Id985d30a3507c57a78e0c46317442000cad3b1aa
@tek @wavewave
Init unit envs only during the metadata step
877610a 
When the metadata step hadn't been introduced yet, we needed to update
the unit env incrementally, because Buck sends granular dependencies for
each module, and therefore the unit env would often be incomplete.

Since Buck sends all dependencies in metadata requests, we only need to
initialize the unit env once, and never for modules.

Change-Id: I71daeebd3fc9a53dda3bf072b74c5a0c3f47850c
@tek @wavewave
Move old unit env init code to test
937ca1a 
This is now not needed in the worker anymore, but the test has not been
adapted yet.

Change-Id: I9af4a39a678413bfebba228d743eb4f896e83277
@tek @wavewave
Write logs to per-module files in /tmp/ghc-persistent-worker
964ec7f 
Change-Id: If1f5d973ffdd4f9b9198a020368d8d0c4ff0a968
@sjoerdvisscher @wavewave
Fix genproto.sh
d07c170 
Change-Id: I20c4cc4d53cec0a464576f0d1d384b3a4ccc91c5
@sjoerdvisscher @wavewave
Don't crash on missing worker-path
029392e 
Change-Id: I5cd647fd83f21930e13a96e9ba61e4ecbd1e695f
@sjoerdvisscher @wavewave
Pass target name on compile end
6fd2cd4 
Change-Id: I0886e3b07cc1d06af447c2e749f7e53e10a5c65d
@sjoerdvisscher @wavewave
Show lists of active tasks and compiled modules
92888bc 
Next step is to split the UI module in smaller parts

Change-Id: Ic9ed93ccd7a88b968aa3ac6bce5542bbb0701230
@wavewave
Use full name of target for log in order not to conflict names.
f5fdea5 
Change-Id: I2b6278c57b2855e00431521ccf9f7dda0dcae56f
@wavewave
Introduce ModeClose to handle --worker-close
1344fb3 
Change-Id: I7f0d8ebcfd2b273eff43301a868530ae31fdeb88
@wavewave
Terminate ghc-worker asynchronously when closing.
7f6e5da 
Change-Id: I6170aa05ccd76af9f165491f1d43fb9c6bdc6500
@wavewave
Separate out buck-worker-common, which provides common gRPC functions
011e9ac 
change the name ghcServerMethods to fromGrpcHandler

Change-Id: I019fb61e8bcba649e5782fc2187ce512b842335d
@wavewave
Single buck-proxy implementation
f042988 
Only single buck-proxy will be initialized from buck2.
then that buck-proxy will start all the ghc-workers on demand.

No more file system lock or such.

Each ghc-worker is created with a socket:  basename_targetid
where basename = traceid_workerspecid.

Change-Id: If1c9ad972a0cc6556645b9554aad4c82acd29bae
@wavewave
Workaround using transient serial processing
c613cfb 
deadlock problem got work arounded by this non-solution yet.

maxLock = 10. up to 10 first modules, the build is serial.

Change-Id: I76aa6ae579713b05140fba276eb4c9c26b216def
@wavewave wavewave merged commit f6f899b into main May 28, 2025
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wavewave @sjoerdvisscher @tek