Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PoC] Provide common make files #34660

Closed
wants to merge 6 commits into from
Closed

Conversation

phil-davis
Copy link
Contributor

@phil-davis phil-davis commented Mar 4, 2019

See discussion below. This PR is being left here for now, until we get back to designing a better solution.

Description

Provide various *.mk files in core that can be included by app Makefile so that chunks of make code do not have to be repeated in every app.

check-composer.mk - the common block of code that gives the warning if composer does not exist. This is included by other includes that care about composer.
check-npm.mk - the common block of code that gives the warning if npm does not exist. This is included by other includes that care about npm.
clean.mk - common clean and clean_deps targets. Other clean targets are added by other include files.
dist.mk - the "standard" way of making the tarball, and of doing a clean dist
help.mk - the "standard" way that help output is done these days in apps
test-acceptance.mk - the make targets for acceptance tests and their dependency rules
test-js.mk - the JS testing make targets and their dependency rules
test-php.mk - the PHP coding standard, analysis and unit test make targets and their dependency rules
test-all.mk - a convenience that includes all 3 test-*.mk files - so "standard" apps can just include this to get "the works" (now I am not so sure this is useful - not many apps have JS tests, some of those do or do not have acceptance tests,... so there is not really a "common" set of PHP+JS+acceptance tests)
vendor-bin.mk - the dependency rules for getting the vendor and vendor-bin stuff happening and cleaned. This is included by other includes that need it.

Related Issue

owncloud/QA#604

Motivation and Context

How Has This Been Tested?

Local runs of JS unit, PHP unit and acceptance tests from various app repos.
PoC drone CI runs from activity, password_policy and... apps.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Database schema changes (next release will require increase of minor version instead of patch)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:

Open tasks:

  • Backport (if applicable set "backport-request" label and remove when the backport was done)

@phil-davis phil-davis self-assigned this Mar 4, 2019
@phil-davis phil-davis changed the title Provide common make files [WIP] Provide common make files Mar 4, 2019
@codecov
Copy link

codecov bot commented Mar 4, 2019

Codecov Report

Merging #34660 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master   #34660   +/-   ##
=========================================
  Coverage     65.25%   65.25%           
  Complexity    18458    18458           
=========================================
  Files          1207     1207           
  Lines         69895    69895           
  Branches       1280     1280           
=========================================
  Hits          45608    45608           
  Misses        23915    23915           
  Partials        372      372
Flag Coverage Δ Complexity Δ
#javascript 53.04% <ø> (ø) 0 <ø> (ø) ⬇️
#phpunit 66.65% <ø> (ø) 18458 <ø> (ø) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03e6987...c358af4. Read the comment docs.

@codecov
Copy link

codecov bot commented Mar 4, 2019

Codecov Report

Merging #34660 into master will increase coverage by <.01%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #34660      +/-   ##
============================================
+ Coverage     65.38%   65.38%   +<.01%     
  Complexity    18592    18592              
============================================
  Files          1213     1213              
  Lines         70408    70408              
  Branches       1295     1295              
============================================
+ Hits          46035    46036       +1     
+ Misses        23999    23998       -1     
  Partials        374      374
Flag Coverage Δ Complexity Δ
#javascript 52.85% <ø> (ø) 0 <ø> (ø) ⬇️
#phpunit 66.83% <ø> (ø) 18592 <ø> (ø) ⬇️
Impacted Files Coverage Δ Complexity Δ
apps/files_trashbin/lib/Expiration.php 98.27% <0%> (+1.72%) 29% <0%> (ø) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1fc5b0...41347fc. Read the comment docs.

build/rules/dist.mk Outdated Show resolved Hide resolved
@phil-davis
Copy link
Contributor Author

phil-davis commented Mar 4, 2019

Example PRs that pass CI using this code from core:
owncloud/password_policy#207
owncloud/activity#714
owncloud/guests#311
https://github.com/owncloud/ransomware_protection/pull/175

Questions:

  • what to put in the default doc_files list?
  • what to put in the default src_files list?
  • what to do about app:check-code ? put it in a common test-app-check-code.mk in core? or just leave that block in the Makefile of apps that happen to have it? or remove it everywhere? or?
  • some apps also make a .tar.bz2 in addition to .tar.gz - is that needed for something special? or just old history?

@phil-davis phil-davis changed the title [WIP] Provide common make files Provide common make files Mar 4, 2019
@phil-davis phil-davis force-pushed the provide-common-makefiles branch 2 times, most recently from ea2aa1a to c485ae3 Compare March 4, 2019 11:48
CLEAN_HAS_BEEN_INCLUDED=true

.PHONY: clean-deps
clean-deps: ## Clean all dependencies
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the clean rules need to be part of each module

the dist rules module has clean-dist, the npm deps has clean-npm-deps, etc

you can use a generic variable to which every module can add their clean rules to the main "clean" target.
see https://github.com/owncloud/customgroups/blob/master/Makefile#L25 and https://github.com/owncloud/customgroups/blob/master/rules/deps.mk#L16

@individual-it
Copy link
Member

don't think we need test-all.mk

@PVince81
Copy link
Contributor

PVince81 commented Mar 5, 2019

there is one drawback with this: app builds will now depend on the core git checkout that was used.

which means that if we fix makefile rules in core we might need to do another release of apps.

we'll need to settle on a single OC version when buildling apps then, we could say it always needs to be from the git checkout of the latest stable release.

note that we had an internal POC from @patrickjahns where we build apps using a docker based on the OC-10.0.10 release. this approach would not work any more because the release tarball does not contain the makefile rules.

might need a different approach then, like putting the makefile rules in a common repo like the coding standards one ?

vendor/bamarni/composer-bin-plugin: composer.lock
composer install

clean_deps_rules+=clean-vendor clean-vendor-bin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, you aready have this

@PVince81
Copy link
Contributor

PVince81 commented Mar 5, 2019

ideas for later: a rules file with checkers.

as this will be used by apps, we can check that many things are already properly in place in the dist tarball, like info.xml, changelog.md exists, etc

build/rules/dist.mk Outdated Show resolved Hide resolved
@phil-davis
Copy link
Contributor Author

which means that if we fix makefile rules in core we might need to do another release of apps.

we'll need to settle on a single OC version when buildling apps then, we could say it always needs to be from the git checkout of the latest stable release.

If the problem caused an app tarball to be released with missing/wrong stuff, then fir sure the app has to be released again. But that is already the case if there is a problem in the local Makefile

The default "recommended" action when making an app release tarball would be to have the end of the current core release branch checked out, e.g. core stable10. That way you get the included *.mk as they were at the last core release, plus any recent fixes that will end up in the next release tag. But also in most cases core master will have exactly the same *.mk as core stable10 so it will not make a real difference.

@phil-davis
Copy link
Contributor Author

@PVince81 IMO this now matches the "standard" Makefile code in the apps, and has flexibility to deal with finding various files. It can reduce the Makefile code duplication across the apps.

If we find we need more flexibility, then it is easy to add ifdef etc conditional statements so that it continues to work for apps that use it, but can also handle some more unusual thing we find in "an app I haven't thought of yet".

Or if the app really is "weird", then it does not have to include all the *.mk files, and can continue to "do its thing its own way" until we really sort it out. I don't believe there are any apps like that.

If we some day want to migrate to putting this in a separate repo, or use some completely different build tool, then that will be possible and should be easier because we have this common set of knowledge about what build and test services are commonly used by apps.

Please review again.

@PVince81
Copy link
Contributor

PVince81 commented Mar 5, 2019

If we some day want to migrate to putting this in a separate repo, or use some completely different build tool, then that will be possible and should be easier because we have this common set of knowledge about what build and test services are commonly used by apps.

Would be better to decide this right away. We're going to need to touch every app already for this change to make sense. If later on we introduce a common location, we'd need to touch all apps yet again.

@phil-davis
Copy link
Contributor Author

OK, so how do we decide?

@patrickjahns patrickjahns self-requested a review March 5, 2019 15:13
@patrickjahns
Copy link
Contributor

patrickjahns commented Mar 5, 2019

Please do not make apps depend on core makefiles.

If we are looking for a central part for makefiles, please put them in a dedicated repository to have clear separation of concerns.
Core repository should be about ownCloud core and not about a build/makefile infrastructure for apps

Example: https://github.com/cloudposse/build-harness

@phil-davis
Copy link
Contributor Author

I removed the "To review" label.
This PR is now a "demonstration" of the makefile targets that could be put somewhere common, and a way that those could be split into logical parts in *.mk files.
We need to research a good solution for where and how to store and retrieve this common build infrastructure - issue owncloud/QA#604
And then perhaps be able to efficiently copy-paste some structure and code from here into the selected separate repo/tool/whatever.

Copy link
Contributor

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👎 for now (to prevent accidental merge)

@PVince81 PVince81 added this to the backlog milestone Mar 11, 2019
@phil-davis phil-davis force-pushed the provide-common-makefiles branch from aec125a to 4e5bb1d Compare March 24, 2019 06:12
@phil-davis phil-davis changed the title Provide common make files [PoC] Provide common make files Mar 24, 2019
@phil-davis phil-davis force-pushed the provide-common-makefiles branch from 4e5bb1d to 41347fc Compare April 3, 2019 05:18
@phil-davis
Copy link
Contributor Author

Closing see comment owncloud/richdocuments#253 (comment)
There will be a different approach "some day"

@phil-davis phil-davis closed this Jul 20, 2019
@phil-davis phil-davis deleted the provide-common-makefiles branch June 18, 2020 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants