[RFC] Travis adding much stricter limits on free users

[jameslamb]

Noticed in dask/community#107 this morning. Travis is putting much much stricter limits on free-tier use of its CI services.

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

For those of you who have been building on public repositories (on travis-ci.com, with no paid subscription), we will upgrade you to our trial (free) plan with a 10K credit allotment (which allows around 1000 minutes in a Linux environment).

We will be offering an allotment of OSS minutes that will be reviewed and allocated on a case by case basis. Should you want to apply for these credits please open a request with Travis CI support stating that you’d like to be considered for the OSS allotment.

The blog post also makes it sound like macOS jobs will no longer be available for free-tier users, but honestly the blog post's discussion of that is very confusing.

I'm not sure yet what we should do with out existing Travis jobs, need to do some more research on how long they take to run and what that means in relation to these limits. 😫

[StrikerRUS]

Thanks a lot for bringing this up @jameslamb !

Yeah, seems that macOS builds will be paid...
Also, please notice we use travis-ci.org, not .com. Looks like this also matters.

[StrikerRUS]

https://docs.travis-ci.com/user/billing-faq#what-if-i-am-building-open-source

Each of the Travis CI Plans can have an amount of special OSS credits per month assigned to run builds only on public repositories. To find out more about it please contact the Travis CI support team.

Very "informative"! 😃

Love for OSS has gone: https://github.com/travis-ci/travis-web/pull/2569/files

[StrikerRUS]

Also, please notice we use travis-ci.org, not .com. Looks like this also matters.

OK, org will be archived 31st December: https://travis-ci.community/t/org-to-com-migration-deadline/10260.

[dgw]

Love for OSS has gone: https://github.com/travis-ci/travis-web/pull/2569/files

I found this thread while googling for more info about the "OSS only credits". That linked change is disheartening.

Looks like my own OSS project will need to discuss the pros and cons of switching over to GitHub Actions instead of migrating our travis-ci.org repos to travis-ci.com. If you go through the process of requesting OSS credits, I'd love to see the results here. It would help us with our own decision-making process!

[jameslamb]

Just got this error on a personal project of mine that runs on travis-ci.org

Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting.

[StrikerRUS]

So, what's the plan?

Mark Travis CI as non-required for the transfer period. Transfer LightGBM project from org to com domain. Write a letter to Travis support team kindly asking them for open source project points. Choose the most basic (and free plan) with ideally unlimited runs/minutes (but probably with just 1 parallel job) for Linux and Windows. Spend all given tokens for macOS builds. Spend some time looking at our spendings. If everything is OK, set Travis CI required back.

Something like this, right?

[jameslamb]

Ok yes, I agree with this plan!

If you can do the org to com transfer, I can take responsibility for writing to Travis support. What do you think?

[StrikerRUS]

I'm not sure I have enough rights for migrating. Anyway, the first step is to mark Travis CI as non-required for PRs, because after migrating we will face runs cancelation as you've pointed in #3519 (comment).

@guolinke Could you please help with marking Travis checks as non-required? Also, WDYT about this topic in general?

[StrikerRUS]

https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567
https://travis-ci.community/t/extremely-poor-official-communication-of-the-org-shutdown/10568

[jameslamb]

It's really unbelievable.

The official Travis documentation on migrating to .com STILL refers to .com as a beta!

https://docs.travis-ci.com/user/migrate/open-source-repository-migration

[jameslamb]

Two tasks just failed too even start on Travis. I'm hoping it's temporary and not sure if it's related to this issue, but want to note the links here in case we need to come back to them later.

https://travis-ci.org/github/microsoft/LightGBM/jobs/743842061
https://travis-ci.org/github/microsoft/LightGBM/jobs/743842062

An error occurred while generating the build script.

[jameslamb]

I think that we need to start making progress on this, since travis-ci.org will be completely shut off 5 weeks from now.

@guolinke the next two things we need to do require administrator access to the repo, so I think that to get it done either you need to do it or @StrikerRUS and I need to be granted that access (for a short time)

• make Travis checks optional on pull requesttts
• move to travis-ci.com: https://docs.travis-ci.com/user/migrate/open-source-repository-migration

Can you either do these two things or grant us the access to do it? Sorry, I know this is annoying :/

[guolinke]

@jameslamb no problem!

[guolinke]

it needs the approval from microsft, i submitted the request.

[jameslamb]

ok great, thank you very much!

[StrikerRUS]

@guolinke

it needs the approval from microsft, i submitted the request.

Any updates?

[guolinke]

it is still in request:

I will try to contact them for the request.

[guolinke]

still didn't get any response, I think we can move some CIs to azure pipeline or github actions.

[jameslamb]

I think we can move some CIs to azure pipeline or github actions.

Currently, we have the following on every PR

• Azure DevOps: 12 jobs
• GitHub Actions: 17 jobs
• Travis: 19 jobs

We're limited to 20 concurrent jobs on GitHub Actions. So if we add more than 3 additional jobs to GitHub Actions, it will increase total CI time.

I've observed that we only have 3 concurrent jobs on Azure, but I don't know if that's the most we can have or just a configuration we can change.

Is it possible for Microsoft to grant us a lot more concurrent jobs on Azure DevOps, to avoid increasing CI times? Then we could move all Travis jobs there. If that's possible, I think it would be the best outcome. Otherwise, we'll have to choose either longer CI times (which slows down development) or removing some CI jobs (which increases the risk of bugs that result in bad user experience or, for R, being rejected by CRAN).

[guolinke]

@jameslamb actually, I already tried to increase the parallel jobs in Azure pipeline. However, it seems the max job is 10 (cannot increase), for public projects.
Another solution is to use self-hosted agents.

https://docs.microsoft.com/en-us/azure/devops/organizations/billing/buy-more-build-vs?view=azure-devops

[guolinke]

For self-hosted agents, we could setup windows and ubuntu VMs, but seems mac os is not possible in azure. So we can use the 10 microsoft-hosted agents to run macos, and self-hosted agents to run windows/linux.

[guolinke]

I just create two self-hosted pooll, for ubuntu and windows, each with 20 max jobs.
you can have a try:

also cc @StrikerRUS

[StrikerRUS]

@jameslamb

Otherwise, we'll have to choose either longer CI times (which slows down development) or removing some CI jobs (which increases the risk of bugs that result in bad user experience or, for R, being rejected by CRAN).

My general position is that slow but full-covering CI services is better than fast and poor ones.

@guolinke

I just create two self-hosted pooll, for ubuntu and windows, each with 20 max jobs.

Just to clarify: are these pools directly sponsored by Microsoft or maybe in some other way? If not, I think we can find better solution than spending your own money.

[StrikerRUS]

still didn't get any response, I think we can move some CIs to azure pipeline or github actions.

I think that except lint and docs jobs for all other important ones we have twins at Azure Pipeline. So there should be nothing critical in case of we lose Travis 31st December.

[StrikerRUS]

My general position is that slow but full-covering CI services is better than fast and poor ones.

I mean, even jobs with several hours length shouldn't hurt our bi-monthly releases.

[jameslamb]

My general position is that slow but full-covering CI services is better than fast and poor ones.

I mean, even jobs with several hours length shouldn't hurt our bi-monthly releases.

I absolutely agree with this.

I think that except lint and docs jobs for all other important ones we have twins at Azure Pipeline. So there should be nothing critical in case of we lose Travis 31st December.

I could move those to GitHub Actions right now! What do you think?

[guolinke]

Just to clarify: are these pools directly sponsored by Microsoft or maybe in some other way? If not, I think we can find better solution than spending your own money.

It is supported by Microsoft, feel free to use them 😄

[StrikerRUS]

@guolinke OK, got it! 🙂

@jameslamb

I could move those to GitHub Actions right now! What do you think?

Then I think it will be better to migrate all tasks from Travis to new Azure agents.

[jameslamb]

Then I think it will be better to migrate all tasks from Travis to new Azure agents.

Ok! I can try that over the next few days.

[StrikerRUS]

@guolinke

I just create two self-hosted pooll, for ubuntu and windows, each with 20 max jobs.

Could you please elaborate on "20 max jobs"? What does it mean? From the docs it is possible to run unlimited number of jobs for free on self-hosted agent.

[guolinke]

@StrikerRUS The self-hosted agents are running on a scale-set, and I currently set its max size to 20. So the max parallel job is 20.
If more jobs are needed, I can increase the size.

[StrikerRUS]

@guolinke So one agent can execute only one job at a time. Got it! Right now we have 16 Linux jobs which utilize self-hosted sh-ubuntu pool. I believe it is more than enough, thanks a lot!
And does one VM can handle only one agent?

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.