Download links all broken?

newer
[GSoC2020][Boost.Real] Greeting

John Maddock

30 Mar 2020 30 Mar '20

5:36 p.m.

All the download links from www.boost.org appear to be currently broken - which is to say no one can download Boost right now. Ideas or fixes anyone? Thanks, John. -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Show replies by date

Rene Rivera

30 Mar 30 Mar

5:40 p.m.

On Mon, Mar 30, 2020 at 12:37 PM John Maddock via Boost < boost@lists.boost.org> wrote:

...

All the download links from www.boost.org appear to be currently broken - which is to say no one can download Boost right now.

Ideas or fixes anyone?

1. We know. 2. It will all start working again tomorrow. As we surpassed the DL limit for the month. 3. Someone could set up a distributed web cache of some sort to fix this in the long run? 4. People should stop using bintray for CI test jobs. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Julien Blanc

7:28 p.m.

Le lundi 30 mars 2020 à 12:40 -0500, Rene Rivera via Boost a écrit :

...

On Mon, Mar 30, 2020 at 12:37 PM John Maddock via Boost < boost@lists.boost.org> wrote:

...
All the download links from www.boost.org appear to be currently broken - which is to say no one can download Boost right now.

Ideas or fixes anyone?

1. We know. 2. It will all start working again tomorrow. As we surpassed the DL limit for the month.

Works for me as of today, although I don't know why.

...

3. Someone could set up a distributed web cache of some sort to fix this in the long run?

The usual solution to this is bittorrent. Providing magnet links can be done quite easily (i can do it if there is some need for it). Regards, Julien

Raffi Enficiaud

8:37 p.m.

On 30.03.20 19:40, Rene Rivera via Boost wrote:

...

On Mon, Mar 30, 2020 at 12:37 PM John Maddock via Boost < boost@lists.boost.org> wrote:

4. People should stop using bintray for CI test jobs.

you mean this for instance? https://github.com/conan-community/conan-boost/blob/release/1.71.0/conanfile... or this? https://github.com/conan-io/conan-center-index/blob/master/recipes/boost/all... TBH, if a package manager is pointing to bintray, and if there is a limit on the download volume, something will fail (whether this is a CI job or a human being pulling the libs). Maybe it is time to reconsider this, for instance: https://help.github.com/en/github/administering-a-repository/about-releases In particular: "Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage." Raffi

Rene Rivera

8:43 p.m.

On Mon, Mar 30, 2020 at 3:38 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...

On 30.03.20 19:40, Rene Rivera via Boost wrote:

...
On Mon, Mar 30, 2020 at 12:37 PM John Maddock via Boost < boost@lists.boost.org> wrote:

4. People should stop using bintray for CI test jobs.

you mean this for instance?

https://github.com/conan-community/conan-boost/blob/release/1.71.0/conanfile...

or this?

https://github.com/conan-io/conan-center-index/blob/master/recipes/boost/all...

No. Which is why I used the word "test" in my statement. TBH, if a package manager is pointing to bintray, and if there is a

...

limit on the download volume, something will fail (whether this is a CI job or a human being pulling the libs).

Yep. Maybe it is time to reconsider this, for instance:

...

https://help.github.com/en/github/administering-a-repository/about-releases

In particular: "Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage."

Someone would have to step up and write the automation scripts to deliver the same functionality as bintray. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Raffi Enficiaud

8:48 p.m.

On 30.03.20 22:43, Rene Rivera via Boost wrote:

...

On Mon, Mar 30, 2020 at 3:38 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...

...
In particular: "Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage."

Someone would have to step up and write the automation scripts to deliver the same functionality as bintray.

I am an automation freak. Where should it be plugged to?

Rene Rivera

8:58 p.m.

On Mon, Mar 30, 2020 at 3:48 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...

On 30.03.20 22:43, Rene Rivera via Boost wrote:

...
On Mon, Mar 30, 2020 at 3:38 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...
...
In particular: "Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage."

Someone would have to step up and write the automation scripts to deliver the same functionality as bintray.

I am an automation freak. Where should it be plugged to?

Some place here.. < https://github.com/boostorg/release-tools/blob/develop/ci_boost_release.py#L...

...

Information on the tail end of the process is here: < https://github.com/boostorg/wiki/wiki/Releases%3A-Preparing> -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Alexander Grund

31 Mar 31 Mar

6:46 a.m.

...

...
I am an automation freak. Where should it be plugged to?

Some place here.. < https://github.com/boostorg/release-tools/blob/develop/ci_boost_release.py#L... Information on the tail end of the process is here: < https://github.com/boostorg/wiki/wiki/Releases%3A-Preparing>

FWIW: I recently set up automated releases on Github via Github Actions which is triggered on tags. You can do pretty much anything and then just hand it over to the API. I think the same is very much possible for general boost releases, so if this is set up in the super-repo and triggered on tags it would likely do what we want. See https://github.com/boostorg/nowide/blob/develop/.github/workflows/release.ym... Note how I convert the code and create documentation before tarring and uploading. In case of Boost it would likely be a submodule update, then tar. Regards, Alex

Raffi Enficiaud

11:54 a.m.

On 30.03.20 22:58, Rene Rivera via Boost wrote:

...

On Mon, Mar 30, 2020 at 3:48 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...
On 30.03.20 22:43, Rene Rivera via Boost wrote:

...
On Mon, Mar 30, 2020 at 3:38 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

...
...
In particular: "Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage."

Someone would have to step up and write the automation scripts to deliver the same functionality as bintray.

I am an automation freak. Where should it be plugged to?

Some place here.. < https://github.com/boostorg/release-tools/blob/develop/ci_boost_release.py#L...

...
Information on the tail end of the process is here: < https://github.com/boostorg/wiki/wiki/Releases%3A-Preparing>

This is the first step: https://github.com/boostorg/release-tools/pull/16 Then it is a matter of preference: * either I port the other files to python3 with some risk of breaking things, and integrate things at the python level * either I call this script as an external process Best, Raffi

Peter Barker

30 Mar 30 Mar

5:41 p.m.

On Mon, 30 Mar 2020 at 18:37, John Maddock via Boost wrote:

...

All the download links from www.boost.org appear to be currently broken - which is to say no one can download Boost right now.

Ideas or fixes anyone?

Thanks, John.

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

There was this recent thread about it: http://boost.2283326.n4.nabble.com/Boost-download-links-seems-to-be-down-td4... Regards, Pete

John Maddock

6:02 p.m.

...

There was this recent thread about it: http://boost.2283326.n4.nabble.com/Boost-download-links-seems-to-be-down-td4...

Ah, got it, and my word we're using a lot of bandwidth! Thanks, John. -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Emil Dotchevski

6:10 p.m.

CI should use tools/boostdep/depinst/depinst.py to grab only the needed repos from github. On Mon, Mar 30, 2020 at 11:03 AM John Maddock via Boost < boost@lists.boost.org> wrote:

...

...
There was this recent thread about it:

http://boost.2283326.n4.nabble.com/Boost-download-links-seems-to-be-down-td4...

Ah, got it, and my word we're using a lot of bandwidth!

Thanks, John.

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Rene Rivera

6:13 p.m.

Top post ;-) On Mon, Mar 30, 2020 at 1:10 PM Emil Dotchevski via Boost < boost@lists.boost.org> wrote:

...

CI should use tools/boostdep/depinst/depinst.py to grab only the needed repos from github.

Some of us test against whole Boost releases and the release archives are the easiest to deal with. Yes, I'm partly to blame :-) But in my defense, I did make changes to use git clones and the release tags. Even if it cost me many minutes of CI time to do it. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Mateusz Loskot

6:27 p.m.

On Mon, 30 Mar 2020 at 20:13, Rene Rivera via Boost wrote:

...

On Mon, Mar 30, 2020 at 1:10 PM Emil Dotchevski via Boost wrote:

...
CI should use tools/boostdep/depinst/depinst.py to grab only the needed repos from github.

Some of us test against whole Boost releases and the release archives are the easiest to deal with. Yes, I'm partly to blame :-) But in my defense, I did make changes to use git clones and the release tags. Even if it cost me many minutes of CI time to do it.

Is it possible to mirror release archives on GitHub Releases or SF.net or both? Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Andrey Semashev

7:04 p.m.

On 2020-03-30 21:13, Rene Rivera via Boost wrote:

...

Some of us test against whole Boost releases and the release archives are the easiest to deal with. Yes, I'm partly to blame :-) But in my defense, I did make changes to use git clones and the release tags. Even if it cost me many minutes of CI time to do it.

Did you try using (a) parallel checkouts and (b) shallow git clone?

Rene Rivera

7:10 p.m.

On Mon, Mar 30, 2020 at 2:05 PM Andrey Semashev via Boost < boost@lists.boost.org> wrote:

...

On 2020-03-30 21:13, Rene Rivera via Boost wrote:

...
Some of us test against whole Boost releases and the release archives are the easiest to deal with. Yes, I'm partly to blame :-) But in my

defense, I

...
did make changes to use git clones and the release tags. Even if it cost me many minutes of CI time to do it.

Did you try using (a) parallel checkouts and (b) shallow git clone?

Didn't try parallel... But did try shallow. For submodules, shallow clones are not supported by GitHub. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Peter Dimov

8:43 p.m.

John Maddock wrote:

...

All the download links from www.boost.org appear to be currently broken - which is to say no one can download Boost right now.

Ideas or fixes anyone?

Our Bintray quota is apparently 32 TB, which isn't quite enough. I second the suggestion that people should stop downloading Boost in their CI. It's not clear how we can effect such a change though. :-)

Peter Dimov

31 Mar 31 Mar

4:09 p.m.

John Maddock wrote:

...

Ideas or fixes anyone?

Other ideas notwithstanding, one relatively trivial thing we could do is stop publishing .zip and .tar.gz, and switch from .tar.bz2 to .tar.xz. This will cut the bandwidth in half. (It's only a palliative measure, which doesn't mean it can't help.)

Emil Dotchevski

5:30 p.m.

On Tue, Mar 31, 2020 at 9:10 AM Peter Dimov via Boost wrote:

...

Other ideas notwithstanding, one relatively trivial thing we could do is stop publishing .zip and .tar.gz, and switch from .tar.bz2 to .tar.xz. This will cut the bandwidth in half. (It's only a palliative measure, which doesn't mean it can't help.)

It will help tremendously, as it will break all the links people have in CI scripts. :)

Mateusz Loskot

5:45 p.m.

On Tue, 31 Mar 2020 at 19:31, Emil Dotchevski via Boost wrote:

...

On Tue, Mar 31, 2020 at 9:10 AM Peter Dimov via Boost wrote:

...
Other ideas notwithstanding, one relatively trivial thing we could do is stop publishing .zip and .tar.gz, and switch from .tar.bz2 to .tar.xz. This will cut the bandwidth in half. (It's only a palliative measure, which doesn't mean it can't help.)

It will help tremendously, as it will break all the links people have in CI scripts. :)

They will arrive here reporting the issues. A rare opportunity to stream direct education :) Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Peter Dimov

6:01 p.m.

Emil Dotchevski wrote:

...

On Tue, Mar 31, 2020 at 9:10 AM Peter Dimov via Boost wrote:

...
Other ideas notwithstanding, one relatively trivial thing we could do is stop publishing .zip and .tar.gz, and switch from .tar.bz2 to .tar.xz. This will cut the bandwidth in half. (It's only a palliative measure, which doesn't mean it can't help.)

It will help tremendously, as it will break all the links people have in CI scripts. :)

Not to spoil the joke, but it won't. Nobody has links to 1_73_0 yet, so nothing will be broken.

Raffi Enficiaud

7:08 p.m.

On 31.03.20 18:09, Peter Dimov via Boost wrote:

...

John Maddock wrote:

...
Ideas or fixes anyone?

Other ideas notwithstanding, one relatively trivial thing we could do is stop publishing .zip and .tar.gz, and switch from .tar.bz2 to .tar.xz. This will cut the bandwidth in half. (It's only a palliative measure, which doesn't mean it can't help.)

Something I dream about: placing documentation to its own separate archive. Raffi

1860

Age (days ago)

1861

Last active (days ago)

List overview

Download

21 comments

10 participants

participants (10)

Alexander Grund
Andrey Semashev
Emil Dotchevski
John Maddock
Julien Blanc
Mateusz Loskot
Peter Barker
Peter Dimov
Raffi Enficiaud
Rene Rivera