On Wed, Apr 8, 2015 at 7:52 AM, Marshall Clow wrote:

On Wed, Apr 8, 2015 at 7:45 AM, Marshall Clow wrote:

...

On Wed, Apr 8, 2015 at 7:35 AM, Marshall Clow wrote:

...

---------- Forwarded message ---------- From: Date: Wed, Apr 8, 2015 at 1:29 AM Subject: Boost.Python build error in 1.58 RC1, 2 and 3 (OK in Beta 1) To: mclow.lists@gmail.com

Hi Marshall,

Please excuse the direct mail, but my mailing list posting wasn't picked up on by anyone. I believe I've found a problem in Boost.Python that

causes

...

build errors if the user has Python 3.0 or later on their system. This change appears to have been introduced after 1.58 Beta 1.

It appears to be this change:

https://github.com/boostorg/python/commit/a911c17fd6e6532807d3a17d597eb391bd...

...

Which was applied directly to the master branch - never appearing on develop.

That's annoying, can we roll this back and do an rc4? It sure would be nice if boost::python supports Python 3. Tom

On Wed, Apr 8, 2015 at 10:58 AM, Tom Kent wrote:

On Wed, Apr 8, 2015 at 7:52 AM, Marshall Clow wrote:

...

...

It appears to be this change:

https://github.com/boostorg/python/commit/a911c17fd6e6532807d3a17d597eb391bd...

...

...

Which was applied directly to the master branch - never appearing on develop.

That's annoying, can we roll this back and do an rc4? It sure would be nice if boost::python supports Python 3.

It would be nice, but would it work? Would that enable Boost.Python to support Python 3? I thought that this change was to make it work on Py3 (though, apparently it was never tested). [ Looking at the code, I'm pretty sure that reverting this change would not affect Py2 use ] -- Marshall

It's Python 3.3.0. In the file c:\Python33\include\fileutils.h there's only the following two fopen functions: PyAPI_FUNC(FILE *) _Py_wfopen( const wchar_t *path, const wchar_t *mode); PyAPI_FUNC(FILE*) _Py_fopen( PyObject *path, const char *mode); It appears there's only a wide char version for Windows. Mark. -- View this message in context: http://boost.2283326.n4.nabble.com/Fwd-Boost-Python-build-error-in-1-58-RC1-... Sent from the Boost - Dev mailing list archive at Nabble.com.

On Thu, Apr 9, 2015 at 6:00 AM, Stefan Seefeld wrote:

*Sigh*. It apears the new API was introduced here: https://hg.python.org/cpython/rev/ef889c3d5dc6 That's the price to pay for using non-public APIs...

I have just committed

https://github.com/boostorg/python/commit/3e405b6fd5db5615bbef241763de070118... , which should preserve the new functionality for Python 3.4, and use a little workaround for Python 3 < 3.4

Stefan -- PLEASE STOP COMMITTING DIRECTLY TO MASTER! That's not how we do development on Boost. Commit your changes to the develop branch, give the testers a chance to run, and *then*, if everything seems OK, merge from develop to master. -- Marshall

On Thu, Apr 9, 2015 at 10:01 AM, Olaf van der Spek wrote:

On Thu, Apr 9, 2015 at 3:51 PM, Marshall Clow wrote:

...

PLEASE STOP COMMITTING DIRECTLY TO MASTER!

That's not how we do development on Boost.

Commit your changes to the develop branch, give the testers a chance to run, and *then*, if everything seems OK, merge from develop to master.

Can't this be handled by code / permissions?

Could you explain what you mean by "handled by code / permissions"? While I understand how a continuous integration framework might only merge to master when develop tests are passing satisfactorily, I'm not aware of any way to prevent breaking merges with our current testing approach. --Beman

On 09/04/15 11:40 AM, Beman Dawes wrote:

On Thu, Apr 9, 2015 at 10:01 AM, Olaf van der Spek wrote:

...

On Thu, Apr 9, 2015 at 3:51 PM, Marshall Clow wrote:

...

PLEASE STOP COMMITTING DIRECTLY TO MASTER!

That's not how we do development on Boost.

Commit your changes to the develop branch, give the testers a chance to run, and *then*, if everything seems OK, merge from develop to master. Can't this be handled by code / permissions?

Could you explain what you mean by "handled by code / permissions"?

While I understand how a continuous integration framework might only merge to master when develop tests are passing satisfactorily, I'm not aware of any way to prevent breaking merges with our current testing approach.

The entire issue is a result of me accepting a pull request to master, followed by a fix to an issue only observed on master. In addition I also wasn't fully aware of the use of master essentially as a release branch. While I will be more careful in the future to avoid such mistakes, I also think it might be useful to prevent (forbid) pull requests to master to begin with. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

On Thu, Apr 9, 2015 at 5:40 PM, Beman Dawes wrote:

On Thu, Apr 9, 2015 at 10:01 AM, Olaf van der Spek wrote:

...

On Thu, Apr 9, 2015 at 3:51 PM, Marshall Clow wrote:

...

PLEASE STOP COMMITTING DIRECTLY TO MASTER!

That's not how we do development on Boost.

Commit your changes to the develop branch, give the testers a chance to run, and *then*, if everything seems OK, merge from develop to master.

Can't this be handled by code / permissions?

Could you explain what you mean by "handled by code / permissions"?

While I understand how a continuous integration framework might only merge to master when develop tests are passing satisfactorily, I'm not aware of any way to prevent breaking merges with our current testing approach.

Is it not possible to only allow merges from develop into master? -- Olaf

On Friday 10 April 2015 09:19:48 Rene Rivera wrote:

On Fri, Apr 10, 2015 at 7:40 AM, Olaf van der Spek wrote:

...

Is it not possible to only allow merges from develop into master?

NO. Github doesn't have that kind of permissions control.

FWIW, I don't think limiting maintainers from accessing their libraries is the right way to go in the first place. One of the reasons why we moved to git was to decentralize Boost and allow more freedom to developers. What needs to be changed is the release process which currently relies on the global freeze instead of branching or forking.

On Wednesday 15 April 2015 15:32:12 Vladimir Prus wrote:

On 04/10/2015 05:28 PM, Andrey Semashev wrote:

...

On Friday 10 April 2015 09:19:48 Rene Rivera wrote:

...

On Fri, Apr 10, 2015 at 7:40 AM, Olaf van der Spek wrote:

...

Is it not possible to only allow merges from develop into master?

NO. Github doesn't have that kind of permissions control.

FWIW, I don't think limiting maintainers from accessing their libraries is the right way to go in the first place. One of the reasons why we moved to git was to decentralize Boost and allow more freedom to developers. What needs to be changed is the release process which currently relies on the global freeze instead of branching or forking.

I don't think there's a primarily technical problem. No matter how release process goes, we either need to assume that 'master' branch of any component is good to be released at all times, or coordinate with maintainers of 120 components to determine which revision of 'master' is really good to be released.

You can guess which approach release managers would prefer.

I'm not arguing against taking master as a starting point for a release; it's a reasonable compromise. What I'm saying is that once release process has started there should be no restriction on updating master in submodules (i.e. the freeze), the release should branch off the main code base and get finished regardless of the modifications developers make. It is possible that master is broken at the point of branching - and part of release process is to fix it for the release or revert to a last known good version. But the probability of breakage is greatly reduced with this approach. Note that when I say "branch" above I don't necessarily mean "git branch". In fact, it may be more convenient to use forks to isolate the code to be released from the main code base and use pull requests for patches that need to get into the release.

On Wednesday 15 April 2015 16:22:33 Peter Dimov wrote:

Andrey Semashev wrote:

...

What I'm saying is that once release process has started there should be no restriction on updating master in submodules (i.e. the freeze), the release should branch off the main code base and get finished regardless of the modifications developers make.

Something like this?

http://thread.gmane.org/gmane.comp.lib.boost.devel/247192

Nowadays, I'd no longer advocate for the odd branch naming where develop corresponds to master - it's too late for that. But the general idea still stands.

I suspect we won't be able to get away with just the superproject branching. If instead of reverting a broken submodule we want to fix it, and the maintainer is unresponsive for some reason, we need either a branch in the submodule or a fork of it. Since the branch can be modified by someone else than a release manager, the fork seems more preferred. The release branch of superproject would have to refer to the fork in this case.

On Wednesday 15 April 2015 18:14:35 Peter Dimov wrote:

Andrey Semashev wrote:

...

I suspect we won't be able to get away with just the superproject branching. If instead of reverting a broken submodule we want to fix it, and the maintainer is unresponsive for some reason, we need either a branch in the submodule or a fork of it. Since the branch can be modified by someone else than a release manager, the fork seems more preferred. The release branch of superproject would have to refer to the fork in this case.

In what way is this an argument against the suggested structure? It supports this workflow. I don't think that something like that will ever prove necessary, but it's possible.

I didn't say I disagree with your superproject structure. The post you referenced only described downgrading submodules as a measure of fixing breakage, I'm just saying we might need to apply patches instead. For that it is not sufficient to only manage the superproject.

On Wed, Apr 15, 2015 at 8:24 AM, Jürgen Hunold wrote:

Hi Andrey,

...

On Wednesday 15 April 2015 15:32:12 Vladimir Prus wrote:

...

I don't think there's a primarily technical problem. No matter how release process goes, we either need to assume that 'master' branch of any component is good to be released at all times, or coordinate with maintainers of 120 components to determine which revision of 'master' is really good to be released.

You can guess which approach release managers would prefer.

I'm not arguing against taking master as a starting point for a release; it's a reasonable compromise. What I'm saying is that once release

Am Mittwoch, 15. April 2015, 15:52:13 schrieb Andrey Semashev: process

...

has started there should be no restriction on updating master in submodules (i.e. the freeze), the release should branch off the main code base and get finished regardless of the modifications developers make. It is possible that master is broken at the point of branching - and part of release process is to fix it for the release or revert to a last known good version. But the probability of breakage is greatly reduced with this approach.

The main problem is the necessary regression testing needed to actually verify that the regressions a) are indeed fixed b) don't introduce new failures.

And our current testing system is hardcoded to "master" and "develop" branches. And can't test anything else, as it relies on the test runners admins to set up which branch to test.

I wouldn't call it hardcoded.. But yes it relies on testers signifying the branch to test. And yes it could be more flexible.

Note that when I say "branch" above I don't necessarily mean "git branch".

...

In fact, it may be more convenient to use forks to isolate the code to be released from the main code base and use pull requests for patches that need to get into the release.

As soon as we get an up-to-date regression testing and reporting system, we can use any branching scheme you can imagine. And a lot more probably.

Not sure what you mean by "up-to-date". Could you be more specific? And I don't accept "use -insert-name- tool" as an answer. I want to know structure and process answers (as tools are just tools). That would mean using either Jenkins or BuildBot as a centralized server and

some dashboard reporting tool for visualisation.

No it doesn't mean that. Although Jenkins or BuildBot (or a variety of other testing resource management) system could be used in a solution. But I'm not aware of *any* current reporting system that meets Boost needs. I do spend a minuscule amount of time writing such a system.

But I don't know anyone with enough spare time to set up such a system without being paid and working full time. Not to mention the hardware necessary for this...

But as you just said.. I volunteer my unpaid free time to maintain and improve the testing infrastructure. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On Wednesday 15 April 2015 15:24:53 Jürgen Hunold wrote:

Hi Andrey,

Am Mittwoch, 15. April 2015, 15:52:13 schrieb Andrey Semashev:

...

On Wednesday 15 April 2015 15:32:12 Vladimir Prus wrote:

...

I don't think there's a primarily technical problem. No matter how release process goes, we either need to assume that 'master' branch of any component is good to be released at all times, or coordinate with maintainers of 120 components to determine which revision of 'master' is really good to be released.

You can guess which approach release managers would prefer.

I'm not arguing against taking master as a starting point for a release; it's a reasonable compromise. What I'm saying is that once release process has started there should be no restriction on updating master in submodules (i.e. the freeze), the release should branch off the main code base and get finished regardless of the modifications developers make. It is possible that master is broken at the point of branching - and part of release process is to fix it for the release or revert to a last known good version. But the probability of breakage is greatly reduced with this approach.

The main problem is the necessary regression testing needed to actually verify that the regressions a) are indeed fixed b) don't introduce new failures.

And our current testing system is hardcoded to "master" and "develop" branches. And can't test anything else, as it relies on the test runners admins to set up which branch to test.

Well, we could switch the master branch testing to release testing for the time while release is being prepared. This could probably be done with a commit to the testing scripts in master?

Beman Dawes wrote

While I understand how a continuous integration framework might only merge to master when develop tests are passing satisfactorily, I'm not aware of any way to prevent breaking merges with our current testing approach.

I think there is a way to "lock" a branch so that no one can push a merged branch http://stackoverflow.com/questions/2471340/is-there-a-way-to-lock-a-branch-i... This looks like it might be useful to us - but don't take my word for it as I'm a git noob Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/Fwd-Boost-Python-build-error-in-1-58-RC1-... Sent from the Boost - Dev mailing list archive at Nabble.com.

On Wed, May 6, 2015 at 2:03 PM, Andrey Semashev wrote:

On Thu, Apr 9, 2015 at 4:00 PM, Stefan Seefeld wrote:

...

On 09/04/15 02:58 AM, Mark Incley wrote:

...

It's Python 3.3.0. In the file c:\Python33\include\fileutils.h there's only the following two fopen functions:

[...]

*Sigh*. It apears the new API was introduced here: https://hg.python.org/cpython/rev/ef889c3d5dc6 That's the price to pay for using non-public APIs...

I have just committed https://github.com/boostorg/python/commit/3e405b6fd5db5615bbef241763de070118..., which should preserve the new functionality for Python 3.4, and use a little workaround for Python 3 < 3.4

This fix doesn't work on Ubuntu 15.04 (python 3.4.3).

libs/python/src/exec.cpp: In function ‘boost::python::api::object boost::python::exec_file(boost::python::str, boost::python::api::object, boost::python::api::object)’: libs/python/src/exec.cpp:93:31: error: cannot convert ‘PyObject* {aka _object*}’ to ‘const char*’ for argument ‘1’ to ‘FILE* _Py_fopen(const char*, const char*)’ FILE *fs = _Py_fopen(fo, "r");

If I'm not mistaken, the version check is incorrect, it should be:

#if PY_VERSION_HEX >= 0x03040000

(note that 4 is shifted left for 16 bits, not 20).

With the above fix it compiled for me. Created a pull request: https://github.com/boostorg/python/pull/19 It would be nice if the patch in the 1.58 release notes was also updated.