Cleanup of test results.
Just an FYI for testers and authors.. I implemented some changes to the report generation that: a) Expires old results to not be shown in the results. I'm considering old to be anything past two weeks of age. This applies to all branches (i.e. develop and master). b) Filtering of master, aka release, results to only include an approved list of testers. This brings in some explicit control to what is considered the release toolsets. Which is something we used to enforce, but lost after the git transition. So it's back now. Rene. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Yes, I see my results were removed from the 'master' table (but still present in 'develop'). It's because we're not 'approved', right? If yes, what need to be included to the approved list of testers? -- Dmitry Moskalchuk On 11/02/15 06:29, Rene Rivera wrote:
Just an FYI for testers and authors..
I implemented some changes to the report generation that:
a) Expires old results to not be shown in the results. I'm considering old to be anything past two weeks of age. This applies to all branches (i.e. develop and master).
b) Filtering of master, aka release, results to only include an approved list of testers. This brings in some explicit control to what is considered the release toolsets. Which is something we used to enforce, but lost after the git transition. So it's back now.
Rene.
On Wed, Feb 11, 2015 at 2:40 AM, Dmitry Moskalchuk
Yes, I see my results were removed from the 'master' table (but still present in 'develop').
It's because we're not 'approved', right?
Yes.
If yes, what need to be included to the approved list of testers?
Roughly it goes like this: 1. Run full and complete results on the develop branch for some time. 2. Prove that you can run tests on a consistent schedule. 3. Prove that you are responsive to platform questions and requests for the tests you run. 4. For new platforms (OS and or toolset) prove that you can support that new platform with patches and collaboration with library authors. 5. Ask to be added to the master results list. 6. Ask to be considered a release supported platform. Note.. * As far as I can tell you are still working on step #1. * For #2 consistent means a predictable schedule, or continuous. The minimum frequency I look for a at least weekly results on develop. For master results I prefer daily but can live with 3 times a week. * #3 means responding to emails on both the testing and developer lists reasonably promptly. And addressing any problems.. Like looking into stalled tests, providing full logs for specific tests, running specific tests for authors, etc. * You are actively pursuing #4.. Which is awesome :-) * #5 just gets the results into the web site. * And #6 gets your platform, Android, listed as supported in the release notes. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On 11/02/15 19:23, Rene Rivera wrote:
Roughly it goes like this:
1. Run full and complete results on the develop branch for some time. 2. Prove that you can run tests on a consistent schedule. 3. Prove that you are responsive to platform questions and requests for the tests you run. 4. For new platforms (OS and or toolset) prove that you can support that new platform with patches and collaboration with library authors. 5. Ask to be added to the master results list. 6. Ask to be considered a release supported platform.
Note..
* As far as I can tell you are still working on step #1. Yes
* For #2 consistent means a predictable schedule, or continuous. The minimum frequency I look for a at least weekly results on develop. For master results I prefer daily but can live with 3 times a week. We have dedicated servers for Boost testing so we definitely will run it continuously; however, full run of all Boost tests on one API level/ABI could take several days - this goes from the cross-build nature of Android testing. We build them on host OS (Linux), upload to target machine (Android) as well as all required libraries and then run tests there. It obviously take more time than just build&run on the same machine. I already optimized many things and probably could optimize more but my rough estimate is still ~3 days for one run.
* #3 means responding to emails on both the testing and developer lists reasonably promptly. And addressing any problems.. Like looking into stalled tests, providing full logs for specific tests, running specific tests for authors, etc. No problem with this. I already subscribed to boost-testing list and we'll monitor it too.
* You are actively pursuing #4.. Which is awesome :-) * #5 just gets the results into the web site. * And #6 gets your platform, Android, listed as supported in the release notes.
OK, got it. Thank you for explanation. We'll definitely go further to apply to these rules. -- Dmitry Moskalchuk
participants (2)
-
Dmitry Moskalchuk -
Rene Rivera