- Boost - lists.preview.boost.org

[filesystem] proposal: treat reparse files as regular files
by Paul Harris 14 Aug '15

14 Aug '15

Hi all, -- Proposal -- tl;dr : I propose that we treat all non-symlink "reparse_files" as "regular_files". If the boost library user wants to do something special with these plain reparse files, they should use alternative means. But typically they are supposed to be treated as regular files. This means we could drop the "reparse_file" enum, or continue to use it for a special-case whats_my_real_status() function. --- Motivation --- Windows Server 2012 uses reparse points to implement deduplification. Those files should be treated as regular files in all circumstances. Currently, they are not classed as "regular" files, so fs::copy() will skip those files, and library-user code written to list files based on official examples will ignore all dedup'd files. This is causing serious and latent problems at the user end, because deduping only happens occasionally after X days, and users cannot easily check if a file is dedup'd (they look just like regular files). --- Real life example --- Another example of reparse use is the "Symantec Enterprise Vault" (version 10), which I found running on one site. It replaces files on the server with reparse-point files. FSUTIL REPARSEPOINT QUERY filename.txt shows the contents of the reparse buffer, which is a URL to an internal HTTP server. The url points to a .asp link with a bunch of codes and dates to identify the file in the server. Copy-pasting that URL into a webbrowser allows you to directly download the file via the webbrowser, which is pretty neat I suppose. In this case, the reparsed-files in Windows Explorer all have grey X crosses on their file icon. If you "type" them (via cmd) or open them, the icon loses the grey cross and the file is no longer a reparse point file. My software refused to read the files because they were "not regular files". Once I adjusted the boost code (described below), my software saw them as regular and opened the files. The file icons lost the grey cross. SO it seems that the file server automatically downloads and replaces the files with the stored content on demand, and the file reading client program should really just treat these files as normal files. --- Short logic --- reparse files (that are not symlinks) should almost always be treated as plain files. They are a mechanism for MS file servers to store files in clever ways, but the client should not care and just read/write them as if they were normal files. This is different to all the other "other" files which can't be treated like normal files: block, character, fifo, socket, unknown So, reparse files should not be grouped with the "other" file types. They are also NOT symlinks, and should not be treated as symlinks (which would require special decisions for copying, or querying the status, or checking if the target still exists). --- What are reparse files --- I did some reading, if I understand correctly: Reparse points give drivers (on the server) a chance to get data through some other specialised means (eg query from a cluster store). They are processed by the server, not the client, so clients should treat reparse data as opaque data. EXCEPT for symlink reparse files. https://msdn.microsoft.com/en-us/library/dd541667.aspx quote:"The following reparse tags, with the exception of IO_REPARSE_TAG_SYMLINK, are processed on the server and are not processed by a client after transmission over the wire. Clients should treat associated reparse data as opaque data." It seems like the rest of the tags are used for connecting files to other types of storage (eg long term storage, cluster storage). Clients may need to do something special with SOME reparse point files, IF the client cares about how long the file read may take. https://msdn.microsoft.com/en-us/library/windows/desktop/aa365505(v=vs.85).… quote: "Most applications should take special actions for files that have been moved to long-term storage, if only to notify the user that it may take a while to retrieve the file." --- Changes required --- Option 1: change is_regular_file() to return true where type==reparse_file I don't like this option, as library-users could be checking the type directly instead of using is_regular_file(). Option 2: These functions return reparse_file: fs::file_type query_file_type(const path& p, error_code* ec) file_status status(const path& p, error_code* ec) file_status symlink_status(const path& p, error_code* ec) They should instead return regular_file instead. --- How to test with dedup files --- Creating dedup'd files is a feature only available on Windows Server 2012, I believe, although Windows XP/Vista/7/8/10 clients all can read dedup files. Here is how I created a windows server to test with (for free!) on a demo Azure cloud server. I have one working, so if anyone would like to use it for their testing, let me know. Step one: follow this blog article: http://blogs.technet.com/b/tommypatterson/p/azureservertrial.aspx once the machine was "running" I clicked Connect at the bottom. That gave me an .rdp file which in theory I could use with rdesktop, but it uses a DNS name that was only just created, so that didn't work. When you click the name of the server in the list, it shows the public IP on the right.. and the port then you can do this $ rdesktop that.ip.addr:port But only if you have the latest rdesktop AND you have set up kerberos something-something. Instead I found a windows computer and used remote desktop from there. --- Once inside, in the "Server Manager --> Dashboard" window on the screen, click "Add Roles" then go next next until "Server Roles" expand "File and Storage services" , "File and iSCSI" , and tick "Data Deduplication" Then next next etc and Install. Wait a bit... and its done. http://www.techrepublic.com/blog/data-center/configuring-windows-server-8-d… --- Continuing on that webpage... Time to enable dedup. There is a temp disk D: so lets enable there. Method 1... I did this and then went to method 2... Start PowerShell, type: "Enable-DedupVolume D:" Method 2... in that same Dashboard, hit the 4th button (File and Storage Services) Then Volumes --> Disks click Volume 1 at the top, and then right click D: at the bottom --> Configure Dedup. To try and accelerate this puppy, I set the "age to dedup" to 0 days. http://www.techrepublic.com/blog/data-center/windows-server-2012-deduplicat… --- Time to make something to dedup. We'll just duplicate the warning.txt file that exists on D: In powershell: PS> D: PS> $file = Get-Content DATALOSS_WARNING_README.txt Then, do these 2 commands a bunch of times until "big.txt" gets to say 6MB PS> Add-Content big.txt $file PS> $file = Get-Content big.txt Then use windows explorer (or other) to make a dozen copies of big.txt Copy c:\windows\explorer.exe to D: to give it something to dedup Go to D: and then copy-paste explorer.exe a dozen times. In PowerShell, type: PS> Update-DedupStatus -Volume D: PS> Start-DedupStatus -Type Optimization -Volume D: and then wait for it to finish. you can track its progress with: PS> Get-DedupJob PS> Get-DedupStatus -Volume D: --- So, once its deduped, you check. PS> FSUTIL REPARSEPOINT QUERY big.txt you should see that its a reparse point with that 0x800etc0013 code. Copy-paste big.txt to big2.txt and check it with the query, and it should tell you big2 is NOT a reparse point. NOW you have some files to test the boost library... You can't zip them up (they lose the dedup tag), you have to run boost binaries ON the computer in the sky. --- Finish --- Thanks for reading, Paul

5 30

[COre] Pull request for enable_if_type
by Joel FALCOU 13 Aug '15

13 Aug '15

Hi, I just refurbished and pushed an old patch adding the enable_if_type SFINAE utility to core. https://github.com/boostorg/core/pull/15 I tried to keep it simple and make the doc proper. This utility has been laying around various boost library and was "reinvetned" in bosot.SIMD. I think it's worth being in core so code duplication is kept to a minimum. Comments welcome

2 3

[spirit] 1.59.0 rc3: compiler warning in real_impl.hpp
by Tobias Loew 13 Aug '15

13 Aug '15

Hi, with MSVC 11, 32-bit build I get a compiler warning in boost\spirit\home\qi\numeric\detail\real_impl.hpp : 287 warning C4244: '=' : conversion from 'unsigned __int64' to 'double', possible loss of data since I usually comile with warnings-as-errors this broke my build. Tobias -- View this message in context: http://boost.2283326.n4.nabble.com/spirit-1-59-0-rc3-compiler-warning-in-re… Sent from the Boost - Dev mailing list archive at Nabble.com.

2 3

Any interest in a M:N green threading library?
by Xu Chen 13 Aug '15

13 Aug '15

Just cleanup some codes I wrote before, there is a M:N green threading library built on top of Boost.Thread and Boost.Coroutine, conceptually it works like goroutines in Go and (once existed) green threads in Rust. With some refinements, it now supports all C++11/14 threading standard, along with some extensions exist in Boost.Thread such as thread_group and chainable futures. It can be integrated with Boost.ASIO, all ASIO async functions can be used as sync ones in green threads, similar to Boost.ASIO/Boost.Coroutine integration. Also it has some facilities to communicate with foreign threads, can by used to integrate with external libs such as database drivers. The code can be found at https://github.com/windoze/boost.green_thread I don't know much about boostbook tools so the documents are kind of crippled, but I'll improve it if needed. Regards, Chen Xu

4 5

Re: [boost] [build] Problems with VC14 and MSPDB140.DLL (was: [testing][teeks] MSVC-14 errors about MSPDB140.DLL)
by Adam Merz 12 Aug '15

12 Aug '15

Andrey Semashev <andrey.semashev <at> gmail.com> writes: > > I created a Connect ticket about it: > > https://connect.microsoft.com/VisualStudio/feedback/details/1651822/incorr ect-mspdb140-dll-version-picked-in-x86-x64-cross-tools-environment > This ticket appears to be set to private; would you mind editing it to public visibility so that we may upvote it? Regards,

5 5

1.59.0 rc3 is now available
by Marshall Clow 12 Aug '15

12 Aug '15

New Release candidate files for 1.59.0 are available at: http://boost.cowic.de/rc/ RC3 should fix the problem with VC14 and MSPDB140.DLL. As always, the release managers would appreciate it if you download the candidate of your choice and give building it a try. Please report both success and failure, and anything else that is noteworthy. This helps ensure the candidates build OK before we push them out to SourceForge. The files (and associated md5s) are: MD5 (boost_1_59_0_rc3.7z) = 0a2e512844f3e30a6240f8139ee983f3 MD5 (boost_1_59_0_rc3.tar.bz2) = 6aa9a5c6a4ca1016edd0ed1178e3cb87 MD5 (boost_1_59_0_rc3.tar.gz) = 51528a0e3b33d9e10aaa311d9eb451e3 MD5 (boost_1_59_0_rc3.zip) = 08d29a2d85db3ebc8c6fdfa3a1f2b83c Thanks! -- The release managers P.S. Release notes are here: http://www.boost.org/users/history/version_1_59_0.html

3 3

[context] msvc /SAFESEH
by Jeff Flinn 12 Aug '15

12 Aug '15

I need to build my application with /SAFESEH which uses boost coroutine which uses context. I'm getting the following link errors: 1>libboost_context-vc100-mt-gd-1_53.lib(jump_i386_ms_pe_masm.obj) : error LNK2026: module unsafe for SAFESEH image. 1>libboost_context-vc100-mt-gd-1_53.lib(make_i386_ms_pe_masm.obj) : error LNK2026: module unsafe for SAFESEH image. I've built boost static libs with the same MSVC vesion and options (sans /SAFESEH as it's not a compile/lib option) Compiler/linker versions: Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86 Microsoft (R) Incremental Linker Version 10.00.40219.01 Has anyone run into this and resolved the error? Any insight is appreciated. Thanks, Jeff

3 9

[build] Overriding warning level for clang
by Edward Diener 11 Aug '15

11 Aug '15

In my user-config.jam for clang I have added: <warnings>on to the configuration section of the 'using clang' toolset. Nonetheless I still see '-Wall' on the command line when I compile. My other <cxxflags> settings which are part of the toolset are reflected by the command line. Even if I change to <warnings>off in the toolset I still see '-Wall' on the command line. Why is this not working and is there any way I can change the warning level for clang so that it is not 'Wall' on the command line ? The clang compiler is wonderful but with 'Wall' it spews out a boatload of information I normally do not need.

1 0

[build] Problems with VC14 and MSPDB140.DLL (was: [testing][teeks] MSVC-14 errors about MSPDB140.DLL)
by Andrey Semashev 11 Aug '15

11 Aug '15

On Sat, May 23, 2015 at 4:40 PM, Hartmut Kaiser <hartmut.kaiser(a)gmail.com> wrote: > >> All tests for Boost.Log (and I suspect any other built libraries) on MSVC- >> 14 >> are failing because of this error: >> >> LINK : fatal error LNK1101: incorrect MSPDB140.DLL version; recheck >> installation of this product > > My workaround was to edit the 'Path' env variable in one of the temp files > generated by bjam (b2_msvc_14.0_vcvarsall_amd64.cmd in your temp directory) > not to point to the x64 cross compiler, but to the native one. For me this > involved changing > > C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64_x86 > > To > > C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64 I encountered the same problem with VC14 RTM on my VM. This was a clean install of VC14 RTM, there were no CTPs or RCs installed on this machine. I removed the b2_msvc_14.0_vcvarsall*.cmd scripts and ran b2 with address-model=64 to build my tests so that these scripts are re-created from scratch, so it looks like there is a problem with how these scripts are generated by b2. I found 3 instances of mspdb140.dll on my system in these locations: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64 The last two are for 32 and 64-bit native compilers. The first one, I assume, is for use by IDE. The problem is that the first path is listed in b2_msvc_14.0_vcvarsall_x86_amd64.cmd in the "SET Path=..." line prior to the compiler-specific paths. Removing it from the Path solved the problem. Could Boost.Build be adjusted so that the cmd files are generated correctly in the first place?

9 25

[range] index adaptor with size_t as index_type
by Tobias Loew 10 Aug '15

10 Aug '15

>From version 1.55 to 1.56 the index_type of the index of the range index-adaptor was changed from size_t to ptr_diff which e.g. allows for negativ start values. But this has the side-effect that comparison with other size_t-values (e.g. a call to vector's size()) results in a compiler warning for comparing signed and unsigned values (on MSVC 11: warning C4018: '<' : signed/unsigned mismatch). I'm proposing a specialized version of the index adaptor, where the index_type is size_t and the start index is 0 (zero) and explicit specification of the start-value is not possible. A possible name could be "uindex". comments / critics / pros / cons are welcome Tobias -- View this message in context: http://boost.2283326.n4.nabble.com/range-index-adaptor-with-size-t-as-index… Sent from the Boost - Dev mailing list archive at Nabble.com.

1 0