On December 2, 2012 8:01:12 PM Olaf van der Spek <ml@vdspek.org> wrote:

On Fri, Nov 30, 2012 at 5:44 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:

...

Does it provide any benefits compared to mounting a loop device and working with it through traditional file system interfaces? It seems odd to have a library duplicating file system operations.

Is it? We don't use loop devices to read/write tar/zip files, do we?

On OS X you do mount packages. You also typically mount various image files. Library-level access to conventional archive files is a legacy from older systems (read Windows and DOS) that did not support flexible mounting.

Can you even mount loop devices as unprivileged user?

Ok, you have a point about unprivileged mounting.

Does Windows even have loop devices?

According to Wikipedia, it does now. But then again, Windows is not the whole world.

On 03/12/2012, at 6:33 AM, Alexander Voitenko <tarmik@meta.ua> wrote:

...

Does it provide any benefits compared to mounting a loop device and working with it through traditional file system interfaces? It seems odd to have a library duplicating file system operations.

First of all, compound files are portable. You can use the same data, in the same compound files within different operating systems, not limited to Linux, Windows and Mac OS.

Please note, that both Unix-like systems and Windows struggling with limit of loop devices http://www.tldp.org/HOWTO/CDServer-HOWTO/addloops.html [Linux] http://stackoverflow.com/questions/1944877/maximum-number-of-drives-in-windo... [Windows]

But with a such library you can simultaneously read and write thousands(millions?) of such files without restrictions from the system side. Library can be linked statically and only thing that you will require to access such file - to launch your application and not necessarily written in C++, because already exist analogous libraries in Java http://poi.apache.org and C# http://openmcdf.sourceforge.net worlds.

Also, if I want to use lot of different compound files with my data within my application, it is to weird mount and unmount devices from the runtime and also require super-user privileges.

Please do not think about compound files only as file system replacement. Imagine software that can produce some documents. Such documents use binary format and have very complex internal structure, so such structure can be represented as file system where some logical parts are grouped in folders and files.

Example of such software that really uses compound files: http://www.corel.com/corel with it .CDX format http://www.amwa.tv/ with it .AWM file format and exist some others, but not so famous.

Of course lot of Microsoft applications use them, but I don't want mention them by religious reasons ;-)

User of such applications even can not know that internal representation of his documents is an entire file system.

In my usual work I deal with compound files like with some sort of archives. Store hundreds of them on my hard drive, often copy, rename, move or zip them and send via e-mail.

Regards, Alexander Voitenko.

I guess you can add to your examples tar. zip, iso as well and these should be accessible at the user level without super/root mount.

-- View this message in context: http://boost.2283326.n4.nabble.com/compond-file-binary-Gauging-interest-in-a... Sent from the Boost - Dev mailing list archive at Nabble.com.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Is it correct that the file content isn't compressed? Yes, this is correct.

Of course, if you need, you can create stream inside compound file and write data that is compressed or encrypted in some way. Overall overhead for representing binary data as file system is small: about 0.8% from a user's data size. If you have any additional questions, please feel free to ask. -- View this message in context: http://boost.2283326.n4.nabble.com/compond-file-binary-Gauging-interest-in-a... Sent from the Boost - Dev mailing list archive at Nabble.com.

Yes, you mentioned the fact that it's not obvious to people not knowing the format that the file is organized like a filesystem. Does this mean that there is active obfuscation or is it just a side effect?

Internal file system format is not obfuscated. Yes, this is just a side effect. For example, Microsoft Project's .mpp files are binary compound files inside, highly structured and contain about 100 directory entries(files and folders), but user just open it within MS Propject or Open Propject(free open source alternative) and work with content and does not care about how his or her data is organized inside that file. -- View this message in context: http://boost.2283326.n4.nabble.com/compond-file-binary-Gauging-interest-in-a... Sent from the Boost - Dev mailing list archive at Nabble.com.

On Tue, Dec 4, 2012 at 7:42 AM, Brian Ravnsgaard Riis <brian@ravnsgaard.net>wrote:

How does your intended solution differ from (or compare to) simply writing documents as a (possibly renamed) .zip file?

Or maybe standardising (as such) an interface for this is the purpose of your submission?

Regards, Brian Riis

I think this is leaning towards my thinking as well: I'd be more interested in a library that forms a generic interface to any hierarchical structured file (tar, zip, compound files, etc), than a library that just did one particular format. ie a common interface with "pluggable" back-ends for various formats. Would that be possible? That would seem more "Boost-ish" to me. Tony

Den 05-12-2012 10:09, Alexander Voitenko skrev:

...

ie a common interface with "pluggable" back-ends for various formats. Would that be possible? I think yes, but with some restrictions, because some formats have unique specific features that are not shared among another formats. What is the boost policy about depending on a third-party software? Of course, I do not want to reinvent the wheel and write yet another implementation of zlib library.

That's understandable, and should not be necessary. I'd suggest trying to define a clean boundary between the interface and the implementation. You already have one back-end that makes the entire package usable. Adding another backend later, or as an option, that supports zlib/bzip2/whatever compression should be easier then. As long as the library has one working imlementation that does not depend on 3rd party software, I don't think there'll be any problems. Boost.Locale already does this with ICU, which is required for certain feature/platform combinations, but the library can be used without ICU; it'll then not support the features that ICU provides. A similar approach should be possible with zlib in your case. Regards, Brian Riis

I'd suggest trying to define a clean boundary between the interface and the implementation. You already have one back-end that makes the entire package usable. Adding another backend later, or as an option, that supports zlib/bzip2/whatever compression should be easier then.

I agree with your thoughts, Brian. To be sure that this point was achieved, I must investigate such possibility well and implement at least one more back-end. For now I have several questions about integration with boost. 1) Compound files use utf-16 encoding for names and I use third party library for conversions utf-8 <-> utf-16 http://utfcpp.sourceforge.net/ But looks like that Boost.Locale can be used as replacement. Is it? 2) I like TDD and automated tests, so implemented lot of them(for now about 7000 lines and plan add more). But most of my tests are not unit tests, but integration test. For such software, testing in terms of usage scenarios is better I think. As result, I have bunch of test data files which have size about 100mb. Of course it is unacceptable to provide all this stuff within the boost distribution. Is it possible to split tests? Module tests can be included in boost distribution that available to the end user, and integration test leave in repository and use only in nightly builds or whatever you have there. 3) Compound files use red-black tree as format feature. And to deal with compound files correctly, I need to access such low level data of a red-black tree as color of a node. I searched on the Internet for a free C++ library that allow to access all the data within a tree but not found any which is well-tested, so decided to make my own implementation using TDD approach. It was developed according to this book: http://www.amazon.com/Introduction-Algorithms-Includes-CD-Rom-Thomas/dp/0072... With my tiny modifications. ]:) Is it any interest for binary and red-black tree implementations as separate component? Or... the boost already have such facilities and I waste my time by reinventing one more wheel? :-) 4) For integral types I use defines from <stdint.h> is it acceptable? Or should I switch to similar types from the boost library? Regards, Alexander. -- View this message in context: http://boost.2283326.n4.nabble.com/compond-file-binary-Gauging-interest-in-a... Sent from the Boost - Dev mailing list archive at Nabble.com.

How does your intended solution differ from (or compare to) simply writing documents as a (possibly renamed) .zip file?

Yes, zip files without compression are similar to the compound files in some way. They both provide some directory entries hierarchy inside and methods to iterate through them, also they both provide acces to the stored data. I am not expert in zip archives, so possible my suggestions are wrong. But I can see several advantages over zip files: - As I understand, zip files work sequentially: you put one file inside, then another, then can delete some files. Only as whole chunks. But with compound files you can open several streams and keep them opened: some for writing, some for reading and make modifications simultaneously. Of course this will lead to data fragmentation, but this is another story ;-) With compound files you do not need to extract internal content to modify it, you can do it "in place", even if you want to expand length of stored data by writing new chunk at the end of some stream. Compound file's internal file system provides all needed facilities to do such operations with minimal cost. - Faster entries searching. All child entries of one directory are organized as red-black tree at format level. So entire directory entries hierarchy look like a tree of red-black trees. - Faster entries deletion. As I can see, zip files explicitly exclude deleted files then recalculate CRC checksum. But compound files only mark sectors for a deleted entry in the "Files Allocation Table" as "unused". Yes, in common cases actual data is not removed and can be recovered by some tools like hex editors. But this is similar to all file systems. -- View this message in context: http://boost.2283326.n4.nabble.com/compond-file-binary-Gauging-interest-in-a... Sent from the Boost - Dev mailing list archive at Nabble.com.

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Hartmut Kaiser Sent: Wednesday, December 05, 2012 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] [compond_file_binary] Gauging interest in a possible library submission.

Is there any interest in a cross-platform C++ library which allows to

...

create/read/write binary compound files?

Briefly, compound file is filesystem for storing files and directories within a single file on a disk. Initially compound file format is developed by Microsoft and now is the part of Microsoft's Open Specifications Documentation. Compound files are used across various platforms and applications, and not restricted by some specific domain. In general, any application can store information in such a way. I've found on the Internet interest in such library and decided to implement it.

Additional information can be found here:

Article on the Wikipedia: http://en.wikipedia.org/wiki/Compound_File_Binary_Format

Benefits of Compound Files: http://msdn.microsoft.com/en- us/library/windows/desktop/aa378938(v=vs.85).aspx

[MS-CFB] Open specification for compound files: http://msdn.microsoft.com/en-us/library/dd942138.aspx

I'm interested too. It seems the sort of weapon that Boost should have in its toolkit? Are there any other implementations (free open-source or commercial). Although access to Microsoft containers will interest many, and http://en.wikipedia.org/wiki/Advanced_Authoring_Format even more are there other structures/format we might consider? Paul PS Boost licence throughout would be absolutely essential of course. --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Alexander Voitenko Sent: Wednesday, December 05, 2012 2:45 PM To: boost@lists.boost.org Subject: Re: [boost] [compond_file_binary] Gauging interest in a possible library submission.

I'm interested too. It seems the sort of weapon that Boost should have in its toolkit? Are there any other implementations (free open-source or commercial).

------------------------- Free implementations ------------------------- In Java: Apache POI (Well known monster :) Users of that library complain that it is difficult to use) http://poi.apache.org/

In C#: Openmcdf library (Lacks of unit testing, consume lot of memory) http://openmcdf.sourceforge.net/

In C++: Pole library (Read only support, need very careful usage, because implementation is unsafe) http://www.dimin.net/software/pole/

------------------------- Commercial implementations ------------------------- In C: SchemaSoft's NFF_StructuredStorage library It is not available on the Internet, but some companies have its sources. This is c library, buggy, difficult to use and unsafe. I worked with it in Quickoffice company. Hate that macro-style nightmare }:->

Doesn't sound like there is much competition ;-)

If you know any other implementations, pleeeese let me know.

I'm ignorant ;-)

...

Although access to Microsoft containers will interest many, and http://en.wikipedia.org/wiki/Advanced_Authoring_Format even more Yep. AFF format is structured storage inside.

...

are there other structures/format we might consider? Hmmm... if you suggest one, I can investigate.

I'm even more ignorant ;-) But perhaps others are knowledgeable? I'm only trying to be sure that the underlying concept is the best (and not just the one that Microsoft has chosen - spec looks rather C/Hungarian-ish?).

...

PS Boost licence throughout would be absolutely essential of course. If you mean that I must release the lib under boost license, this is not a problem.

Yes - that's all - sounds not a problem provided you hold the copyright. Thanks Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

First of all I want to mention that my library is not finished yet. I am at the end of implementing planned features for the first release, but some additional effort is needed. I have no plans to add support for the all features that MS implementation provides in the first release, but want to develop library in evolutionary way. Also I plan add some features in next releases that are not supported by MS implementation: storage compacting and defragmentation. I found on the Internet that users need them.

Sounds good.

...

More generally, does your library support the properties of the MS compound storage implementation as described here:

http://msdn.microsoft.com/en- us/library/windows/desktop/aa378871(v=vs.85).aspx ?

Here is that list with my comments.

[Incremental access.] Implemented and tested.

[Multiple use.] Not implemented, but I think often about that. The main problem: I can not decide use C++11 threading model or switch to some cross-platform library(Boost.Thread may be?) For now supported one thread per one compound files. Also, threading implementation and testing will require lot of effort. And I not plan to include multithreading support in the first release.

I don't think the library has to care about what type of threading model is used. I imagine the problem of making the files accessible concurrently is even wider as you have to enable multiple processes to read/write concurrently. The MS implementation achieves this using the transactional mechanisms, which moves the responsibility for making it concurrency safe into user land. Essentially every write creates a new subtree (stream), which then is made current by a single atomic write operation, invalidating the previous version of the data.

[Transaction processing.] Not supported. Planned in next releases.

Good, that's one of the main advantages of this model, in my book.

[Low-memory saves.] Yes. I designed the library to be memory efficient. All needed memory is allocated on stream opening. I plan to test memory issues by emulating std::bad_alloc in various places.

Thanks! Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu