Transferring files across a network.
We have a situation where there are several computers that have a folder where files are periodically deposited. Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running. The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network. What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database. So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique. Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet! The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment. I would like to stick with boost if possible. Thanks Andrew -- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
I would think a combination of Boost.Asio and Boost.FileSystem would do the job for you. Plus there's been some prototype (and maybe some production ready) code that does asynchronous file system monitoring (as provided by OS level facilities) floating around on the Asio mailing list. This could replace your periodic "directory poll", making it that much more efficient (if that matters). Read up on both of the above libraries and ask questions as appropriate. Cliff
This is good news. I am very interested in following this up as it sounds just what I need. Can you give me some more specific pointers? The asynchronous file system monitoring sounds really good. Thanks in advance Andrew On Fri, Jan 16, 2009 at 4:06 PM, Cliff Green <cliffg@codewrangler.net> wrote:
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
I would think a combination of Boost.Asio and Boost.FileSystem would do the job for you. Plus there's been some prototype (and maybe some production ready) code that does asynchronous file system monitoring (as provided by OS level facilities) floating around on the Asio mailing list. This could replace your periodic "directory poll", making it that much more efficient (if that matters).
Read up on both of the above libraries and ask questions as appropriate.
Cliff
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
http://archives.free.net.ph/message/20080807.085127.9c0ee43d.de.html 2009/1/16 Max <loadcom@sina.com>
This is good news. I am very interested in following this up as it sounds just what I need. Can you give me some more specific pointers? The asynchronous file system monitoring sounds really good.
Thanks in advance Andrew
I'm also interested in this.
B/Rgds Max
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Hello Andrew, I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side. Thank you Rahul To create something exceptional ________________________________ -Master4Master "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system. Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment." Speak out for justice and peace, even if your voice trembles. ________________________________ From: Andrew Maclean <andrew.amaclean@gmail.com> To: boost-users@lists.boost.org Sent: Friday, 16 January, 2009 9:22:42 AM Subject: [Boost-users] Transferring files across a network.. We have a situation where there are several computers that have a folder where files are periodically deposited. Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running. The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network. What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database. So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique. Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet! The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment. I would like to stick with boost if possible. Thanks Andrew -- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________ _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thanks, Rahul and Boris (for original code) - that should help Andrew! One question, though (and this is to the general list, not just Rahul and Boris) - in client.cpp there's the following code: boost::array<char, 4096> buffer; ... std::size_t size = std::min(sizeof(buffer), file.size() - sent); The above line assumes that the size of a boost::array object is the size of the internal char buffer - I'm not sure this is a good assumption (it assumes zero space overhead for boost::array, which might be the case in the implementation, but seems dangerous - I don't see this documented or guaranteed anywhere in the boost::array docs, plus there's always the possibility of compiler added padding). I would think better and safer (and guaranteed to be correct by boost::array) is: std::size_t size = std::min(buffer.size(), file.size() - sent); HTH, Cliff
This will definitely get me started. It is so elegant! The more I delve into asio the more I like it! Thankyou very much. You are a great help. Andrew On Fri, Jan 16, 2009 at 6:03 PM, master master <master_4_master@yahoo.com> wrote:
Hello Andrew,
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thank you Rahul
To create something exceptional ________________________________ -Master4Master
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system.
Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Speak out for justice and peace, even if your voice trembles.
________________________________ From: Andrew Maclean <andrew.amaclean@gmail.com> To: boost-users@lists.boost.org Sent: Friday, 16 January, 2009 9:22:42 AM Subject: [Boost-users] Transferring files across a network.
We have a situation where there are several computers that have a folder where files are periodically deposited.
Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running.
The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network.
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique.
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment.
I would like to stick with boost if possible.
Thanks Andrew
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________ _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
________________________________ Add more friends to your messenger and enjoy! Invite them now. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
Thankyou Raul for your example code. For the server I have been able to modify the code so that instead of exiting after getting a file it sits and waits indefinitely (see the attached code). I don't know whether this is "correct" but it seems to work Ok. Any advice is appreciated. Now my problem is that I want the client to send several files instead of just one before exiting. In the final application groups of files will need to be sent on a periodic basis. I really can't see how to do it, I have tried to create a similar structure to that for the modified server code but I am not having much luck. I must admit this is a new area for me! So any help is appreciated. Thanks in advance for any help. Andrew On Fri, Jan 16, 2009 at 6:03 PM, master master <master_4_master@yahoo.com> wrote:
Hello Andrew,
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thank you Rahul
To create something exceptional ________________________________ -Master4Master
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system.
Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Speak out for justice and peace, even if your voice trembles.
________________________________ From: Andrew Maclean <andrew.amaclean@gmail.com> To: boost-users@lists.boost.org Sent: Friday, 16 January, 2009 9:22:42 AM Subject: [Boost-users] Transferring files across a network.
We have a situation where there are several computers that have a folder where files are periodically deposited.
Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running.
The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network.
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique.
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment.
I would like to stick with boost if possible.
Thanks Andrew
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________ _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
________________________________ Add more friends to your messenger and enjoy! Invite them now. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
"Andrew Maclean" <andrew.amaclean@gmail.com> writes: [...]
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
Just something to think about: On Unix at least, it would be very simple to do this with rsync and a couple cron jobs. I suspect compatible programs and similar mechanisms exist on most other OS's. You may want to look at what higher-level tools you could use to accomplish this before you delve into the code. If you do decide to code it up, asio is very nice, and quite fun to learn. Good luck! ----Scott.
Asio is not just nice, it is elegant, just like most of boost. I have been using boost for years (since 2000) and pushing its usage here for that time. However people still reinvent the wheel ... often poorly. The libraries I use most are threads and filesystem. I will certainly be using it. I see now that my stumbling block was integrating the filesystem with asio. One thing with boost: whenever you use it to solve a problem, I find that it often exposes new ways of doing things. Thanks Andrew On Sat, Jan 17, 2009 at 7:47 AM, Scott Gifford <sgifford@suspectclass.com> wrote:
"Andrew Maclean" <andrew.amaclean@gmail.com> writes:
[...]
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
Just something to think about: On Unix at least, it would be very simple to do this with rsync and a couple cron jobs. I suspect compatible programs and similar mechanisms exist on most other OS's. You may want to look at what higher-level tools you could use to accomplish this before you delve into the code.
If you do decide to code it up, asio is very nice, and quite fun to learn.
Good luck!
----Scott. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
participants (6)
-
Andrew Maclean
-
Cliff Green
-
Igor R
-
master master
-
Max
-
Scott Gifford