Transferring files across a network.

We have a situation where there are several computers that have a folder where files are periodically deposited. Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running. The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network. What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database. So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique. Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet! The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment. I would like to stick with boost if possible. Thanks Andrew -- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________

Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
I would think a combination of Boost.Asio and Boost.FileSystem would do the job for you. Plus there's been some prototype (and maybe some production ready) code that does asynchronous file system monitoring (as provided by OS level facilities) floating around on the Asio mailing list. This could replace your periodic "directory poll", making it that much more efficient (if that matters). Read up on both of the above libraries and ask questions as appropriate. Cliff

This is good news.
I am very interested in following this up as it sounds just what I
need. Can you give me some more specific pointers? The asynchronous
file system monitoring sounds really good.
Thanks in advance
Andrew
On Fri, Jan 16, 2009 at 4:06 PM, Cliff Green
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
I would think a combination of Boost.Asio and Boost.FileSystem would do the job for you. Plus there's been some prototype (and maybe some production ready) code that does asynchronous file system monitoring (as provided by OS level facilities) floating around on the Asio mailing list. This could replace your periodic "directory poll", making it that much more efficient (if that matters).
Read up on both of the above libraries and ask questions as appropriate.
Cliff
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________

http://archives.free.net.ph/message/20080807.085127.9c0ee43d.de.html
2009/1/16 Max
This is good news. I am very interested in following this up as it sounds just what I need. Can you give me some more specific pointers? The asynchronous file system monitoring sounds really good.
Thanks in advance Andrew
I'm also interested in this.
B/Rgds Max
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Hello Andrew,
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thank you
Rahul
To create something exceptional
________________________________
-Master4Master
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system.
Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Speak out for justice and peace,
even if your voice trembles.
________________________________
From: Andrew Maclean

I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thanks, Rahul and Boris (for original code) - that should help Andrew!
One question, though (and this is to the general list, not just Rahul and Boris) - in client.cpp there's the following code:
boost::array

This will definitely get me started. It is so elegant!
The more I delve into asio the more I like it!
Thankyou very much. You are a great help.
Andrew
On Fri, Jan 16, 2009 at 6:03 PM, master master
Hello Andrew,
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thank you Rahul
To create something exceptional ________________________________ -Master4Master
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system.
Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Speak out for justice and peace, even if your voice trembles.
________________________________ From: Andrew Maclean
To: boost-users@lists.boost.org Sent: Friday, 16 January, 2009 9:22:42 AM Subject: [Boost-users] Transferring files across a network. We have a situation where there are several computers that have a folder where files are periodically deposited.
Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running.
The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network.
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique.
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment.
I would like to stick with boost if possible.
Thanks Andrew
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________ _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
________________________________ Add more friends to your messenger and enjoy! Invite them now. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________

Thankyou Raul for your example code.
For the server I have been able to modify the code so that instead of
exiting after getting a file it sits and waits indefinitely (see the
attached code). I don't know whether this is "correct" but it seems to
work Ok. Any advice is appreciated.
Now my problem is that I want the client to send several files instead
of just one before exiting. In the final application groups of files
will need to be sent on a periodic basis. I really can't see how to do
it, I have tried to create a similar structure to that for the
modified server code but I am not having much luck.
I must admit this is a new area for me!
So any help is appreciated.
Thanks in advance for any help.
Andrew
On Fri, Jan 16, 2009 at 6:03 PM, master master
Hello Andrew,
I hope attached files will help you. This is a client that can send files to server. I got this example from Boris and it is working at my side.
Thank you Rahul
To create something exceptional ________________________________ -Master4Master
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email, and delete this message and any attachments from your system.
Virus Warning: Although I have taken reasonable precautions to ensure no viruses are present in this email, I cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Speak out for justice and peace, even if your voice trembles.
________________________________ From: Andrew Maclean
To: boost-users@lists.boost.org Sent: Friday, 16 January, 2009 9:22:42 AM Subject: [Boost-users] Transferring files across a network. We have a situation where there are several computers that have a folder where files are periodically deposited.
Currently, in each computer, this folder is scanned periodically and the files are parsed by a program that then sends the SQL data to a central machine where there is postgresql server running.
The files are generally small because they are compressed. But the uncompressing and subsequent parsing into SQL strings dramatically inflates the amount of data that is sent across the network.
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
So I need a program to periodically scan the folder on the computer and send these files to the central machine. On the central machine I need a similar program to listen for incoming files and put them into a directory for subsequent processing. The listner on the central machine will need to process files arriving simultaneously from several machine. There is no issue with file name conflicts as the names are unique.
Can anyone give me any pointers? I think I should use boost asio but I don't know where to start with it. The tutorial and examples, while good haven't provided me with an insight yet!
The main requirement I have is that it should be cross-platform. I looked at cURL/cURLpp but there are immense difficulties building this in a windows environment.
I would like to stick with boost if possible.
Thanks Andrew
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________ _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
________________________________ Add more friends to your messenger and enjoy! Invite them now. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________

"Andrew Maclean"
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
Just something to think about: On Unix at least, it would be very simple to do this with rsync and a couple cron jobs. I suspect compatible programs and similar mechanisms exist on most other OS's. You may want to look at what higher-level tools you could use to accomplish this before you delve into the code. If you do decide to code it up, asio is very nice, and quite fun to learn. Good luck! ----Scott.

Asio is not just nice, it is elegant, just like most of boost. I have
been using boost for years (since 2000) and pushing its usage here for
that time. However people still reinvent the wheel ... often poorly.
The libraries I use most are threads and filesystem.
I will certainly be using it. I see now that my stumbling block was
integrating the filesystem with asio.
One thing with boost: whenever you use it to solve a problem, I find
that it often exposes new ways of doing things.
Thanks
Andrew
On Sat, Jan 17, 2009 at 7:47 AM, Scott Gifford
"Andrew Maclean"
writes: [...]
What I am thinking of doing is sending the all the files to a directory on the central machine then parsing them there for subsequent inclusion into the database.
Just something to think about: On Unix at least, it would be very simple to do this with rsync and a couple cron jobs. I suspect compatible programs and similar mechanisms exist on most other OS's. You may want to look at what higher-level tools you could use to accomplish this before you delve into the code.
If you do decide to code it up, asio is very nice, and quite fun to learn.
Good luck!
----Scott. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- ___________________________________________ Andrew J. P. Maclean Centre for Autonomous Systems The Rose Street Building J04 The University of Sydney 2006 NSW AUSTRALIA Ph: +61 2 9351 3283 Fax: +61 2 9351 7474 URL: http://www.acfr.usyd.edu.au/ ___________________________________________
participants (6)
-
Andrew Maclean
-
Cliff Green
-
Igor R
-
master master
-
Max
-
Scott Gifford