callback or file based construction of CSR graph
Is there any way for CSR graph to provide a constructor that takes a file stream pointer or a callback functor? If not, how hard is it to add it? I am using edges_are_sorted CSR constuctor. It doubles memory I passed in during constructor calls. My graph is huge (>30 million arcs) and It took >8gb ram to construct my CSR. If I can have callback or file-based construction, I could cut down the memory usage by half. Dan _________________________________________________________________ Hotmail & Messenger. Get them on your phone now. http://go.microsoft.com/?linkid=9724463
On Sat, 24 Apr 2010, Dan Jiang wrote:
Is there any way for CSR graph to provide a constructor that takes a file stream pointer or a callback functor? If not, how hard is it to add it? I am using edges_are_sorted CSR constuctor. It doubles memory I passed in during constructor calls. My graph is huge (>30 million arcs) and It took >8gb ram to construct my CSR. If I can have callback or file-based construction, I could cut down the memory usage by half.
You can use arbitrary iterators in most of the constructors, including the one you are using. You can use Boost.Iterator to create a custom iterator for your file data structure and use that with the CSR constructors. Note that some of the unsorted ones require multi-pass iterators, which would be harder to implement on top of a file, but edges_are_sorted is single-pass. -- Jeremiah Willcock
Hi Jeremiah, As always, thanks a lot for your insights. Dan
Date: Sat, 24 Apr 2010 12:48:07 -0400 From: jewillco@osl.iu.edu To: boost@lists.boost.org Subject: Re: [boost] callback or file based construction of CSR graph
On Sat, 24 Apr 2010, Dan Jiang wrote:
Is there any way for CSR graph to provide a constructor that takes a file stream pointer or a callback functor? If not, how hard is it to add it? I am using edges_are_sorted CSR constuctor. It doubles memory I passed in during constructor calls. My graph is huge (>30 million arcs) and It took >8gb ram to construct my CSR. If I can have callback or file-based construction, I could cut down the memory usage by half.
You can use arbitrary iterators in most of the constructors, including the one you are using. You can use Boost.Iterator to create a custom iterator for your file data structure and use that with the CSR constructors. Note that some of the unsorted ones require multi-pass iterators, which would be harder to implement on top of a file, but edges_are_sorted is single-pass.
-- Jeremiah Willcock _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_________________________________________________________________ Got a phone? Get Hotmail & Messenger for mobile! http://go.microsoft.com/?linkid=9724464
participants (2)
-
Dan Jiang -
Jeremiah Willcock