Insufficient memory

[Bancam]

I can’t display a cluster grouped network of about 43000 people. Sometimes I can see the graph, but generally just an error message about shutting down other programs or restarting (all of which I have done).
My computer has 8GB of memory, Windows 10 1803 build 17134.407, has an i7-4650U processor with 4 logical cores. When I check resource monitor after starting up, a bit more than one third of memory is in use, ditto standby. Free memory is about 1700 MB. Excel, Defender and Photos are the biggest individual memory users, unless I run Chrome.
I don’t think I can add more memory (laptop)
Should I run it on my wife’s Windows 8.1 desktop instead?
How much free memory do I need before I start Excel/NodeXL spreadsheet?

[Marc Smith]

“OutOfMemoryException” – What does that mean?

If this message shows up, you have reached the memory storage limits of your machine.

How large a network can NodeXL handle?

NodeXL can manage different numbers of edges based on the available computer system resources. The following are rough estimates:

4 GB of RAM: SMALL networks only less than a few thousand edges
8 GB of RAM: MEDIUM networks of less than 10-15 thousand edges
16GB of RAM: LARGE networks of less than 80-100 thousand edges
32 GB of RAM: VERY LARGE networks of less than 200 thousand edges

How to deal with a VERY LARGE NETWORK?

Network analysis can require large amounts of computing resources or require long periods of time to perform (or both!).

Reducing the number of metrics calculated is one way to speed up analysis of a very large network.

If you do not plan on using all the different types of “centrality” metrics, you may just want to skip creating them. To do this, modify the metrics selected in the NodeXL>Analysis>Graph Metrics dialog.

In many cases network data sets can be reduced in useful ways that retain many important features and insights.

In many networks it can be possible to count and then remove all “isolate” vertices (nodes with zero connections).

All “singleton” or “pendant” vertices can also be counted and removed (these are vertices with just one connection to the network).

All duplicated edges can be removed. Optionally, duplicate edges can be counted and removed and the count added to the remaining edge as a weight representing the number of removed edges.

To perform this operation, see: NodeXL>Data>Prepare Data> Count and Merge Duplicate Edges:

[Marc Smith]

Please consider scheduling a discussion with our consultants – we can help you navigate large data sets through NodeXL most effectively!

Contact us via: info@smrfoundation.org

[Bancam]

Thanks for the details Mark.
I will turn off the metrics, but the specific error was insufficient memory to layout the chart.
I will take you suggestion about singletons and duplicate edges, but for a quick analysis, I might be better to reduce the number of records involved, especially because I can’t upgrade the memory of my laptop. I was surprised that NodeXL would not cache it, or is that an excel thing. Maybe I reached a limit there too.
I opted for Cluster grouping to simplify the chart. Once I get to that point, I might give some feedback about small improvements with the interface.
I’m still exploring NodeXL to analyse my data, but in the short term I have had to resort to manually examining it for key connections. Once connections are found, of course I have to go elsewhere to see what they mean.

Cheers,
Peter

[Author]

Posts