300,000+ XML Nodes

by **ChrisFlynn** » Sat, 07 Jul 2007 00:24:23 GMT

Hi all

 // begin sarcasm
 So I have this little XML file, with 300,000+ nodes (318,006 lines in this XML 
code to be exact).
 // end sarcasm

 Right now, I'm loading this monster in. When I finish loading, I want to take 
all these nodes and save them to an Object, so I can easily reference them 
later.

 The way I'm doing this now is with a for-loop, which in the end, gathers about 
5000 nodes, each with anywhere between 10-30 children.

 As best as I can tell, I have gotten this working successfully to this point. 
The problem I'm having now is that with this much data, Flash chokes on it a 
bit. So after my load sequence, there's about 5 or 6 seconds where nothing 
happens, then everything suddenly works.

 Now granted, this test XML I'm using is a worse-case-scenario. And I think 5 
or 6 seconds isn't bad. But I'm on a fairly fast and new machine and I'm 
worried about a lesser machine's performance. 

 So is there a better way to do this? Should I not use a for-loop? Should I not 
save this data much data to an Object?

 Any suggestions are welcome.

 Thanks
 Chris

by **GWD** » Sat, 07 Jul 2007 01:48:08 GMT

<genuineQuestion>
 Shouldn't that be ?
   <sarcasm>
   So I have this little XML file, with 300,000+ nodes (318,006 lines in this 
XML code to be exact).
   </sarcasm>
 </genuineQuestion>

 Its certainly a hefty load. I've never tried working with data that large 
before. 
 I'd run through a few things for testing...
 Isolate timings with getTimer to understand how long the whole process is 
taking.
 load time: difference between load command issued and onData 
 parsing time: difference between onData (see the livedocs) and (you need to 
call it explicitly) onLoad
 processing time: your looping and extraction of portions. You're right this is 
likely to be the part that takes a long time if you're looping through and 
possibly recursively down the xml structure to pick out data.

 Questions I would ask myself: 
 -would it make sense (if possible) to have the source xml provided in 
bite-size chunks?
 -does XPathAPI or xfactorstudios xpath offer any improvement for the search 
and extraction part of what you're doing? This way you may be able to separate 
the node location logic and the new object representation. In saying that, its 
also possible (and is much easier with an XPath implementation) to make use of 
the data directly in its XML object representation.

 Nothing concrete there I'm afraid... but you said that all suggestions were 
welcome...and I really wanted to put the bit about the sarcasm tags at the 
beginning.

by **ChrisFlynn** » Sat, 07 Jul 2007 02:06:08 GMT

Ive seen sarcasm tags both ways.

 I'm actually working on an attempt with onEnterFrame instead of for-loops. My 
hope is with the onEnterFrame, I'd be able to incorporate some sort of Process 
Bar that will at least let the user know something is going on. The for-loop 
won't really let you do that.

 One of my thoughts was to do try and get this XML data into smaller bits. But 
since I'm not actually developing the XML end of things, that's nothing I have 
the final say so in. The XML we're using is dynamically created from a 
database, and apparently is fairly complex to get it the way we have it now. 
That might have to wait till version 2.0. 

 Thanks again GWD
 Chris

by **Rothrock** » Sat, 07 Jul 2007 02:36:20 GMT

I would definitely recommend trying it on an older slower computer. You can 
certainly break up the part where you are searching through like you have 
suggested, but the initial parse by Flash is not something you can break up and 
I've seen folks post here about getting the "slow script" error during that 
part.

 If the file is dynamically created from a database it should be fairly easy to 
break it into chunks which could be loaded, parsed, and then appended together 
in Flash. It would break up some of that initial parse.

by **GWD** » Sat, 07 Jul 2007 06:25:59 GMT

I've never tried anything like this...  but if the internal parsing by flash is 
a major factor and you are not able to get the source split into smaller 
chunks.....

 Perhaps its possible to break up the parsing of the raw xml string from inside 
your onData handler... e.g. it would presumably work best with well spaced, 
infrequent closing tags from a child of the root node (that does not occur any 
lower in the hierachy, anywhere) to use for a String.split and would also 
require tweaking each subsequent string element to add back the closing tag 
used for the split and closing/opening root tags - either or both, depending on 
its index in the array of strings-  before parsing each bit...
 I don't know if its practical or makes sense  but its something else to 
explore as an idea/option. Its a definite second to breaking it down at the 
source, but it may be another option.

by **ChrisFlynn** » Sat, 07 Jul 2007 08:12:51 GMT

You know, the onEnterFrame method seems to have worked for me, but its much 
much slower. I think its slower, because I'm actually displaying what node is 
being processed, as its being processed. So it looks like:

 0 of 5000 processed
 1 of 5000 processed
 2 of 5000 processed

 ...

 5000 of 5000 processed.

 Of course with that many, it took a full 2 minutes, but it didn't freeze up. 

 My hope is that with using setInterval, and not displaying each one as they're 
processed, this will go much faster and it won't lock up.

 Thanks again
 Chris

by **Rothrock** » Sat, 07 Jul 2007 08:17:33 GMT

I wouldn't necessarily do one node on each frame. You will probably be wasting 
time then. Certainly there is some chunk size that can be handled quickly in a 
frame or a setInterval. I would probably try and process them in chunks of 1000 
or 5000 on each frame and see how that went.

by **Newton** » Sat, 07 Jul 2007 12:51:41 GMT



i would rather suggest to break up your xml file, and load the needed
part, as because if you are loading  much data then in some cases the
IE or Firefox may crash due to lack of memory.

Pradyumna

300,000+ XML Nodes

300,000+ XML Nodes

Jump to:

Next

300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Re: 300,000+ XML Nodes

Similar Threads:

Who is online