regex: parsing out varying length substr from varying string

by **Matt** » Thu, 08 Dec 2005 12:02:03 GMT

I am trying to parse some HTML with a Perl script. The script is not
working correctly. The parsing portion of the script reads:

getDealers {
	my $URL = shift;
	my $fullString = get($URL);
	my $startIndex = 0;
	my $endIndex = 0;

	my $thisDealerURL;
	my $thisDealerTitle;

	# dealer URL
	my $urlMatch = 'Bookseller: <a
href=\"/servlet/BookDetailsPL\?bi=\d\d\d\d\d\d\d\d\d&tab=1&searchurl=bx%3Doff%26sts%3Dt%26ds%3D100%26bi%3D0%26isbn%3D0805074562';
	my $endUrlMatch = "\"";
	my $urlPrepend = $baseURL .
'/servlet/BookDetailsPL\?bi=\d\d\d\d\d\d\d\d\d&tab=1&searchurl=bx%3Doff%26sts%3Dt%26ds%3D100%26bi%3D0%26isbn%3D0805074562';

# dealer name
	my  $dealerMatch = ">";
	my  $endDealerMatch = "</A>";


	while (1) {
		print ".";
		# get the URL
		$startIndex = index($fullString,$urlMatch,$startIndex) +
length($urlMatch);
		if ($startIndex == (-1 + length($urlMatch)) ) {
			print "\n";
			last; # break!
		}
		$endIndex = index($fullString,$endUrlMatch,$startIndex);
		$thisDealerURL = $urlPrepend .
substr($fullString,$startIndex,($endIndex - $startIndex));
		$startIndex = $endIndex; # advance the starting index to where we
stopped

		# get the title
		$startIndex = index($fullString,$dealerMatch,$startIndex) +
length($dealerMatch);
		$endIndex = index($fullString,$endDealerMatch,$startIndex);
		$thisDealerTitle = substr($fullString,$startIndex,($endIndex -
$startIndex));
		$startIndex = $endIndex;

}

---------------------------------------------

Here's a sample of the HTML:

<td class="bookseller" width="25%">Bookseller: <a
href="/servlet/BookDetailsPL?bi=602397062&tab=1&searchurl=bx%3Doff%26sts%3Dt%26ds%3D100%26bi%3D0%26isbn%3D%252F0805074562">Snowbound
Books</a><br/>
      <span class="scndInfo">(Marquette, MI, U.S.A.)</span></td>

<td class="bookseller" width="25%">Bookseller: <a
href="/servlet/BookDetailsPL?bi=527331473&tab=1&searchurl=bx%3Doff%26sts%3Dt%26ds%3D100%26bi%3D0%26isbn%3D%252F0805074562">LDS
Heritage Books</a><br/>
      <span class="scndInfo">(Bountiful, UT, U.S.A.)</span></td>

I want the script to put the dealer names (in this case, "Snowbound
Books" and "LDS Heritage Books") into the variable $thisDealerTitle. I
am pulling the HTML fine; when I ask it to print the source, it does
so. But running the above script returns "0 dealers loaded." Can anyone
see where I've gone wrong? Mere hints are appreciated.

thanks,
matt

by **Purl Gurl** » Thu, 08 Dec 2005 12:51:05 GMT

att wrote:

(snipped - see original article)

Your code is performing a lot more work than needed. You can reduce
your code to just a few lines, maybe four to seven lines, at the most.

Below, a conceptual example, this is, an example which displays a
methodology, not a solution.

Parsing html is very challenging. For simple needs, you can often be
successful. For complex pages, you will rarely be successful.

General rule is do not mix regex matching and substring / index functions.
Use one, but not both. Regex matching for html documents is not a good
practice, unless needs are very simple. Substring and indexing are well
suited for pulling data out of an html page when there is consistent format.
This type of page with which you are working, are often very consistent
in format. Markup and coding are of a template nature; predictable.

A frequently successful approach is to look for "common flags" throughout
your data. A flag, a marker, a begin and an end, these are the precise same
data, near or at, your data of interest. Look at these snippets of your data:

">Snowbound Books</a>

">LDS Heritage Books</a>

What is both common and predictable for both data sets?

"> </a>

Additionally, your data displays a format of a singular newline (\n) separator.
Otherwords, what some, in error, call a "blank line" between data sets.

You have two markers, a begin and an end, for your data. Use those to find
your data by numerical position, then start taking slices. In this case example,
I slice out what is not needed, then slice out what is needed. You will note I
start at the end of data, then move backwards. Sometimes it is easier to start
at the beginning of data, and move forward.

Basic concept is I take out what is not needed until target data is easy to grab.
This involves taking out identical flags which are "in the way" of data recovery.
Different wording, peel the onion until you reach the heart.

$/ = "";

Set the record separator to paragraph mode; two or more concurrent newlines.

$_ =~ tr/\n/ /;

Get rid of newlines with a space, so "words" do not run together. Extra spaces can be removed, later.

substr ($_, rindex ($_, "</a>"), length ($_), "");

Slice off a big chunk from the end of data to clean up. I am reverse indexing (rindex) to the first </a>
found, setting a length greater than needed to slice off everything, regardless of actual length, from
the _end_ of the data. This leaves the end of each book title exposed, otherwords, the last letter in
each book title, is the end of the data, which is my "true beginning point" for moving backwards.

print substr ($_, rindex ($_, ">") + 1), "\n";

Move backwards through data until the first > is found, move forward one character, capture and print.

This is a conceptual example. Rather than look for what you can "match" with a regex, look for
what is common, what is repetitive, what is predictable. For your data samples, there is a very
clear predictable pattern. Take advantage of that.

Only rule is, you must have uniform predictable data, or data which can be made so.

Piece of cake, yes? Just a matter of a little cleaning up, if needed; frosting for your cake.

Purl Gurl

#!perl

$/ = "";

while (<DATA>)
{
$_ =~ tr/\n/ /;
substr ($_, rindex ($_, "</a>"), length ($_), "");
print substr ($_, rindex ($_, ">

by **Matt** » Thu, 08 Dec 2005 13:19:33 GMT

Thank you--the "no mixing of regex/substr" guideline is very helpful.
The full HTML is a bit more complex than the above, but I think I can
work it out. I'll let you know what I come up with.

by **Purl Gurl** » Thu, 08 Dec 2005 13:31:26 GMT




No problem. I enjoy these types of articles. Solving this type of common
problem with little used methods presents a chance for all readers to
learn something, or affirm what they already know. All benefit.

Easy to remember trick for this methodology; look for what is predictable.

There is a gotcha! In time, the author of a page may change his format.
No doubt, that will toss a greasy monkey-wrench in your spinning gears.
If you are planning to pull this information on a regular basis, be sure to
check returns periodically.

This is a highly related article of mine, which employs the same methodology
but is very subtle; there are no "visual" flags with which to work. Looking for
the predictable in data, often is not what you see, but what you discern.

 http://www.**--****.com/ 

You could use the method exemplified in my cited article to "narrow" down
your data to single lines, very easily.

Look for the predictable, which is often quite invisible.

Purl Gurl

by **usenet** » Thu, 08 Dec 2005 14:16:06 GMT



Before you completely reinvent the wheel, you might check out:

   perldoc -q "How do I remove HTML from a string?"

and be sure to check out HTML::Parser on CPAN.

by **Matt** » Fri, 09 Dec 2005 06:42:13 GMT

Here's what I've got...the next step is to make it loop through the
entire page. Right now, I'm counting from the beginning of the HTML; I
need to make it advance $startindex to the next "Bookseller:"
occurrence (and likewise for $endindex). Or is the below code
loop-unfriendly?

Unlike the original HTML that I posted, there is lots of junk in
between each <td>. The junk is a repeating pattern, but the patterns I
use below are not repeated in the junk.

my $startsellertext = "Bookseller:";
my $startindex = index($fullString,$startsellertext);
my $endsellertext = "</a>";
my $endindex = index($fullString,$endsellertext);
my $sellertext = substr($fullString,$startindex,($endindex -
$startindex));
my $sellername = substr($sellertext,138,(length($sellertext)-138));
print "seller name is $sellername";

regex: parsing out varying length substr from varying string

regex: parsing out varying length substr from varying string

Jump to:

Next

regex: parsing out varying length substr from varying string

Re: regex: parsing out varying length substr from varying string

Re: regex: parsing out varying length substr from varying string

Re: regex: parsing out varying length substr from varying string

Re: regex: parsing out varying length substr from varying string

Re: regex: parsing out varying length substr from varying string

Similar Threads:

Who is online