Using regex in web harvest xml

Question

I'm using web harvest to scrap some e-commerce site.I'm iterating over the search page and getting each product details in output xml.But now I want to use regular expression in anchor(a) tag while scraping and get particular string.i.e.,

let $linktoprod :=data($item//a[@class="fk-anchor-link"]/@href)

The above line returns anchor tag href value of each product i.e., for first product the value returned is,

/casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&amp;ref=6df83d8f-f61f-4648-b846-403938ae92fa

Now I want to using the regular expression like /([^/\?]+)\? and get the string between last / and ? i.e.,

itmdaqmvzyy23hz5

in the output xml. Please anyone who has any idea regarding this help me. Thank you.

Updated -

  <?xml version="1.0" encoding="UTF-8"?>

<config charset="ISO-8859-1">

<function name="download-multipage-list">
        <return>
            <while condition="${pageUrl.toString().length() != 0}" maxloops="${maxloops}" index="i">
                <empty>
                    <var-def name="content">
                        <html-to-xml>
                            <http url="${pageUrl}"/>
                        </html-to-xml>
                    </var-def>

                    <var-def name="nextLinkUrl">
                        <xpath expression="${nextXPath}">
                            <var name="content"/>
                        </xpath>
                    </var-def>

                    <var-def name="pageUrl">
                        <template>${sys.fullUrl(pageUrl.toString(), nextLinkUrl.toString())}</template>
                    </var-def>
                </empty>

                <xpath expression="${itemXPath}">
                 <var name="content"/>  
                </xpath>
            </while>
        </return>
    </function>

    <var-def name="products">          
        <call name="download-multipage-list">
                 <call-param name="pageUrl">http://www.flipkart.com/watches/pr?sid=reh%2Cr18</call-param>  
                 <call-param name="nextXPath">//a[starts-with(., 'Next')]/@href</call-param>
         <call-param name="itemXPath">//div[@class="product browse-product "]</call-param>
         <call-param name="pids"></call-param>
            <call-param name="maxloops">5</call-param>
         </call>

    </var-def>
   <var-def name="scrappedContent"> 
    <!-- iterates over all collected products and extract desired data -->

    <![CDATA[ <catalog> ]]> 

        <loop item="item" index="i">
            <list><var name="products"/></list>
            <body>
            <xquery>
                    <xq-param name="item" type="node()"><var name="item"/></xq-param>
                    <xq-expression><![CDATA[
                            declare variable $item as node() external;

                    let $linktoprod :=data($item//a[@class="fk-anchor-link"]/@href)
                  let $name := data($item//div[@class="title"])

                      return
                      <product>
                      <link>{$linktoprod}</link>
                    <title>{normalize-space($name)}</title>

                     </product>
                    ]]></xq-expression>
                </xquery>

            </body>
        </loop>
        <![CDATA[ </catalog> ]]>

 </var-def>  
</config>

My config xml is as show above.Where to use regexp code block in my xml? And I want the regexp to be applied to linktoprod and finally get the regexp output in link tag as output xml.Please anyone guide me. Thank you.

Please show the current web harvest xml in your question. – Olaf Dietsche Jan 07 '13 at 08:48 — Olaf Dietsche, Jan 07 '13 at 08:48

Olaf Dietsche · Answer 1 · 2013-01-08T08:45:58.677

I don't know about web harvest, but if it supports a non greedy quantifier, you can use this pattern

/([^/]+?)\?

According to Web Harvest User manual - regexp you must insert something like this

<regexp>
    <regexp-pattern>/([^/]+?)\?</regexp-pattern>
    <regexp-source>
        /casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&amp;ref=6df83d8f-f61f-4648-b846-403938ae92fa
    </regexp-source>
    <regexp-result>
        <template>Last URL part is "${_1}"</template>
    </regexp-result>
</regexp>

In the <regexp-source> part you must insert your URL or variable to search for. Guessing from the manual and your config xml it might be something like

<regexp-source>
    <var>scrappedContent</var>
</regexp-source>

or

<regexp-source>
    ${linktoprod}
</regexp-source>

I think you must experiment a bit.

okay. But I dont know where and how to use and apply the regexp in the webharvest xml.So please guide me ragarding that. — Ashwini, Jan 07 '13 at 08:41
@user1785585 Just add the `` to your config xml. For the ``, I'm not really sure. You should look at the manual and try several combinations. — Olaf Dietsche, Jan 08 '13 at 08:49

Cerbrus · Answer 2 · 2013-01-07T08:45:11.987

0

Try this regex:

/([^/]+)\?

You might need to strip the leading / and trailing ?.

To illustrate that the regex works, this is it's result in JavaScript:

var s = "/casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&amp;ref=6df83d8f-f61f-4648-b846-403938ae92fa"
console.log(s.match(/\/([^/]+)\?/g)); //  /itmdaqmvzyy23hz5?

edited Jan 07 '13 at 08:45

answered Jan 07 '13 at 08:39

Cerbrus

70,800
18
132
147

Using regex in web harvest xml

2 Answers2