Stack Overflow
Stack Overflow

Questions tagged [xidel]

Xidel is a command line tool to download and extract data from HTML/XML pages as well as JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern templates. It can also edit or create new XML/HTML/JSON documents.

Xidel supports:

Extract expressions

  • CSS 3 Selectors: to extract simple elements
  • XPath 3.0: to extract values and calculate things with them
  • XQuery 3.0: to create new documents from the extracted values
  • JSONiq: to work with JSON apis
  • Templates: to extract several expressions in an easy way using an annotated version of the page for pattern-matching
  • XPath 2.0/XQuery 1.0: compatibility mode for the old XPath/XQuery version

Following

  • HTTP Codes: Redirections like 30x are automatically followed, while keeping things like cookies
  • Links: It can follow all links on a page as well as some extracted values
  • Forms: It can fill in arbitrary data and submit the form

Output formats

  • Adhoc: just prints the data in a human readable format
  • XML: encodes the data as XML
  • HTML: encodes the data as HTML
  • JSON: encodes the data as JSON
  • bash/cmd: exports the data as shell variables

Connections

  • HTTP / HTTPS, as well as local files and stdin

Systems

  • Windows (using wininet), Linux (using synapse+openssl), Mac (synapse)
81 questions
1
vote
1 answer

XPATH returning nothing

I'm trying to extract a data from this website, somehow I get nothing out of any text I'm trying to get. I'm using Xidel to scrape the data. xidel -e '//span[@class="main-price"]/text()' 'https://www.tokopedia.com/emas/harga-hari-ini' ****…
CuriousNewbie
  • 319
  • 4
  • 13
1
vote
3 answers

How to deal with empty node in xquery?

I want to print a TSV output like this. But when $x/span does not find anything, there will be only two columns printed. How to insert an empty field in the 2nd column when $x/span finds nothing? Thanks. for $x in…
user1424739
  • 11,937
  • 17
  • 63
  • 152
1
vote
2 answers

how to extract data from XML with xidel

I'm getting my feet wet with xidel and want to use it together with namesilo.com API for updating DNS records. I'm having trouble constructing the right selector. Let's say, I had the following xml response, how would I go about selecting the…
leggewie
  • 242
  • 2
  • 9
1
vote
2 answers

Xidel: How to process multiple result from one node?

Say, we have a code: xidel -s https://www.example.com -e '(//span[@class="number"])' and the output are: 111111 222222 333333 can I do this one below? for ((n=1;n<=3;n++)) do a=$(xidel -s https://www.example.com -e '(//span[@class="number"])…
CuriousNewbie
  • 319
  • 4
  • 13
1
vote
1 answer

Xidel: How to printout the node instead of the value?

html example:

Rp 910.000

so in xidel, i do this: xidel -se '(//p[@class="si-price"])[1]' 'https://www.anekalogam.co.id/id' what I want is 910000 which is in the data-price instead of Rp 910.000 can I…
CuriousNewbie
  • 319
  • 4
  • 13
1
vote
1 answer

Xidel: How to choose only 1 from many same value/class and remove unneeded elements from the result?

xidel -se '//strong[@class="n-heading"][1]/text()[1]' 'https://www.anekalogam.co.id/id' will print out 3 same outputs 15 June 2020   15 June 2020   15 June 2020   so, what should I do in order to choose…
CuriousNewbie
  • 319
  • 4
  • 13
1
vote
1 answer

How to select which line to scrape from a file with Xidel?

If you have a text file file.txt with multiple lines of text e.g. asd asd asdasd asdasd How do I select that I want to scrape line 2 asdasd asdasd? And select line 1 later on etc. for /f %a in ('^" xidel --data=file.txt --extract=$raw ^"') do set…
user3108268
  • 1,043
  • 3
  • 18
  • 37
1
vote
0 answers

xidel: add object into json array

I want extract some information on a website. For this, i use xidel. I use this command line: xidel --template-file template.xml --dot-notation=on and here my template file : ... code before ... …
Nicolas
  • 11
  • 2
1
vote
1 answer

xidel template extract from button tag

xidel https://www.url.com/folder -e "" I am trying to extract what's in between the single quotes with this xidel template and I am getting…
Lid
  • 115
  • 11
1
vote
1 answer

Extract values from xml document by xidel using xpath

Trying to extract 'text and value' pairs from all lines of DNS XML output using Xidel/Xpath but have no luck with such expression: xidel -e "/status/section[*]//item[*]@value/@text" In fact, I am not able to extraxt any single pair nor single value…
user2956477
  • 1,208
  • 9
  • 17
1
vote
3 answers

How to get the cell above in a table using XPath?

I'm trying to put all the info about each cell in a table in a single line. And I need to figure out how to print the header of each column in the table. td, table { border: 2px black solid; } …
user37421
  • 407
  • 1
  • 5
  • 12
1
vote
1 answer

list all json pointers with xidel

Is it possible to list/generate all the JSON Pointers from a file using xidel or another command line tool? I've tried with xidel test.json -e '$json()' But it only lists the top-level fields, while i want a recursive listing like the one i get…
eadmaster
  • 1,347
  • 13
  • 23
1
vote
2 answers

xidel how to follow pagination html and extract URL?

On windows 7 in batch and xidel I test on a website with pagination like this example : link1 link2 link3 1 2 3 4 5 6 7 8 9 10 Next i find a way to have first 10 links : xidel.exe https://www.website.es/search?q=xidel+follow+pagination^&start=0…
jhonitalia
  • 69
  • 8
1
vote
2 answers

How to extract class names from html tag

I am trying extract 2nd class name from tag. Due xidel documentation is really poor. I can't understand how to use function filter() or contains() and match tag with class name "userstatus" and extract 2nd class name. I have this at…
obi2016
  • 13
  • 5
1 2
3
4 5 6
a1