0

I'm using Shodan's API through the Anaconda Terminal on Windows 10 to get data against the query below, but after a few seconds of running, the ETA timer freezes, and my network activity drops to zero. Hitting Control+C restarts it when this happens and gets it moving for a few seconds again, but it stops soon after.

shodan download --limit 3100000 data state:"wa"

Also, when it is running- the download speeds seem pretty slow; and I wanted to inquire if there was any way I can speed it up? My Universities internet is capable of upwards of 300 Mbps, but the download seems to cap at 5 Mbps.

I don't know how to solve either of these issues; my device has enough space and my internet isn't disconnecting. We've tried running the Anaconda Terminal as an Administrator, but that hasn't helped either.

JsDart
  • 183
  • 13

2 Answers2

1

I am not familiar with the specific website, but in general seeing limited speed or stopped downloads are not caused by things 'on your side' like the university connection, or even your download script.

Odds are that the website wants to protect itself, and that you need to use the api differently (for example with a different account). Or that you have some usage limits in place based on your account, that you hit.

The best course of action may be to contact the website and ask them how to do this.

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122
  • From what I'm reading; increasing the amount of data that is stored in memory before writing may help with this. I'm exploring ways on how to do this at the moment. – JsDart Mar 02 '19 at 23:51
  • 1
    @JsDart no, those aspects don't factor into what you're observing. Please wait to hear back from our support staff. I noticed you sent an email but it's weekend for us so we haven't yet responded; will get back to you during business hours. – achillean Mar 03 '19 at 03:08
0

I heard back from Shodan support; cross-posting some of their reply here-

The API is not designed for large, bulk export of data. As a result, you're encountering a few problems/ limits:

  1. There is a hard limit of 1 million results per search query. This means that it isn't possible to download all results for the search query "state:wa".
  2. The search API performs best on the first few pages and progressively responds slower the deeper into the results you get. This means that the first few pages return instantly whereas the 100th page will take potentially 10+ seconds.
  3. You can only send 1 request per second so you can't multiplex/ parallelize the search requests.

A lot of high-level analysis can be performed using search facets.

There's documentation on facets in the shodan.pdf booklet floating around their site for returning summary information from their API.

JsDart
  • 183
  • 13