2

I just installed this library that scrapes twitter data: https://github.com/kennethreitz/twitter-scraper

I wanted to find out the library's functions and methods so I can start interacting with the library. I have looked around StackOverflow on this topic and tried the following:

  • pydoc twitter_scraper

  • help(twitter_scraper)

  • dir(twitter_scraper)

  • imported inspect and ran functions = inspect.getmembers(module, inspect.isfunction)

Of the four things I have tried, I have only gotten an output from the inspect option so far. I am also unsure (excluding inspect) whether these codes should go in the terminal or a scratch file.

Still quite new at this. Thank you so much for reading everybody!

PaperRockBazooka
  • 157
  • 2
  • 9
  • 1
    The source code for the library is in the link you posted. Just look up the functions there https://github.com/kennethreitz/twitter-scraper/blob/master/twitter_scraper.py – Novaterata Nov 07 '18 at 02:23
  • See this aswell: https://stackoverflow.com/questions/713138/getting-the-docstring-from-a-function – stacksonstacks Nov 07 '18 at 02:31

2 Answers2

2

Excellent question! There are a few options in trying to grok (fully understand) a new library. In your specific case, twitter-scraper, the only function is get-tweets() and the entire library is under 80 lines long.

For the general case, in decreasing order of usefulness.

  1. Carefully read the project's description on GitHub. The ReadMe is usually the most carefully written piece of documentation.
  2. Larger libraries have formatted documentation at http://(package-name).readthedocs.org.
  3. pydoc module_name works when the module is installed. ``help(module_name)works in an interactive Python session after you have done animport module_name. These both work from the "docstrings" or strategically placed comments in the source code. This is also whatmodule_name?` does in iPython.
  4. dir(module_name) also requires an import. It lists all the entrypoints to the module, including lots of odd "dunder", or double underscore, you would not normally call or change.
  5. Read the source code. Often, this is easier and more complete than the documentation. If you can bring up the code in an IDE, then jumping around works quickly.

Also, you asked about what can be used within a script:

import os

print("Welcome, human.")
print("dir() is a function, returning a list.")
print("This has no output")
a_list = dir(os)
print("but this does", dir(os))
print("The help() command uses pydoc to print to stdout")
help(os)
print("This program is gratified to be of use.")
Charles Merriam
  • 19,908
  • 6
  • 73
  • 83
  • 1
    Thank you for the extensive answer! I am understanding that pydoc and dir() both can be accessed through writing the code within the script (as opposed to the terminal). Is this correct? – PaperRockBazooka Nov 07 '18 at 23:52
  • Hey Charles. Thank you very much. I got a better sense of how to access the information now. Just used it to better understand the sklearn library! – PaperRockBazooka Nov 16 '18 at 19:08
0

It seems like this library lacks proper documentation, but the GitHub page provides some usage examples to help you get started.

>>> from twitter_scraper import get_tweets

>>> for tweet in get_tweets('kennethreitz', pages=1):
>>>     print(tweet['text'])
P.S. your API is a user interface
s3monkey just hit 100 github stars! Thanks, y’all!
I’m not sure what this /dev/fd/5 business is, but it’s driving me up the wall.
…

To get more information, simply look at the source code at https://github.com/kennethreitz/twitter-scraper/blob/master/twitter_scraper.py. It seems like the only function is get_tweets, which, looking at the source code, takes in two arguments, the username and the number of pages (optional, defaults to 25).

iz_
  • 15,923
  • 3
  • 25
  • 40