0

I'm trying to scrape SEC 10-Q and 10-K filings. Though I'm able to extract the tables, the CSV output is a bit messy. Is there any way that I can merge the columns with similar header names with pandas? Or any libraries that can help me export SEC filing data tables as csv?


[user@server sec_parser]$ /usr/bin/python3 /home/user/work_files/sec_parser/parser.py --file 10-Q-cmcsa-3312017x10q.htm 
                                                   0   1       2       3     4   5       6       7     8   9      10   11
0                                       (in millions) NaN    2017    2017  2017 NaN    2016    2016  2016 NaN    NaN  NaN
1                                             Revenue NaN       $   20463   NaN NaN       $   18790   NaN NaN    8.9    %
2                                 Costs and Expenses: NaN     NaN     NaN   NaN NaN     NaN     NaN   NaN NaN    NaN  NaN
3                          Programming and production NaN    6074    6074   NaN NaN    5431    5431   NaN NaN   11.8  NaN
4                  Other operating and administrative NaN    5827    5827   NaN NaN    5526    5526   NaN NaN    5.4  NaN
5                Advertising, marketing and promotion NaN    1530    1530   NaN NaN    1466    1466   NaN NaN    4.4  NaN
6                                        Depreciation NaN    1915    1915   NaN NaN    1785    1785   NaN NaN    7.3  NaN
7                                        Amortization NaN     587     587   NaN NaN     493     493   NaN NaN   19.0  NaN
8                                    Operating income NaN    4530    4530   NaN NaN    4089    4089   NaN NaN   10.8  NaN
9                   Other income (expense) items, net NaN    (625    (625     ) NaN    (554    (554     ) NaN   13.0  NaN
10                         Income before income taxes NaN    3905    3905   NaN NaN    3535    3535   NaN NaN   10.4  NaN
11                                 Income tax expense NaN  (1,258  (1,258     ) NaN  (1,311  (1,311     ) NaN   (4.1    )
12                                         Net income NaN    2647    2647   NaN NaN    2224    2224   NaN NaN   19.0  NaN
13  Net (income) loss attributable to noncontrolli... NaN     (81     (81     ) NaN     (90     (90     ) NaN  (10.2    )
14     Net income attributable to Comcast Corporation NaN       $    2566   NaN NaN       $    2134   NaN NaN   20.2    %

The sample table that I'm trying to convert as CSV https://edgartable.netlify.app/.

Here's my code

import os
import argparse
import sys
from bs4 import BeautifulSoup
import argparse
import pandas as pd


args = argparse.ArgumentParser()
args.add_argument('--file', type=str)
args.add_argument('--list', type=str)

opts = args.parse_args()


def parse_file(file):

    data_map = []
    div = []
    tables = []
    soup = BeautifulSoup(open(file, 'r'), 'html.parser')

    for div in soup.find_all('div'):
        if 'Consolidated Operating Results' not in str(div.find('font')): continue

        table = div.find('table')

        dataset = pd.read_html(str(table), skiprows=3)

        print(dataset[0])

        for i, data in enumerate(dataset):
            data.to_csv(f'test{i}.csv', '|', index=False, header=False)


def main():
    parse_file(opts.file)

if __name__ == "__main__": main()
Jacob Hernandez
  • 1

1 Answers1

1

Try this:

    import pandas as pd
    df = pd.read_html('https://edgartable.netlify.app/')
    df = df[0]
    df.to_csv('test.csv')
Sachin Gupta
  • 186
  • 1
  • 14