0

Sorry if the title is somewhat misleading. I'm using an InfiniteLoader with a Table and the problem is that almost always the total number of data I want to load is huge. And if I were to append the data every time loadMoreRows was called, I would end up storing maybe more than 100000 entries in the state, which I think would be bad for performance.

I was wondering if it's possible to not append data everytime. I tried only setting the logs and the status to loaded only from startIndex to stopIndex, but every time I scroll loadMoreRows is called multiple times.

Here is what I have so far, with what I tried as mentioned above

'use strict';

import React = require('react');
import _ = require('lodash');
import Immutable = require('immutable');
import moment = require('moment-timezone');
import {AutoSizer, InfiniteLoader, Table, Column} from 'react-virtualized';

interface Props {
    logEntries: Immutable.List<Immutable.Map<string, any>>;
    count: number;
    timezone: string;
    logLimit: number;
    loadedRowsMap: { [index: number]: number; };
    onLoadMoreRows: (param: {startIndex: number, stopIndex: number}) => Promise<any>;
}

class LogLoader extends React.Component<Props, {}> {
    render() {
        const {logEntries, count, logLimit} = this.props;
        const headers: {
            name: string;
            dataKey: string;
            width: number;
            cellDataGetter?: (param: {rowData: any}) => any;
        }[] = [
            { name: 'Time', dataKey: 'dtCreated', width: 95, cellDataGetter: this.renderTime.bind(this) },
            { name: 'Level', dataKey: 'levelname', width: 65 },
            { name: 'Message', dataKey: 'message', width: 70, cellDataGetter: this.renderMessage.bind(this) }
        ];

        return (
            <InfiniteLoader isRowLoaded={this.isRowLoaded.bind(this)}
                            loadMoreRows={this.props.onLoadMoreRows}
                            minimumBatchSize={logLimit}
                            rowCount={count} >
                {
                    ({onRowsRendered, registerChild}) => (
                        <AutoSizer disableHeight>
                            {
                                ({width}) => (
                                    <Table headerHeight={20}
                                           height={400}
                                           onRowsRendered={onRowsRendered}
                                           rowRenderer={this.rowRenderer}
                                           ref={registerChild}
                                           rowCount={count}
                                           className='log-entries'
                                           gridClassName='grid'
                                           headerClassName='header'
                                           rowClassName={this.getRowClassName.bind(this)}
                                           rowGetter={({index}) => logEntries.get(index)}
                                           rowHeight={this.calculateRowHeight.bind(this)}
                                           width={width} >
                                        {
                                            headers.map(({name, dataKey, cellDataGetter, width}) => 
                                                <Column label={name}
                                                        key={name}
                                                        className={`${name.toLowerCase()} column`}
                                                        dataKey={dataKey}
                                                        cellDataGetter={cellDataGetter || this.renderTableCell.bind(this)}
                                                        width={width} />
                                            )
                                        }
                                    </Table>
                                )
                            }
                        </AutoSizer>
                    )
                }
            </InfiniteLoader>
        );
    }

    private calculateRowHeight({index}) {
        const rowData = this.props.logEntries.get(index);
        if(!rowData) {
            return 0;
        }
        const msg = this.renderMessage({rowData});

        const div = document.createElement('div');
        const span = document.createElement('span');
        span.style.whiteSpace = 'pre';
        span.style.wordBreak = 'break-all';
        span.style.fontSize = '12px';
        span.style.fontFamily = 'monospace';
        span.style.display = 'table-cell';
        span.innerHTML = msg;

        div.appendChild(span);
        document.body.appendChild(div);
        const height = div.offsetHeight;
        document.body.removeChild(div);

        return height;
    }

    private rowRenderer(params: any) {
        const {key, className, columns, rowData, style} = params;
        if(!rowData) {
            return (
                <div className={className}
                     key={key}
                     style={style} >
                    Loading...
                </div>
            );
        }

        return (
            <div className={className}
                 key={key}
                 style={style} >
                {columns}
            </div>
        );
    }

    private renderTableCell({rowData, dataKey}) {
        if(!rowData) {
            return null;
        }

        return rowData.get(dataKey);
    }

    private renderMessage({rowData}) {
        if(!rowData) {
            return null;
        }

        return rowData.get('message');
    }

    private renderTime({rowData}) {
        if(!rowData) {
            return null;
        }

        return moment(rowData.get('dtCreated'))
            .tz(this.props.timezone)
            .format('HH:mm:ss.SSS');
    }

    private getRowClassName({index}) {
        const {logEntries} = this.props;
        const data = logEntries.get(index);

        if(data) {
            return `log-entry ${data.get('levelname').toLowerCase()}`;
        }

        return '';
    }

    private isRowLoaded({index}) {
        return !!this.props.loadedRowsMap[index];
    }
}

export = LogLoader;

and here is loadMoreRows passed down from the parent component

private loadMoreRows({startIndex, stopIndex}) {
    const {loadedRowsMap, logEntries} = this.state,
          indexRange = _.range(startIndex, stopIndex + 1),
          updatedLoadedRowsMap = {};

    indexRange.forEach(i => { updatedLoadedRowsMap[i] = STATUS_LOADING; });
    this.setState({ loadedRowsMap: updatedLoadedRowsMap, loading: true });

    return Api.scriptLogs(null, { id: this.props.id })
        .then(({body: [count, logs]}) => {
            indexRange.forEach(i => { updatedLoadedRowsMap[i] = STATUS_LOADED; });
            const newLogs = logEntries.splice((stopIndex - startIndex + 1), 0, ...logs).toJS();
            this.setState({
                count,
                logEntries: Immutable.fromJS(newLogs),
                loadedRowsMap: updatedLoadedRowsMap,
                loading: false
            });
        });
}
XeniaSis
  • 2,192
  • 5
  • 24
  • 39

1 Answers1

3

Updated answer

The key to understand what you're seeing involves a few things.

  1. First, you are wiping out the loadedRowsMap in your example each time you load new rows. (Once you load rows 200-399 you throw away rows 0-199.)
  2. You're setting a very large minimumBatchSize in your example (200). This tells InfiniteLoader that each chunk of rows it loads should- if at all possible- cover a range of at least 200 items.
  3. InfiniteLoader defines a threshold prop that controls how far ahead it pre-loads rows. This defaults to 15, meaning that when you scroll within 15 rows of an unloaded range (eg 185 in your case) it will load a block in advance- with the hope that by the time the user scrolls the rest of the way to that range, it will have been loaded.
  4. InfiniteLoader checks both ahead and behind by the threshold amount since a user could scroll up or down and either direction may contain unloaded rows.

This all leads to the following scenario:

  1. User scrolls until a new range of rows is loaded.
  2. Your demo app throws away the previous range.
  3. User scrolls a little further and InfiniteLoader checks 15 rows ahead and behind to see if data has been loaded in both directions (as per reason 4 above).
  4. Data appears to not have been loaded above/before and so InfiniteLoader tries to load at least minimumBatchSize records in that direction (eg 0-199)

The solution to this problem could be a few things:

  • Don't throw away your previously loaded data. (It probably takes up minimal memory anyway so just keep it around.)
  • If you must throw it away- don't throw away all of it. Keep some near the range the user is currently within. (Enough to make sure that a threshold check is safe in both directions.)

Original answer

Maybe I'm misunderstanding you, but this is what the minimumBatchSize prop is for. It configures InfiniteLoader to load large enough chunks of data that you won't get a lot of repeated load requests as a user is slowly scrolling through the data. If the user scrolls quickly- you might. There's no way around that though.

If that's problematic for you I'd recommend using a debounce and/or throttling approach to prevent firing off too many HTTP requests. Debounce can help you avoid loading rows that a user quickly scrolls past (and wouldn't see anyway) and throttle can help you avoid sending too many HTTP requests in parallel.

At a glance, your isRowLoaded function and the mapping done in loadMoreRows look right. But you might also want to verify that. InfiniteLoader is not stateful- so it will keep asking for the same rows over and over again unless you let it know that they're either loaded already or in the process of being loaded.

bvaughn
  • 13,300
  • 45
  • 46
  • I think you misunderstood my question. Let's see if I can explain it better. Every time `loadMoreRows` is called I just set the data from `startIndex` to `stopIndex`. For example, I store from 0-199 and then from 200-399, so 0-199 is empty. But whenever I scroll somewhere between 200-399, `loadMoreRows` is called for 0-199. I would like it to be called only when I'm about to render the indices before or after my stored data. Is that possible? – XeniaSis Feb 21 '17 at 12:02
  • That doesn't sound right. `InfiniteLoader` only scans and requests rows to be loaded that are currently rendered. Have this running anywhere I could see it in action? – bvaughn Feb 21 '17 at 17:46
  • Unfortunately no. I edited my `loadMoreRows` to the version I currently have and the logs I see when `console.log`ging `startIndex` and `stopIndex` are: 200 399, 0 199, 200 399, 0 199, 400 58727(my total count), 133 332. I don't get it honestly – XeniaSis Feb 22 '17 at 08:37
  • I created a sample fiddle. Not working there so you have to run it locally if possible. https://jsfiddle.net/XeniaSiskaki/0Ld61rdo/ – XeniaSis Feb 22 '17 at 12:07
  • This could have been made to work on JSfiddle. It's a lot to ask for someone to setup a local project, install deps, compile TS, etc. to repro a bug. :) – bvaughn Feb 22 '17 at 16:51
  • I'll try to do so. Sorry – XeniaSis Feb 22 '17 at 17:04
  • It's okay. I have a working version of it running in Plnkr now @ https://plnkr.co/edit/5AC6jbVng0JgAoWrduki?p=preview – bvaughn Feb 22 '17 at 18:25
  • There is something funky going on but I don't have the time to dig into it now. Day job ;) I'll try to look more this evening. – bvaughn Feb 22 '17 at 18:41
  • I have edited my answer with an explanation of why you are seeing this. – bvaughn Feb 23 '17 at 05:39
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/136435/discussion-between-xeniasis-and-brianvaughn). – XeniaSis Feb 23 '17 at 08:35