Heavy load on thread causing memory issues

Question

In the logcat I keep seeing this:

I/art﹕ Background sticky concurrent mark sweep GC freed 141468(7MB) AllocSpace objects, 3(255KB) LOS objects, 25% free, 21MB/29MB, paused 1.228ms total 132.652ms

and this:

W/art﹕ Suspending all threads took:xxx milliseconds

And it causes me to get a org.apache.http.conn.HttpHostConnectException Connection to https://plcloud.c6.ixsecure.com refused around the last web service method call that I make.

How can I stop the "suspending all threads" and heavy garbage collection usage so that I can get through all of my web service method calls without running into memory issues?

At the moment it gets data from the web service and stores it in a local SQLite database. Here is my code to do so:

Code calling the sync classes "get" methods (which both get the data from the webservice and save it to the sqlite database):

class SyncThread extends Thread {
    @Override
    public void run() {
        try {
            syncHandler.sendEmptyMessage(16);
            Credentials credentials = new Credentials(usernameField
                    .getText().toString(), passwordField.getText()
                    .toString());
            UserCredentialsOut userCredentials = credentials
                    .getCredentials();

            if (userCredentials.getReturnCode() == 1) {
                Settings.getInstance().setEntityId(
                        userCredentials.getEntityId());

                syncHandler.sendEmptyMessage(1);
                CustomerSync custSync = new CustomerSync(context,
                        syncProgressDialog);
                Log.i("Customer Sync", "getCustomers");
                custSync.getCustomers(credentials);

                syncHandler.sendEmptyMessage(2);
                PetSync petSync = new PetSync(context, syncProgressDialog);
                Log.i("Pet Sync", "getPets");
                petSync.getPets(credentials);

                syncHandler.sendEmptyMessage(4);
                SystemSync systemSync = new SystemSync(context,
                        syncProgressDialog);
                Log.i("ListValues Sync", "getListValuesInitials");
                systemSync.getListValuesInitials(credentials);

                syncHandler.sendEmptyMessage(5);
                Log.i("Vets Sync", "getVets");
                systemSync.getVets(credentials);

                syncHandler.sendEmptyMessage(6);
                Log.i("Groomers Sync", "getGroomers");
                try{
                    systemSync.getGroomers(credentials);
                }catch (NullPointerException e) {
                    Log.e("NullPointerException", e.getMessage()+"benebne");
                }

                syncHandler.sendEmptyMessage(7);
                Log.i("Services Sync", "getServices");
                systemSync.getServices(credentials);

                syncHandler.sendEmptyMessage(8);
                Log.i("CalendarDays Sync", "getCalendarDays");
                systemSync.getCalendarDays(credentials);

                syncHandler.sendEmptyMessage(10);
                Log.i("loyeeUnavailTimes Sync",
                        "getEmployeeUnavailTimes");
                systemSync.getEmployeeUnavailTimes(credentials);

                syncHandler.sendEmptyMessage(12);
                Log.i("Runs Sync", "getRuns");
                systemSync.getRuns(credentials);

                syncHandler.sendEmptyMessage(14);
                Log.i("DaycareGroups Sync", "getDaycareGroups");
                systemSync.getDaycareGroups(credentials);

                syncHandler.sendEmptyMessage(3);
                OptionsSync optSync = new OptionsSync(context,
                        syncProgressDialog);
                Log.i("SystemOptions Sync", "getSystemOptions");
                SystemOptions systemOptions = optSync
                        .getSystemOptions(credentials);

                if (systemOptions != null) {

                    if (systemOptions.isModuleGroom()) {
                        syncHandler.sendEmptyMessage(9);
                        PetGroomSync petGroomSync = new PetGroomSync(
                                context, syncProgressDialog);
                        Log.i("PetGrooms Sync", "getPetGrooms");
                        petGroomSync.getPetGrooms(credentials);

                        syncHandler.sendEmptyMessage(11);
                        AppointmentSync appointmentSync = new AppointmentSync(
                                context, syncProgressDialog);
                        Log.i("Appointments Sync", "getAppointments");
                        appointmentSync.getAppointments(credentials);
                    }

                    if (systemOptions.isModuleBoard()) {
                        syncHandler.sendEmptyMessage(13);
                        AppointmentBoardSync appointmentBoardSync = new AppointmentBoardSync(
                                context, syncProgressDialog);
                        Log.i("ApptBoards Sync", "getApptBoards");
                        appointmentBoardSync.getApptBoards(credentials);
                    }

                    if (systemOptions.isModuleDaycare()) {
                        syncHandler.sendEmptyMessage(15);
                        AppointmentDaycareSync appointmentDaycareSync = new AppointmentDaycareSync(
                                context, syncProgressDialog);
                        Log.i("ApptDayCares Sync", "getApptDayCares");
                        appointmentDaycareSync.getApptDaycares(credentials);
                    }
                }

                syncHandler.sendEmptyMessage(0);
                return;
            } else {
                syncHandler.sendMessage(syncHandler.obtainMessage(-1,
                        userCredentials.getMessage()));
                return;
            }

        } catch (ClientProtocolException e) {
            Log.e("ClientProtocolException", e.toString());
        } catch (IOException e) {
            Log.e("IOException", e.toString());
        } catch (Exception e) {
            Log.e("SyncException", e.toString());
        }

        syncHandler.sendMessage(syncHandler.obtainMessage(-1,
                getString(R.string.connError)));
    }
}

One actual sync class - All sync classes are roughly the same so you only need to see one of them - take note of the "getCustomers" method which calls "initCustomersFromJson" - these are the parts that get the data and then save it to the sqlite db:

public class CustomerSync implements ICustomerSync {
    public static final int QUANTITY = 0;
    private long offset;
    private ProgressDialog progressDialog;
    private Context context;
    private HttpClient client;
    private final String HTTPS_GET_CUSTOMERS = "https://plcloud.c6.ixsecure.com/PLService.svc/GetCustomers";
    private final String GET_URL = "{0}?quant={1}&offset={2}";
    private final String HTTPS_SYNC_CUSTOMERS = "https://plcloud.c6.ixsecure.com/PLService.svc/SyncCustomers";

    private CustomerSync() {
        client = new DefaultHttpClient();
    }

    public CustomerSync(Context context, ProgressDialog progressDialog) {
        this();
        this.context = context;
        this.progressDialog = progressDialog;
    }

    public Customer[] initCustomersFromJson(Credentials credentials)
            throws ClientProtocolException, IOException, URISyntaxException {
        String getUri;
        getUri = MessageFormat.format(GET_URL, HTTPS_GET_CUSTOMERS,
                QUANTITY + "", offset + "");

        credentials.initGetOAuthStructure(HTTPS_GET_CUSTOMERS);

        HttpGet request = new HttpGet(getUri + "&"
                + credentials.getOauthStructure());
        String content = client.execute(request, new BasicResponseHandler());

        Gson gson = new GsonBuilder().serializeNulls()
                .excludeFieldsWithoutExposeAnnotation().create();
        return gson.fromJson(content, Customer[].class);
    }

    @Override
    public void getCustomers(Credentials credentials)
            throws ClientProtocolException, IOException, URISyntaxException {
        ICustomerDAO customerDAO = (ICustomerDAO) DAOFactory.getDAO(
                ICustomerDAO.class.getName(), context);
        customerDAO.open();

        Customer[] customers;


            customers = initCustomersFromJson(credentials);
            progressDialog.setMax(customers.length);
            progressDialog.setProgress(0);
            customerDAO.saveCustomers(customers, progressDialog);

            if (customers.length > 0) {
                offset = customers[customers.length - 1].getId();
            }


        customerDAO.close();

        Settings.getInstance().setLastSyncCustomerDatetime(new Date());
    }

@nayoso Yes. There are 1160 customers. And a customer has quite a few properties. Then it goes and gets all the pets and there are a lot of them too, and so on. — BeniaminoBaggins, Jul 28 '15 at 01:09
use some type of paging so you dont have to download all the customer data at once — Eoin, Jul 28 '15 at 01:13
@Modge We aren't displaying the data during this code execution. We just need to get all of the data and save it to the local sqlite database. But yes, that may be important for the other activities that do display the data. — BeniaminoBaggins, Jul 28 '15 at 01:15
possibly a good idea then to still page it. Download some at a time save it to the DB then move onto the next page. I see you have a quantity and offset you can use — Eoin, Jul 28 '15 at 01:17

Niko Adrianus Yuwono · Accepted Answer · 2015-07-29T01:16:07.163

1

I think it's better to stream the data if the data really big. I see in your code you already using GSON and you can use GSON to stream the json data.

For example :

        String getUri;
        getUri = MessageFormat.format(GET_URL, HTTPS_GET_CUSTOMERS,
                QUANTITY + "", offset + "");

        credentials.initGetOAuthStructure(HTTPS_GET_CUSTOMERS);

        HttpGet request = new HttpGet(getUri + "&"
                + credentials.getOauthStructure());
        HttpResponse response = client.execute(request);

        Reader streamReader = new InputStreamReader(response
                    .getEntity().getContent());
        JsonReader reader = new JsonReader(streamReader);
        ArrayList<Customer> customerList = new ArrayList<Customer>();
        reader.beginArray();
        while (reader.hasNext()) {
            Customer customer = new Customer();
            reader.beginObject();
            while (reader.hasNext()) {
                String name = reader.nextName();
                if (name.equals("Action")) {
                    customer.setAction(reader.nextString());
                } else if (name.equals("Addr1")) {
                    customer.setAddr1(reader.nextString());
                } else if (name.equals("Addr2")) {
                    customer.setAddr2(reader.nextString());
                } else if (//Other condition and value)
                    ...
                } else {
                    reader.skipValue();
                }
            }
            reader.endObject();
            customerList.add(customer);
        }
        reader.endArray();
        reader.close();

With JsonReader you can stream the data one by one and process it directly rather than saving all into one Array and then process it which can cause a memory issue.

edited Jul 29 '15 at 01:16

answered Jul 28 '15 at 01:18

Niko Adrianus Yuwono

11,012
8
42
64

Thanks. I'm finding it hard to get this to work. I have a new question over [here](http://stackoverflow.com/questions/31688637/expecting-a-string-but-was-begin-object-jsonreader?noredirect=1#comment51319418_31688637) – BeniaminoBaggins Jul 28 '15 at 23:38
@user3935156 No need to post a new question. Just post your JSON example and I will show you how to parse it :) – Niko Adrianus Yuwono Jul 28 '15 at 23:58
Thanks. The JSON seems to have been cut off, when I get it this way. The value of "reader" is "JsonReader near [{"Action":null,"Addr". So Seems to be cut off near the start of the JSON. I also get this exception: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was BEGIN_ARRAY I think that occurs on the reader.hasNext() line. I changed it to "beginArray" and had more issues. – BeniaminoBaggins Jul 29 '15 at 00:02
@user3935156 Are you printing it into logcat? if yes it will be cutted because it is too long. try to make a call from hurl.it (https://www.hurl.it/) or postman (https://www.getpostman.com/) – Niko Adrianus Yuwono Jul 29 '15 at 00:11
I think you are right. Because I got the full JSON as a string and then I converted that string to an InputStream and then with that inputStream I continued with your code to make the JsonReader, and continued on to hasNext() and the value was the same, it always looks like it is cut off, so maybe it is not cut off. Can I talk to you in a private chat to help me get to a solution? I want to show you the JSON privately. – BeniaminoBaggins Jul 29 '15 at 00:50
@user3935156 yes you can, let's talk at stackoverflow chat here -> http://chat.stackoverflow.com/rooms/84518/chat-with-user3935156 – Niko Adrianus Yuwono Jul 29 '15 at 00:53

Heavy load on thread causing memory issues

1 Answers1