How can my program collect anonymous usage data about itself?

Question

Some software has the option to collect anonymous usage data. How does that work? How is it collected and sent? I'd like to write a small test to try this myself, but I'm not sure where to begin.

Answer 1

Collecting data: You can accumulate whatever data you want to accumulate. For example, we wanted to know what forms our users were using (we have a lot of forms). So, in every form's FormCreate, we call code that appends Self.Name to a text file. Now we are tracking form creations, and the order in which our users visit the forms.

You could collect any data you wanted. For example, if you wanted to know how many times a user got a certain error message, whenever you show the error message, append the name of the error message to a file. If you wanted to know how long a user spent on a certain screen, you could note Now() when the form is opened and Now() when the form is closed and then write the difference to a data file.

Sending the data: You need to transfer either the raw data you collected to your server or you need to pre-process that data and send that. In our example above, we would just send the text file. You can use any internet library of your choice to upload a file to your server. We use Indy FTP, since it comes with Delphi. Upload the file, giving it a unique name (maybe a GUID if your server accepts that format?) to your server. (Choosing a good file name that hasn't already been uploaded to the server by another user's application is one of the challenges you'll have.) Be sure that you don't include anything in the file or file name that could be used to identify the user unless you've gotten permission to do so & understand any legal ramifications. Decide you want to upload the file maybe once a week, or once a day at a random time so all users aren't uploading at the same time. Of course you might want to pre-process the file before uploading it, collapsing the data in some way to make the file smaller.

Be sure your data collection file doesn't grow too big. You probably should delete it after uploading it. Also, if the upload file is big, it will cause a noticeable delay and freeze in your application unless you take steps to upload in the background, etc.

Note that if your users have very strict firewalls or security software, sending a file like this might be prohibited and could even cause your software to be flagged as malware. You'll need to carefully consider this issue and evaluate the various ways that data can be sent over the web in a way that is safe, unobtrusive and ANONYMOUS, and allowed by various security applications. For example, you will need to understand whatever protocol you use to upload and how much information that it might provide your server about the identify (like IP address, which might be vaguely personal with the right tools or search warrant.)

Then, at your server, over weeks (or whatever time frame you choose to upload files) you will have a lot of files that your software uploaded from users' machines. These files contain the names of the forms your users loaded, or the names of the error messages they got, or elapsed times, or whatever data you collected and uploaded. You would then decide how to process that data into a meaningful reports. Examining all the files, you might learn something like: 50% of our users never opened form X. Or: most users never saw error message #17 or only got error message #22 on form TForm3, or users spent an average of 45 seconds with Form4 visible.

I've simplified almost everything above. There are of course, for example, much better ways to save the collected data than appending to a text file. A text file might grow too big and too slow. There might be legal or ethical issues you'll need to consider.

But this is the general idea.

This is not a casual project to put into an application that others will use unless you fully understand all the issues and design and code it well. (That, I suppose, can be said about any coding!) But, as I noted, above I've written an overview of how you might do it for, say, an homework assignment or to just explore.)