29

I am going to launch my app on the AppStore and I would like to keep track of crashes and fix them as soon as possible. If possible, it would be nice to collect also some additional information about user activity and other useful stuff. In order to do so, I have looked for some crash reporting tools and the most interesting ones I have found are: TestFlight Live, QuincyKit and Crashlytics.

Among those three, QuincyKit should be the lightest one but the other two seem to be quite interesting since they provide more sophisticated reports and other interesting stuff.

My goal is to have as many information as I can on any issue the user can experience, but at the same time I don't want to make the app slower or consume more resources.

  1. In your opinion and from your personal experience, which of these tools is the best one (taking into account my goal and my needs)?
  2. By using TestFlight Live or Crashlytics I would make my app too slow?
  3. Is there a risk to overload the device?
  4. Reports provided by QuincyKit are precise enough? How many information can I retrieve from them?

Thanks!

Here is what I decided:

I am using Crashlytics for crash reporting (yes, it seems to be really great) and TestFlight for tracking user activity (checkpoints are really useful to find out what users generally do and figure out what the tendency is). I followed the instructions written here

pAkY88
  • 6,262
  • 11
  • 46
  • 58

5 Answers5

43

I honestly think Crashlytics is a better solution than Testflight for crash reporting.

Here's what you get with Crashlytics that you don't get with the others.

  • Duplicate culling (TF does this too, but its not too great at it, Crashlytics is damned near perfect)
  • You can actually mark crashes as closed/resolved, and get them out of your list for a given version.
  • Crashlytics does everything TF's Crash reporting does, but better and then some (logging, stack traces, etc.)
  • Percentage of affected users, and the numbers that go with that. (ie: should i fix the bug that happened to one guy, or the one that's happening to 10k?) Testflight doesn't tell you this.
  • Prioritization based on occurrence. This is probably the most important gain in my opinion.

These are just a few, but I figure they're probably the most important ones for you.

We used Testflight's crash reporting for close to 2 years on an extremely popular app (several million D/Ls). Its definitely better than nothing, and very convenient if you're using TF for distribution as well, however you get many more benefits from Crashlytics. We switched to Crashlytics this summer and now we're actually able to manage crashes and make smart decisions about what to fix and when, instead of just sifting through a giant never ending list.

I see you already accepted an answer, but I'd seriously give it another look even if you opt to continue with Testflight. I've found its hard to really grasp what you're missing until your app has shipped, at which point is even harder to change.

bsneed
  • 579
  • 4
  • 8
  • Thank you very much for your very detailed answer! I will definitively have a look at Crashlytics and after comparing it with TF I will make the final decision! ;-) – pAkY88 Dec 27 '12 at 21:18
  • anyone can please tell or suggest the tutorial for using/ integrating Crashlytics in ios application.... – Venk Jun 06 '13 at 05:58
16

Crashlytics is second to none for crash reporting.

We were in the same boat as you trying to find the best crash reporting solution. After some thorough investigation and test runs of TestFlight, HockeyApp, and Crashlytics, we originally chose HockeyApp because they allowed us beta distribution along with crash reporting on both iOS and Android (we wanted both in one solution for both platforms). However, HockeyApp's exception backtracing was just not giving us any additional crash details. This is where Crashlytics shines. Their exception backtracing is amazing. Period.

So here's my summary of all 3 SDKs:

Crashlytics

  • #1 crash reporting
  • #1 exception backtracing, bar none (provides very useful extra crash details)
  • Extremely fast and lightweight
  • Custom key logging for additional crash context
  • Best duplicate crash recognition and culling
  • Automatic SDK updates (Their Mac app automatically updates the Crashlytics iOS SDK in your project)
  • No beta distribution (I'd love a one stop solution for crash reporting and beta distribution)
  • Automatic build server support

TestFlight

  • Somewhat heavy, and adds bloat to your app package
  • Great beta distribution
  • No Android support (at least when we tested back 6+ months ago)

HockeyApp (HockeyKit - Beta distribution, QuincyKit - Crash reporting)

  • Lightweight
  • Crash reporting UI a bit confusing
  • Exception backtracing severely limited (at least when we tested in March of 2011)
  • Very good beta distribution

All that said, we chose Crashlytics for crash reporting, and HockeyApp for beta distribution. But you must chose what works best for your needs.

Bryan
  • 614
  • 1
  • 8
  • 18
  • Could you please contact us (HockeyApp) and tell us what is wrong with stack unwinding? We'd also love to know what about the crash reporting UI you find confusing, so we can improve it. – Kerni Jan 03 '13 at 22:08
  • 2
    Can you expand a bit on what you mean by "#1 crash reporting", "#1 stack unwinding", and "extremely fast and lightweight"? – landonf Jan 03 '13 at 22:08
  • @Kemi, some of Hockeyapps more complicated crash reports never gave us any useful information in determining what caused the crash; meaning your ability to unwind the stack of what exactly was going on prior to the crash, was not useful, leaving us very vague crash reports. Apples crash reports have the same problem. For example, we saw many main.m crashes, reason:"No reason found.", with an unhelpful stack trace. The most confusing part of your website is the overall UX. There are multiple rows & levels of tabs, and very confusing when you get many levels deep, how to just go "back" one level – Bryan Jan 03 '13 at 22:41
  • @landonf #1 crash reporting means its the best overall solution. It has the best webUI (with some improvements needed), the easiest integration, and the best crash report results. Bar none. #1 stack unwinding means that they provide the best and most helpful stack trace and process breakdown of what exactly was going on when your app crashed, instead of returning some unhelpful results (see my comment above about Hockeyapp's results). Extremely fast and lightweight means their SDK takes up very little device cpu time (no app blocking, etc), and very little additional app bundle space. – Bryan Jan 03 '13 at 22:47
  • It would be great to have examples regarding getting different stack traces to improve the underlying open source framework PLCrashReporter. In general, having no reason and none of you app's calls in the crashing thread is mostly a memory leak or zombie that can be anywhere in your code and is mostly shown by crashes mentioning "obj_msgSend" in the crashing thread. – Kerni Jan 03 '13 at 23:23
  • I'm not suggesting that the stack traces are different (I would have to test both frameworks with a known crash to verify). I'm stating that even if the stack traces are exactly the same, sometimes having the stack trace isn't always enough. The exception type and reason for crashing (which Crashlytics supplies, not sure how they determine those bits of info, but they do) can be the extra bit of info you need to fix a very difficult crash to debug, let alone reproduce. This is and has been the problem with PLCrashReporter; it just doesn't give enough helpful information regarding the crash. – Bryan Jan 04 '13 at 01:36
  • PLCrashReporter has included the exception type, reason, and backtrace since September of 2011 (on top of which, Crashlytics' initial version was actually based on PLCrashReporter). – landonf Jan 04 '13 at 08:16
  • I'm going to submit an edit to this review; The evaluation of HockeyApp predated both Crashlytics and PLCrashReporter's support for providing the exception backtrace. When Crashlytics *did* ship in 2011, it too was based on PLCrashReporter, and had (and has) the same exception handling functionality as HockeyApp. As of today, as far as I am aware, they both present the same information for exceptions. On top of this, "#1 crash reporting" and "#1 exception backtracing" aren't useful descriptions of an engineering solution, or accurate. – landonf Jan 06 '13 at 16:36
  • 1
    When Crashlytics first shipped, it was based on PLCrashReporter, and was subject to the same shortcomings Bryan and others have pointed out. So we wrote our own (see: http://www.crashlytics.com/blog/major-update-crashlytics-for-ios-2-0/ for lots of details). – Jeff Seibert Jan 07 '13 at 00:30
  • 1
    @Kerni To summarize, PL has a couple major shortcomings: 1. Its reliance on signal handlers instead of Mach Exceptions prevents it from detecting/capturing a number of types of stack overflow crashes. 2. Its stack unwinder misbehaves when objc_msgSend crashes, obscuring the calling function. To be clear, this is only a summary - we've collected extensive crash data in the process of tuning our techniques and there are dozens of scenarios in which PL/HockeyApp fails to collect complete information. – Jeff Seibert Jan 07 '13 at 00:31
  • 3
    Historically PLCrashReporter worked with sigaltstack(), but Apple broke that in later releases. Mach exception handlers come with issues on iOS; namely, the APIs are not fully public, which means you have to rely on undocumented structures and constants to implement something complete. I've written up my initial findings here: http://bit.ly/Woghtd. Once I get around to wrapping up that implementation, I'll probably write up something a bit more formal. As for the rest, it's unfortunate that you haven't shared any of that information to help improve a completely free OSS library you relied on. – landonf Jan 07 '13 at 15:22
  • 1
    Regardless, I'll be deploying enhanced stack unwinding in the next release (again, as OSS, for free, on my own time), which ought to improve some of the edge cases that we *are* aware of; if there are any other pointers you can share, I'd love to hear them. I'll note that in the case of a crash in objc_msgSend(), the necessary information *is* available in the crash report via the register dump, and it's mainly an issue of interpreting that correctly on the server. – landonf Jan 07 '13 at 15:26
  • 1
    @Jeff Oh, and of course, if there are any issues with my evaluation of mach handling, any insight you can provide would be most appreciated. I'll probably wrap up the implementation this week and continue to see if I can't find any reasonable work-arounds for the stated issues. Hopefully Apple will get back to me soon too, seeing as I spent a DTS incident on the subject. I'll probably write up something formal once they do. – landonf Jan 07 '13 at 15:47
  • What's the difference between Backtrace in TF and in Crashlytics? Which one is better and why? – pronebird Oct 25 '13 at 10:16
9

Definitely recommend Crashlytics as well.

TestFlight Live has given me issues in the past. It seems every time I go to use TestFlight, it is down anyway.

Crashlytics is awesome. Here's why:

  • Adding it to your project couldn't be easier. There is a Mac app that does most of the hard work for you.
  • Automatically updates itself
  • Prioritizes crashes for you
  • Provides handy stats like OS and device percentages as well as average memory available, etc

I use Crashlytics in all of my applications. I added it to Hipstamatic when I was there and the data we got was shocking. It really helped improve our product. I also tried TestFlight Live and quickly removed it after the first beta since it was causing crashes.

Crashlytics is awesome. You should use it.

Sam Soffes
  • 14,831
  • 9
  • 76
  • 80
6

If we're only talking about crash reporting, Crashlytics is far better than TestFlight. (Never tried QuincyKit, so I can't compare the 3 options)

We've been using Crashlytics for over a year on Weddar and it has been great. Having tried other solutions before I have to say that before installing it I was kind of suspicious of the great features they were stating, but the installation was indeed done in about 5 minutes and it only added around 40-45Kb to the app.

The crash reports are incredibly detailed making it really fast to pinpoint solutions for the bugs and the updates to the sdk are pretty steady and stable. The team is incredibly supportive too. I remember that we had a problem with the new ARM7s when iPhone5 came out and they solved it in about 30 minutes.

I use TestFlight for user beta testing management so I've tried TestFlight Live SDK in the summer just to see if it was a solution to have all integrated in one service, but we had a really bad experience with it. I had 2 updates rejected in the App Store Approval for the first time (Weddar was launched in April 2011) and we lost about a month trying to catch the bug. When LIVE beta testing, no user would complain about any problem, we "solved" it by removing the TF SDK. Never quite understood what was the problem. We contacted TestFlight team and never had contact back. (Another big big detail is that TF SDK added about 800Kb to our app.)

So, even if I still use TestFLight for beta testing, if you're looking for a great and lightweight crash reporting SDK, I definitely say that you should use Crashlytics.

Hope this helps.

Catarino
  • 233
  • 2
  • 8
4

I'd say go with TestFlight (Live)

In my experience the TestFlight SDK won't crash/slow-down your device and has very versatile crash reporting - allowing you to debug reported errors fairly accurately.

TestFlight also doubles-up as a feedback package for when you're testing in-development.

It's also a pretty light SDK.

To be more specific (in answering your list of questions):

  1. TestFlight allows you to scalp for user 'checkpoints' and has its own version of NSLog that allows you to dynamically log events at runtime.
  2. Your app wouldn't slow down as network requests are handled off the main thread.
  3. I don't understand why a device would get overloaded by using either of the SDKs you've mentioned.
  4. QuincyKit reports seem reasonably precise, however you need to make up your own mind about the precision you need - you can find QuincyKit docs here.
Sarreph
  • 1,986
  • 2
  • 19
  • 41
  • 2
    QuincyKit gives you complete standard format crash reports. So it has the same information details as any crash report from Apple, including all running threads and in case of an exception the Last Exception Backtrace. This is important, since exceptions are re-thrown by the runtime and otherwise you wouldn't know where the actual exception occurred. Since it is all open source, you need to set it up on your own server and also setup the symbolication process yourself. You can also add your own logging data and attach that to the crashes, e.g. by using CocoaLumberjack. – Kerni Dec 27 '12 at 12:36