0

In short: What are the limits of both cloud projects and service accounts per project? How can they be increased? Is the architecture a good idea at all?

I am developing an IoT application with tens of thousands of planned devices in the field, having a few devices per customer and hundreds of customers. Every device will continuously (24/7) stream measurement data directly to BigQuery with one dataset (or table) per device at sample rates of at least 100Hz.

Of course, every device needs to be authenticated and authorized to gain tightly restricted access to its cloud dataset. As stated in the Auth Guide API keys are not very secure. Therefore, the most appropriate solutions seems to have one service account per customer with one account key per device (as suggested in this GCP article). However, the FAQs of Cloud IAM state that the number of service accounts is limited to 100 per project.

  • This limit could be reached quickly. If so, how easily/costly is it to increase this limit towards thousands or tens of thousands of service accounts per project?
  • In such a scenario also the number of needed projects could easily grow to hundreds or thousands. Would this be feasible?
  • Is this overall concept practical or are there better approaches within GCP?
ROMANIA_engineer
  • 54,432
  • 29
  • 203
  • 199
a13x6au3r
  • 1
  • 1
  • Have you looked into [Firebase](https://firebase.google.com/docs/analytics/)? That may be easier to set up, and you can [import your Firebase analytics data into BigQuery](https://cloud.google.com/solutions/mobile/mobile-firebase-analytics-big-query). Note that there is a delay before Firebase data gets imported, however. – Elliott Brossard Jun 08 '17 at 16:12
  • Have you looked into [`Google Cloud IoT`](https://cloud.google.com/solutions/iot/)? It supports ingesting data into `Cloud PubSub` and from there into any of the Data analytics services like `BigQuery` – Tuxdude Jun 08 '17 at 18:43
  • @Elliott Brossard I just looked into Firebase. It looks very interesting in general but concerning its options for authentication it seems like either a federated identity provider (Google, Facebook, Twitter, ...) is needed or at least an email address or a phone number. This is fine for user-based applications. But authenticating every single device with its own email address or phone number doesn't feel very practical to me. Also forgot to mention that the devices have no input capabilities at all. – a13x6au3r Jun 09 '17 at 08:04
  • @Tuxdude Yes, I have looked into Google Cloud IoT and [this GCP article](https://cloud.google.com/solutions/iot-overview) before and got very excited about its announced features. I already requested the private beta last week. Still pending though... Does anyone know how devices are going to be authenticated? – a13x6au3r Jun 09 '17 at 08:09

0 Answers0