0

It seems that SQL Alchemy can connect to MySQL table running on Google Cloud SQL. However, I spent time to look for wrapper of Google Cloud Bigtable, a NoSQL database, and could not find anything enough mature.

Just wondering how to manage Google Cloud Bigtable from SQL Alchemy.

Misha Brukman
  • 12,938
  • 4
  • 61
  • 78
tensor
  • 3,088
  • 8
  • 37
  • 71
  • As I mentioned in [my answer below](http://stackoverflow.com/a/40774789/3618671), SQLAlchemy simply cannot support NoSQL storage systems. They have been asked to support HBase and Bigtable, but they only support SQL storage systems. Can you clarify what you would like to do with Bigtable from Python, and whether the [Google Cloud Bigtable Python library](https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigtable) ([docs](https://googlecloudplatform.github.io/google-cloud-python/stable/bigtable-usage.html)) is sufficient for your use case? – Misha Brukman Dec 18 '16 at 00:02

2 Answers2

1

There is some Python API to connect to Big Table Cloud: https://googlecloudplatform.github.io/google-cloud-python/stable/

The google-cloud library is pip install-able:
$ pip install google-cloud

Cloud Datastore
from google.cloud import datastore

client = datastore.Client()
key = client.key('Person')
entity = datastore.Entity(key=key)
entity['name'] = 'Your name'
entity['age'] = 25
client.put(entity)

However, this is still not integrated through SQL Alchemy, this is not clear that Schema can be easily integrated.

tensor
  • 3,088
  • 8
  • 37
  • 71
  • Note that your code sample uses Google Cloud Datastore. For Bigtable, please see [these docs and sample code](https://googlecloudplatform.github.io/google-cloud-python/stable/bigtable-usage.html). – Misha Brukman Dec 18 '16 at 00:00
0

This is not possible, because SQLAlchemy can only manage SQL-based RDBMS-type systems, while Bigtable (and HBase) are NoSQL, non-relational systems.

Here's my detailed response on a feature request that was filed for the Google Cloud Python library project which has more context and alternative suggestions:

The integration between SQLAlchemy and Google Cloud Bigtable would have to be done in SQLAlchemy. I was going to file a bug on SQLAlchemy on your behalf, but looks like you've already filed a feature request and it was closed as wontfix:

unfortunately Google bigtable is non-relational and non-SQL, SQLAlchemy does not have support for key/value stores.

and a previous email thread on the sqlalchemy@ list about adding support for NoSQL databases like HBase (which is very similar to Bigtable) ended up without any answers.

Thus, I am afraid we won't be able to help you use SQLAlchemy together with Bigtable.

That said, as an alternative, consider using Apache Hue, which works with Apache HBase, and can be made to work similarly with Bigtable. We don't have a simple howto for how to connect Apache Hue to Cloud Bigtable yet, but I imagine it can be done as follows:

  1. Apache Hue -> (a: Thrift API) -> Apache HBase Thrift proxy -> (b: gRPC API) -> Google Cloud Bigtable

    The first connection (a) should work out-of-the-box for Hue and HBase. The second connection can use the Google Cloud Bigtable Java client for HBase. This is not as complicated as it looks, although there are several parts to connect together to make it all work.

  2. Apache Hue -> (gRPC API) -> Google Cloud Bigtable

    This could be done using the Google Cloud Bigtable Java client for HBase, but it requires Apache Hue to use the HBase 1.x API (which I believe is not yet the case, I believe it's using 0.9x API and/or Thrift), so I would recommend following option (1) above for now instead.

Hope this is helpful.

Misha Brukman
  • 12,938
  • 4
  • 61
  • 78