1

I am querying impala using sqlalchemy which internally uses impyla.

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

engine = create_engine("impala://{host}:{port}/{database}".format(
    host=host,
    port=21051,
    database=schema
))

Session = sessionmaker(bind=engine, autocommit=False)
_session = Session()
result = _session.execute("""SELECT * FROM TABLE1 LIMIT 2""")
data = result.fetchall()
print(data)
_session.close()
Session.close_all()

Now I want to get the query_id than can be used to monitor the progress of this query in Cloudera manager UI.

For example, if I execute the same query using imapal-shell in console.

I get this:

[MYMACHINE:21001] > select * from table1 limit 2;
Query: select * from table1 limit 2
Query submitted at: 2020-08-24 10:28:15 (Coordinator: http://MYMACHINE:25000)
Query progress can be monitored at: http://MYMACHINE:25000/query_plan?query_id=8646e69bf8aa9707:df03d6ff00000000

I need to log this query-id 8646e69bf8aa9707:df03d6ff00000000.

Is there any way I can fetch this using sqlalchemy or any other library?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Sandeep Roy
  • 31
  • 9
  • 2
    Impyla doesn't expose this directly - you could use get_profile to get the full query profile which would include that - https://github.com/cloudera/impyla/blob/master/impala/hiveserver2.py#L606 It seems like something we could expose to impyla but I don't know if that helps with the bigger problem. I don't know the sqlalchemy layer well though – Tim Armstrong Aug 24 '20 at 18:55
  • Thanks @TimArmstrong for your reply. As you have suggested, I can access _query_id_ of impala by `result.cursor.get_profile()`. But this is only working if the query is a success. If it fails for some reason, its giving _NoneType_ error. Can I get the _query_id_ in the case of error also? – Sandeep Roy Aug 25 '20 at 06:02
  • I think that's a good point and think it's a good idea to extend it in future - I created https://github.com/cloudera/impyla/issues/413 to track this. – Tim Armstrong Aug 25 '20 at 17:12

0 Answers0