Questions tagged [kiba-etl]
36 questions
0
votes
1 answer
Recommended way to achieve `rescue-ensure` kind of functionality with Kiba?
We have a Kiba pipeline where we need to do some task after the job has ended, no matter if there were errors or not (the whole pipeline doesn't fail, we just have couple of validation errors or similar).
This is what the documentation…
Viktor
- 2,982
- 27
- 32
0
votes
1 answer
How to implement local DB + external API sources in Kiba
I'm trying to implement the following ETL job in Kiba, in the context of a Rails app:
For a given local database record
Search its name using a remote application API (Evernote)
For each record found with the API, get the GUIDs of all the tags used…
Spone
- 1,324
- 1
- 9
- 20
0
votes
1 answer
How to transform nested JSON-payloads with Kiba-ETL?
I want to transform nested JSON-payloads into relational tables with Kiba-ETL. Here's a simplified pseudo-JSON-payload:
{
"bookings": [
{
"bookingNumber": "1111",
"name": "Booking 1111",
"services": [
{
…
wuarmin
- 3,274
- 3
- 18
- 31
0
votes
1 answer
Is there a way to write to Kiba CSV destination line by line or in batches instead of all at once?
Kiba is really cool!
I'm trying to set up a ETL process in my Rails app where I'll dump a large amount of data from my SQL DB to a CSV file. If I were to implement this myself I'd use something like find_each to load say 1000 records at a time and…
user1376350
- 563
- 4
- 4
0
votes
1 answer
Using Kiba: Is it possible to define and run two pipelines in the same file? Using an intermediate destination & a second source
My processing has a "condense" step before needing further processing:
Source: Raw event/analytics logs of various users.
Transform: Insert each row into a hash according to UserID.
Destination / Output: An in-memory hash like:
{
"user1" =>…
AlexHeeton
- 1
- 1
0
votes
1 answer
Tmdb::InvalidApiKeyError (Tmdb::InvalidApiKeyError)
When I am using Kiba ELT, I have followed the tutorials in YouTube as well as the tutorials provided by the owner. Yet, I am getting this error:
bitlasoft@Bitlasoft-TS-22:~/test01$ bundle exec kiba movies.etl
{
"title: Blade Runner" => "title:…
