Stack Overflow
Stack Overflow

Questions tagged [meltano]

Meltano integrates best-in-class open source components and enables teams to collaborate on data projects and pipelines more efficiently and with higher confidence. The focus is bringing these qualities to the first step in any data journey: integration and transformation, aka EL(T): Extract, Load, Transform.

22 questions
0
votes
0 answers

Meltano tap-mysql throwing `No RECORD message received` error

I'm building an ETL pipeline using Meltano that extracts data from a db using tap-mysql extractor and dumps the data into an S3 bucket in AWS. Here's what I'm doing - testing locally with this meltano.yml file: version: 1 default_environment:…
b-onmo
  • 13
  • 3
0
votes
1 answer

Meltano ETL not deleting rows in destination PostgreSQL database after deleting from source PostgreSQL database

I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (TransferWise variant). However, I have noticed that when…
vuonglv
  • 31
  • 2
0
votes
1 answer

Where files from tap are kept in Meltano

I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet. I want to extract data from tap-marketo from data A to date B in the past. I saw that we can only define start_date and max_export_days. I have tried to…
marz
  • 83
  • 5
0
votes
1 answer

Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit

I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql". how could I fix that? psycopg2.errors.ConfigurationLimitExceeded: temporary file…
Dev024
  • 13
  • 3
0
votes
1 answer

Cron job with an alias

While I work on setting up Meltano, I'd like to create a simple cron job to schedule my Pipelinewise tap/target on my Ubuntu server. Transferwise references using the following crontab: */5 * * * * pipelinewise run_tap --tap microserv_1 --target…
motherbuzzard
  • 13
  • 4
0
votes
1 answer

Meltano Kafka Monk.io integration

I'm playing a bit with Monk.io and Kafka - Meltano integration. So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano. I'm using https://github.com/lensesio/fast-data-dev for Kafka env. What would be the best…
shonster88
  • 84
  • 9
1
2