Kafka consumer fetches new messages only after restarting the consumer

Question

I'm facing an issue with my kafka consumer job written in scala. when we start the consumer, it fetches all messages available in the broker from the last consumed offset, process those JSON messages and writes them to Hive table. After writing, it has to again fetch new messages, but that's not happening instead we have to forcefully stop the job and restart it to fetch new messages.

Here are the steps we perform in the Kafka consumer job.

1. set the params/properties
2. start the stream and messages based on the offsets
3. Parse the JSON messages for few house keeping columns
4. check for the duplicates in current stream comparing with last 5 days of hive partitions
5. write to Hive partitioned table
6. create trigger files for the successor job to parse the JSON
7. Repeat above process for new messages in stream

Here are some of the properties we set

kafka_props.put("bootstrap.servers", BROKER_SERVERS)
kafka_props.put("key.deserializer", classOf[StringDeserializer])
kafka_props.put("value.deserializer", classOf[StringDeserializer])
kafka_props.put("group.id", GROUP_ID)
kafka_props.put("auto.offset.reset", offset)
kafka_props.put("security.protocol", "SSL")
kafka_props.put("ssl.truststore.location", TRUSTSTORE_LOCATION)
kafka_props.put("ssl.truststore.password", SSL_TRUST_PWD)
kafka_props.put("ssl.keystore.location", KEYSTORE_LOCATION)
kafka_props.put("ssl.keystore.password", SSL_KEY_PWD)
kafka_props.put("enable.auto.commit", (false: java.lang.Boolean))
kafka_props.put("session.timeout.ms", (30000: java.lang.Integer))
kafka_props.put("heartbeat.interval.ms", (3000: java.lang.Integer))

we tried changing parameters but not much help. Not sure what is being missed and need help to fix it.