How to get UIDNEXT in python procmail pipe script?

Question

I have this python script for inserting raw email to db. Do not ask me why I am inserting raw mail to database.

import sys
from DB import *
import email

full_msg = sys.stdin.readlines()
j =  ''.join(full_msg)
msg = email.message_from_string(j)

sql = '''INSERT INTO Email(Id, Message) VALUES (NULL, %s)'''
db = DB()
db.query(sql, (msg, ))

It would be great if I can get uid of that message, so if I for example delete message from db I can also delete message my uid on imap server.

I do not want to login to imap server and then delete message by uid because I do not know user password since it is encrypted.

I was thinking to get for example msg['Message-Id'] and then to grep files in user maildir for that Message-Id and delete actual file but that sound totally wrong to me.

I know in python you have something like UIDNEXT in imaplib but that is under assumption I am logged in which I'm not.

UPDATE:

With this I can fetch next uid but I have to login. How to get UIDNEXT without login? By the way I use postfix/dovecot with mysql.

import getpass, sys
from imapclient import IMAPClient

try:
    hostname, username = sys.argv[1:]
except ValueError:
    print 'usage %s hostname username' % sys.argv[0]

c = IMAPClient(hostname, ssl=True)

try:
    c.login(username, getpass.getpass())
except c.Error, e:
    print "Could not login in:", e
    sys.exit(1)

else:
    select_dict = c.select_folder('INBOX', readonly=True)
    for k, v in select_dict.items():
        if k == 'UIDNEXT':
            print '%s: %r' % (k,v)
    c.logout()

NEW UPDATE

Sample of dovecot-uidlist

16762 W105493 S104093 :1417408077.2609_1.zumance
16763 S18340 W18608 :1417429204.3464_1.zumance

Code for geeting last line of dovecot-uidlist uid:

l = open("dovecot-uidlist").readlines()
print l[-1].split(" ")[0]

This is completed script for mail pipe:

import sys
import email
import re
from DB import *
full_msg = sys.stdin.readlines()
j =  ''.join(full_msg)

msg = email.message_from_string(j)
match = re.search(r'[\w\.-]+@[\w\.-]+', msg['to'])

address = match.group(0)
address = address.split("@")

with open("/var/vmail/"+address[1]+"/"+address[0]+"/dovecot-uidlist", 'r') as f:
  first_line = f.readline()
  nextuid = first_line.split(" ")
  nextuid = re.findall(r'\d+', nextuid[2])

sql = '''INSERT INTO Email(Id, Message, Uid, Domain, Mbox) VALUES (NULL, %s, %s, %s, %s)'''
db = DB()
db.query(sql, (msg, nextuid[0], address[1], address[0],  ))

Blog post with files at https://pregmatch.org/read/python-procmail

Answer 1

Dovecot maintains the mapping between UID and filename in the file dovecot-uidlist. The file contains first a header line and then one line per message.

The header line looks like this:

1 1173189136 20221

The first digit is the version, the second the IMAP UIDVALIDITY, and the last is the next UID that will be used.

After that, each message has its own line looking like this:

20220 1035478339.27041_118.example.org:2,S

The first word is the UID, the next is the filename.

There's more information at the dovecot wiki.

Answer 2

Since parsing dovecot-uidlist will not work because list will not be updated until you check email with your email client, I decide to go with other solution. That solution is dovecot pre-auth mechanism. In my python procmail pipe script I decided to do somethig like this:

import subprocess
p = subprocess.Popen( "/usr/libexec/dovecot/imap -u "+user, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
p.stdin.write("b select INBOX\n")
p.stdin.write("e logout\n")
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata

print stderrdata

stdoutdata gives me my output that look like this:

* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in as nikola@pregmatch.org
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk $Forwarded)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk $Forwarded \*)] Flags permitted.
* 5574 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1412448500] UIDs valid
* OK [UIDNEXT 16875] Predicted next UID
* OK [HIGHESTMODSEQ 3051] Highest
b OK [READ-WRITE] Select completed (0.009 secs).
* BYE Logging out
e OK Logout completed.

Now all i have to do is parse this part of that output:

* OK [UIDVALIDITY 1412448500] UIDs valid
* OK [UIDNEXT 16875] Predicted next UID

This pre-auth solved my problem. I will post complete solution with parsing part later today (and on my blog).

SOLUTION:

import subprocess
pSub = subprocess.Popen( "/usr/libexec/dovecot/imap -u "+username+"@"+domain_parsed, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
pSub.stdin.write("b select INBOX\n")
pSub.stdin.write("e logout\n")
(stdoutdata, stderrdata) = pSub.communicate()
dovecotStream = open("/var/www/domain.com/scripts/server/dovecot","w")
dovecotStream.write(stdoutdata)
dovecotStream.close()


nextuidNo = []
with open("/var/www/domain.com/scripts/server/dovecot") as dovecotFile:
    dovecotFilelines = dovecotFile.read()
for dovecotFileline in dovecotFilelines.split('\n'):
    matchCheck = re.findall(r'\[UIDNEXT.+\]', dovecotFileline)
        if len(matchCheck):
            nextuidNo = re.findall(r'\d+', matchCheck[0])

print nextuidNo #this is list

Complete blog post at: https://pregmatch.org/read/dovecot-preauth-python