;t/Uc@sddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZd efd YZdS( iNi(tIDENTITYt GLOBAL_LOCKER(tConnectionPool(tFieldtQueryt Expression(tSQLALL(tuse_common_filtersi(t NoSQLAdaptert IMAPAdapterc BseZdZd.Zi ed6ed6ejd6ejd6ed6ed6e d6ed 6ed 6ed 6Z d Z e j d Zdd/deiied/dZd/edZdZdZd/edZedZddZdZdZdZdZd/dZdZ dZ!dZ"d Z#d/d!Z$d"Z%d#Z&ed$Z'd%Z(d&Z)d'Z*d(Z+d/d)Z,d*Z-d+Z.d,Z/d-Z0RS(0s IMAP server adapter This class is intended as an interface with email IMAP servers to perform simple queries in the web2py DAL query syntax, so email read, search and other related IMAP mail services (as those implemented by brands like Google(r), and Yahoo!(r) can be managed from web2py applications. The code uses examples by Yuji Tomita on this post: http://yuji.wordpress.com/2011/06/22/python-imaplib-imap-example-with-gmail/#comment-1137 and is based in docs for Python imaplib, python email and email IETF's (i.e. RFC2060 and RFC3501) This adapter was tested with a small set of operations with Gmail(r). Other services requests could raise command syntax and response data issues. It creates its table and field names "statically", meaning that the developer should leave the table and field definitions to the DAL instance by calling the adapter's .define_tables() method. The tables are defined with the IMAP server mailbox list information. .define_tables() returns a dictionary mapping dal tablenames to the server mailbox names with the following structure: {: str } Here is a list of supported fields: =========== ============== =========== Field Type Description =========== ============== =========== uid string answered boolean Flag created date content list:string A list of dict text or html parts to string cc string bcc string size integer the amount of octets of the message* deleted boolean Flag draft boolean Flag flagged boolean Flag sender string recent boolean Flag seen boolean Flag subject string mime string The mime header declaration email string The complete RFC822 message (*) attachments list Each non text part as dict encoding string The main detected encoding =========== ============== =========== (*) At the application side it is measured as the length of the RFC822 message string WARNING: As row id's are mapped to email sequence numbers, make sure your imap client web2py app does not delete messages during select or update actions, to prevent updating or deleting different messages. Sequence numbers change whenever the mailbox is updated. To avoid this sequence numbers issues, it is recommended the use of uid fields in query references (although the update and delete in separate actions rule still applies). :: # This is the code recommended to start imap support # at the app's model: imapdb = DAL("imap://user:password@server:port", pool_size=1) # port 993 for ssl imapdb.define_tables() Here is an (incomplete) list of possible imap commands:: # Count today's unseen messages # smaller than 6000 octets from the # inbox mailbox q = imapdb.INBOX.seen == False q &= imapdb.INBOX.created == datetime.date.today() q &= imapdb.INBOX.size < 6000 unread = imapdb(q).count() # Fetch last query messages rows = imapdb(q).select() # it is also possible to filter query select results with limitby and # sequences of mailbox fields set.select(, limitby=(, )) # Mark last query messages as seen messages = [row.uid for row in rows] seen = imapdb(imapdb.INBOX.uid.belongs(messages)).update(seen=True) # Delete messages in the imap database that have mails from mr. Gumby deleted = 0 for mailbox in imapdb.tables deleted += imapdb(imapdb[mailbox].sender.contains("gumby")).delete() # It is possible also to mark messages for deletion instead of ereasing them # directly with set.update(deleted=True) # This object give access # to the adapter auto mailbox # mapped names (which native # mailbox has what table name) imapdb.mailboxes # tablename, server native name pairs # To retrieve a table native mailbox name use: imapdb..mailbox ### New features v2.4.1: # Declare mailboxes statically with tablename, name pairs # This avoids the extra server names retrieval imapdb.define_tables({"inbox": "INBOX"}) # Selects without content/attachments/email columns will only # fetch header and flags imapdb(q).select(imapdb.INBOX.sender, imapdb.INBOX.subject) timaplibtstringttexttdatetdatetimetidtbooleantintegertbiginttblobs list:stringtimapsP^(?P[^:]+)(\:(?P[^@]*))?@(?P[^\:@]+)(\:(?P[0-9]+))?$isUTF-8c  sD|_|_| r(j|n|_|_|_| _|_|_|_ d_ d_ t j_d_|jdd}idd6dd6dd6d d 6d d 6d d6_idd6dd6dd6dd6dd6dd6dd6dd6dd 6dd6dd6dd6d d 6d d 6d!d"6d d6dd#6dd$6dd%6_d&|d'<jj|} | jd(} | jd)} | jd*}t| jd+}t|d,krtn|jd*|d+|d)| d(| |fd-}jj_|_| r@jndS(.Ns://is\Deletedtdeleteds\Drafttdrafts\Flaggedtflaggeds\Recenttrecents\Seentseens \AnsweredtansweredtMESSAGERtDATEtcreatedtUIDtuidtFROMtsendertTOttotCCtcctBCCtbcctTEXTtcontenttSIZEtsizetSUBJECTtsubjecttmimetemailt attachmentstt_lastsqltusertpasswordthosttporticstrjj_njj_j|d|d}|j|d|d}d|_d|_|S(NR5R6R3R4cSstS(N(tTrue(((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyts(tdrivert IMAP4_SSLtimap4tIMAP4tlogintNonet mailbox_namestcursor(t driver_argst connectiontdata(tover_ssltself(sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt connectors  (tdbturit find_drivert pool_sizetfoldertdb_codect_after_connectiontcredential_decoderRAt adapter_argsR>t mailbox_sizet static_namestsystgetfilesystemencodingtcharsetR;tsplittflagst search_fieldst REGEX_URItmatchtgrouptinttFalseR7tupdatet define_tablesRFt reconnect(RERGRHRJRKRLRNRAROt do_connecttafter_connectiontmR3R4R5R6RF((RDREsP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt__init__sZ                " cCsjt|dddk rdS|dkr4|j}n|jsd||_|o[|jj|_ntj}|j}xt r[t j ||krg||RFRJRBR@RtPOOLSRHR7Rtacquiretpoptreleasetcheck_active_connectiontlisttafter_connection_hook(REtfR@ReRHtresultRC((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyR_s8               cCsd}t|jjts(|jnyI|jj|jj|}t|dd}|dkrpd}nWnNtt t t fk rt j d}|jjjddt|nX|S(Niis!Error retrieving the last mailboxs sequence number. %s(R>t isinstanceRBR?tdictt get_mailboxestselectR[t IndexErrort ValueErrort TypeErrortKeyErrorRRtexc_infoRGtloggertdebugtstr(REt tablenamet last_messageRmte((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytget_last_message$s    cCst|jjts"|jn|j|}|jjddd\}}|djj }t |dkr{dS|d|dfSdS(Ntsearchs(ALL)ii( RnRBR?RoRpR}RR>tstripRUtlen(RERzR{RmRCtuid_list((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytget_uid_bounds6s c Cs|dkrtj}ndddddddddd d d d g }t|tr_yd |kr~|jd \}}n d|}}|jj}t|d}|j|dj } t|d} t t|djd} tj|| | | d| d| d|SWqt t t fk r[} |jjjd|| fdSXnEt|tjtjfr|rd} nd} ||j| SdSdS(NtJANtFEBtMARtAPRtMAYtJUNtJULtAUGtSEPtOCTtNOVtDECt,iiiit:s!Could not parse date text: %s. %ss%a, %d %b %Y %H:%M:%S %zs%d-%b-%Y(R>Rt timedeltaRnt basestringRURR[tindextuppertmapRstAttributeErrorRrRGRwterrorR tstrftime(RER taddtimftmonthstdaynamet datestringt date_listtyeartmonthtdaythmsR|t date_format((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt convert_dateCs4   ! cCsKddlm}||d\}}|rG|j|jd}n|S(Ni(t decode_headerisutf-8(t email.headerRtdecodetencode(RltrRR tencoding((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytheader_representgs treplacecCs}|dkrd}n[t|trZ|dkrEt|d|}qpt|||}ntdt||jdS(s! convert text for mail to unicodeR1sutf-8sUnsupported mail text type %sN(R>RnRytunicodet ExceptionttypeR(RER RTterrors((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt encode_textos   cCs|j}|S(N(tget_content_charset(REtmessageRT((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt get_charset}s c Cs6|jr%|j|j_|jjS|jj}t|j_t}d}x|dD]}|d}|j}d|jkr]|jd}g|D]$}t |jdkr|^q}|dj}t j ddt j ddt j d d |}|j |||jj|thasattrRRnRRzR(REtquerytnofieldRztattr((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytget_query_mailboxs  cCs0|jj|d|jjkr(tStSdS(N(RWtgetR>RVtvaluesR7R\(REtflag((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytis_flags$cCs0|r||_n d|_t|jjts=|jn|jjj}x|D]}|jj d|t ddt t dddt t dddt t ddt t d dt t d dt t d dt t d d dt t ddt t ddt t dddt dt t dddt dt t ddt t ddt ddt ddt ddt dddt t dd|jj||j|_ |j |j|j_|j|j_|j|j_|j|j_|j|j_qVW|jj|j_|jjS(s| Auto create common IMAP fileds This function creates fields definitions "statically" meaning that custom fields as in other adapters should not be supported and definitions handled on a service/mode basis (local syntax for Gmail(r), Ymail(r) Returns a dictionary with tablename, server native mailbox name pairs. s%sRtwritableRRR)R R#R%R'R!R+RR-R.R/treadableR0RRRRRRRRN(RQR>RnRBR?RoRpRRGt define_tableRR\RRR#t representR%R'R!R-R(RER?tnamestname((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyR^s>         ^cOsdS(N((REtargstkwargs((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt create_tablesc, Cs2 t|r-|j||j|g}nddl}d}t}t|tr|j|}|j j j |d}|dkrt d|q|j j |t\}} |dkrtd| nt| d|_dt|j} |j jdd| } | ddkr|j d d} | d dj} | j| dk r| t| dt| d !} ntg|D]}|jdEk^qrd}nd}t| dkrx| D]}|j jd||\}}|dkrit|ddjdd6t|d6|j|dd d6|dd d6}|dj|d<|jj|d |d<|j|qtd|qWqqtd| d n-t|t t!frt"n t#di}i}t|d kr7t|dt$r7t}n!t|dkrRt}nt%}|rg|j&j'D]}d||f^qn}n&g|D]}d||jf^q}x|D]}|||6|'d?6|$j1d@6|$dAdB6|j|(qDd=|%krDd:||krD|(j0i|j2|&|j)|$d=6|j|(qDqGndC||kr|$dk r|tt|$7}qqqW||d:|<||d9|<||dC|<|j|qWxF|D]>}t})x|D]}*|)j||*qW|j|)qW|}|j dD|j3}+|+|||S(Fs> Searches and Fetches records and return web2py rows iNsMailbox name not found: %stOKsIMAP error: %sis(%s)R~tlimitbyiR)R+R0R/s(RFC822 FLAGS)s(RFC822.HEADER FLAGS)tfetchRRt raw_messaget multipartRVs"IMAP error retrieving the body: %ssIMAP search error: %ssUnexpected query types%s.%ss%s.ids %s.createdtDates%s.uids %s.sendertFroms%s.totTos%s.cctCcR1s%s.bcctBccs %s.deleteds\Deleteds%s.drafts\Drafts %s.flaggeds\Flaggeds %s.recents\Recents%s.seens\Seens %s.subjecttSubjects %s.answereds \Answereds%s.mimes %s.encodings%s.emails%s.attachmentss %s.contentRR.R tpayloadtfilenameRsContent-Dispositiont dispositions%s.sizet processor(scontentssizes attachmentssemail(4Rt common_filterRR/R>RjRnRt get_tableRBR?RRsRqR7RR[RPRyRRRUtreversetanyRRtlongtmessage_from_stringt is_multipartR9t ParseFlagsRRRtNotImplementedErrorRtRR\RWRRoRRtget_content_typetwalktget_content_maintypet get_payloadt get_filenameR]RRtparse(,RERtfieldst attributesR/Rzt fetch_resultsRRmtselectedt search_queryt search_resultRt messages_settfieldt imap_fieldsRttypRCtfrt imapqry_dicttimapfields_dictt allfieldstcolnamestkt imapqry_listt imapqry_arrayR0R)R+tnt item_dictRRTRVRtparttmaintypeRRRtimapqry_array_itemt fieldnameR((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyRqs !      $"   "   %  /&                  c sRfd}|j}td|D}|jdpItjj}|j}t|dkrB|jdd}|jdg} |jdg} djgd$D](} |j| t rd| j ^q} |sddl m |jdd} |jdd}}|jdd|d<|jdd|d<|j |dt|d<| r||j| n|r|j|nxbd%D]Z}|j|d}t|tr|||tset_typet set_payloadtattach(RtobjRRR.(tMessage(sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyt add_payloads"      css$|]\}}|j|fVqdS(N(R(t.0Rtv((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pys sRiR/R0R)t RRRRRRs\%si(RR.RR!R1tfromR-RR R#R%R't;RR Rs\d+sIMAP message append failed: %ss$IMAP empty insert is not implemented(sansweredsdeletedsdraftsflaggedsrecentsseen(stosccsbcc(%RRoRRtnowt timetupleRR>tjoinR\t capitalizet email.messageRRR7RRRnRRRt startswithRt as_stringRBRR[RtfindallRyRGRRqRRRR(REttableRRRtdt date_timet struct_timeRR0R)RRVR.RTRtvaluetitctaRmRCR((RsP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytinserts\   5    !     ! "+cCs2t}d}t|r3|j||g}ng}g}|rx|D]}|d} | j} |d} |j| rL|j| } | dk r| dkr| r|j| q|j| qqLqLW|jj |jj |\} }d|}|jj d|\} }g|dj D]$}|j jr(|j ^q(}x|D]|}d} t|dkr|j|dddj|fnt|dkrY|j|dddj|fqYqYWnxO|D]G}|jj|\} }| dkr|d7}qtd |qW|S( Niis\Recents(%s)s+FLAGSRs-FLAGSRsIMAP storing error: %s(RjRRRRRWR>RRBRqR?R~RURtisdigitRR tstoreR(RERzRRtcommandstrowcounttmarktunmarkRRRR*RRmRCt string_queryt store_listtnumbertcommand((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyR]sF         $ &-   c Csd}|j|}|r|dk rt|rK|j||g}n|jj|jj|\}}d|}|jjd|\}}g|djD]$}|j j r|j ^q} t | }n|S(Nis(%s)( RR>RRRBRqR?R~RURR/R( RERtdistincttcounterRzRmRCR5RR6((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytcount<s " ;c Cs5d}|r1t|r0|j||g}n|jj|jj|\}}d|}|jjd|\}}g|djD]$}|jj r|j^q}xU|D]M} |jj | dd\}}|dkr|d7}qt d|qW|dkr1|jj \}}q1n|S(Nis(%s)s+FLAGSs (\Deleted)RisIMAP store error: %s( RRRBRqR?R~R>RURR/R0Rtexpunge( RERzRR:RmRCR5RR6R7((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytdeleteIs  " ;    cCsd}|j|j}|dkrog|D]$}t|jr)t|^q)}ddj|j}ne|dkrg|D]$}t|jrt|^q}ddj|j}n td|S(NRs%sRRsUID %ssOperation not supported(R>RWRRyR/R RR(RERtsecondRmRtvalR((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytBELONGS\s 1 1 cCsd}|j|j}|d kr>d||j|f}ne|jd krrd|jj|j|f}n1|jdkrd |j|}n td |S( NR R"R,R(s%s "%s"R%R'R.sHEADER Content-Type "%s"sOperation not supported(sFROMsTOsSUBJECTsTEXT(sccsbcc(R>RWRtexpandRR(RERR>tcase_sensitiveRmR((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytCONTAINSls % c Csd}|j|j}|dkrZ|j|j}dt|j|d|f}n.|dkr)y|j|j\}}Wn>tk rt j d}|j j j dt|dSXyt|j|d} Wn3ttfk rt j d}tdnXd| |f}n_|d krZd |j|d tjd}n.|d kr|d |j|}n td|S(NRs%d:%diRsError requesting uid bounds: %sR1s)Operation not supported (non integer UID)s UID %s:%sRsSINCE %sRR*s LARGER %ssOperation not supported(R>RWRR}RzR[RARRtRRRvRGRwRxRyRsRRRR( RERR>RmRR{tpedestalt thresholdR|t lower_limit((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytGT|s0 &   %  c Cs d}|j|j}|dkrP|j|j}d|j||f}n|dkry|j|j\}}Wn>tk rtj d}|j j j dt |dSX|j|} d| |f}n.|dkrd |j|}n td |S( NRs%s:%sRisError requesting uid bounds: %sR1s UID %s:%sRsSINCE %ssOperation not supported(R>RWRR}RzRARRtRRRvRGRwRxRyRR( RERR>RmRR{RDRER|RF((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytGEs$     c Cskd}|j|j}|dkrHddt|j|df}n|dkry|j|j\}}Wn>tk rtj d}|j j j dt |dSXyt|j|d}Wn3ttfk rtj d}tdnXd||f}nP|d kr9d |j|}n.|d kr[d |j|}n td |S(NRs%s:%siRsError requesting uid bounds: %sR1s)Operation not supported (non integer UID)s UID %s:%sRs BEFORE %sR*s SMALLER %ssOperation not supported(R>RWRR[RARRzRtRRRvRGRwRxRyRsRR( RERR>RmRRDRER|t upper_limit((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytLTs. &     c Csd}|j|j}|dkr>dd|j|f}n|dkry|j|j\}}Wn>tk rtjd}|j j j dt |dSXt |j|}d||f}n=|dkrd |j|d tjd}n td |S( NRs%s:%siRsError requesting uid bounds: %sR1s UID %s:%sRs BEFORE %sRsOperation not supported(R>RWRRARRzRtRRRvRGRwRxRyR[RRRR( RERR>RmRRDRER|RI((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytLEs"    % cCst|dkr=t|tr=|jdkr=|j|dSn|j|j||}|jddj}|S(NRisNOT NOTR1( R>RnRRRHtNOTtEQRR(RERR>Rm((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytNEs cCs|j|j}d}|dk r|dkrDd|j|}q|dkrfd|j|}q|dkrd|j|}q||jjkr|rd|jd}qd|jd}qtd n td |S( NRs%sRsUID %sRsON %sisNOT %ssOperation not supported( RWRR>RARRVRRR(RERR>RRm((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyRMs      cCs&d|j||j|f}|S(Ns%s %s(RA(RERR>Rm((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pytANDs"cCs6d|j||j|f}d|jddS(NsOR %s %ss%ssOR ORtOR(RAR(RERR>Rm((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyRPs"cCsd|j|}|S(NsNOT %s(RA(RERRm((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyRLs(simaplibN(1t__name__t __module__t__doc__tdriversRyRR RtboolR[ttypestdbengineRtcompileRXR>RR7RcR_R}RR\Rt staticmethodRRRRpRRR^RRqR.R]R;R=R@RCRGRHRJRKRNRMRORPRL(((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyR sf    J-  $      ;  I *          (RRRRt_globalsRRRBRtobjectsRRRthelpers.classesRthelpers.methodsRtbaseRR (((sP/var/www/c4bv.valis/web2py-ge-R-2.10.3/gluon/packages/dal/pydal/adapters/imap.pyts