jump to navigation

Blog moving to new home July 14, 2009

Posted by atmospherian in InterIMAP.
2 comments

http://www.interaxia.net/blog

InterIMAP: Async Update March 11, 2009

Posted by atmospherian in InterIMAP.
Tags:
4 comments

Post as moved here

InterIMAP: Requirements January 20, 2009

Posted by atmospherian in InterIMAP.
1 comment so far

This post will serve as the official requirements documentation for the InterIMAP library. Please comment on this post with your suggestions for further requirements.

Core Requirements

  • Multi-threaded, multi-connection, asynchronous processing
  • Ability to support multiple accounts simultaneously
  • Ability to detect and retrieve only new messages for any or all folders
  • Ability to receive and save all file attachments, and message content
  • Ability to read/write messages in EML format
  • Optimized single process/connection synchronous client for repeated, specific purposes

Storage Requirements

  •  Provide 3 storage mechanisms: Virtual, Physical, Database
  • Ability to store folder and message data in memory temporarily
  • Ability to store folder and message data including attachments in a file on disk that can allow a mailbox to be cached, and to make that data portable.
  • Ability to store folder and message data in a relation database such as SQL Server, so that the data can be easily shared among other applications without having to distribute a local cache file, or stream the data to other services.
  • Ability to use Physical and Database storage mechanisms concurrently. 

API Requirements

  • Provide clean and clear interface to IMAP protocol
  • Provide easy to use Request/Response system
  • Provide clean object model

InterIMAP: Taking a step back January 20, 2009

Posted by atmospherian in InterIMAP.
add a comment

So i’ve been reading Steve McConnell’s Code Complete lately, and it’s got me thinking quite a bit about the new work i’ve been doing on the InterIMAP library. When i started working on the new implementation i had this grand view of a well thought-out, well designed system, and to a certain extent, i feel i accomplished that, but in some areas, i fell short. 

I initially felt that the new system would only ever need to connect to one server at a time, and as a result i wrote all the manager classes as singletons. I have since come to the conclusion that it may be necessary for multiple servers/accounts to be accessed concurrently, so that will be one of the major requirements going forward. Additionally, it may also be important to some people to be able to store the information retrieved from a mailbox in a full fledged RDBMS, as well as a “local cache” type system, so i will also be working that into the new architecture.

My next post will be a summary of what the requirements of the library will be. Please leave comments as to what you would like to have available in the library.

InterIMAP Redesign Notes January 6, 2009

Posted by atmospherian in InterIMAP.
1 comment so far

The time has come for me to continue work on InterIMAP. I have many changes and improvements planned for this library that will improve its usability. In an effort to flesh out my ideas and implementation, i will be putting together different posts for each topic that needs exploration. I welcome all comments and suggestions from those few of you who find their way here. 

To kick things off, here is a partial list of the features i am planning to implement or improve (in order of priority):

  • Better code structure utilizing more standard patterns
  • Completely Asynchronous client API
  • Improved offline cache system based on a ‘mailbox’ file that will contain the configuration information, messages and attachments in a zip format. This will allow an entire mailbox to be portable between different systems.
  • Build a basic graphical client to demonstrate the new asynchronous API and other features
  • Compatibility with Mono so the library can be used on Linux, Mac OS X, and any other supported platforms. 
  • Convert the graphical configuration editor into a component that can be dropped into any application.
  • Build a second configuration component based on WPF that can be easily customized. 

My next post will go into detail about what i plan to do about the code structure and how i intend to refactor, and reorganize everything.

InterIMAP Progress Update #7 May 22, 2008

Posted by atmospherian in InterIMAP.
3 comments

I am very excited to let you guys know that the re-engineering of the message processing architecture is currently stable and working. My new system for processing message content has not only reduced the number of send/receives from the server to a maximum of 3, but has reduced the code of IMAP.cs by roughly 50-60% making it much cleaner and easier to read and understand.

The original way of processing the message content, which was written by the original author of the code, Rohit Joshi, worked well, except for the fact that it choked on many messages from Gmail, which as i understand is one of the main services that this library is being used with (Exchange is the other). Modifying his code to work with gmail proved nearly impossible due to the extreme variations that can occur in body structure. The process involved first getting the BODYSTRUCTURE of the message, parsing that and then getting each individual BODY[x(.x.x)] section. While this is a valid solution, it is very difficult to write something that can parse the BODYSTRUCTURE in its seemingly infinite variations.

The solution is to pull the full content of the message directly using the BODY[] parameter. The result of this command is every section of the message being return in sequence, usually demarcated with a ‘boundary’ value (except in the case of messages with a single text/plain content type). After this boundary, is the header meta data for that section, which can include the content-type, transfer encoding, disposition, etc. making it easy to parse this header data and properly construct IMAPMessageContent objects.

What does this mean for you? it means that there is now a complete, accurate, and efficient process in place for retrieving the message content from both gmail and exchange.

So whats next? Well, during testing, it became clear to me that for large accounts, having all of the message attachments stored in memory could require a significant amount system resources, much more than an email system should need. So with that in mind i am starting to think about ways to store attachments outside of the main cache file, so that when the cache is loaded, the attachments arent loaded until they are requested.

InterIMAP Progress Update #6 May 15, 2008

Posted by atmospherian in InterIMAP.
add a comment

Major updates in the works this week. After some intial testing with GMail, several show-stopping bugs were discovered in the body structure processing code. Modfying the existing code wasn’t possible as the differences were too severe. Instead i opted to re-write the entire message parsing system to be more flexible and to directly interact with the object model, as opposed to generating XML data first and then using a DataSet to read in that XML which was a cheap hack i employed to get to the message data.

I am not quite ready to upload the newest code as i want to perform some more testing with gmail and exchange before i feel it is ready for a wider audience. There is one bug in particular that is show-stopping with gmail that i am trying to address. Once i get that cleared up we should be in good shape.

Stay tuned for more updates as testing on the re-write continues…

InterIMAP Progress Update #5 May 7, 2008

Posted by atmospherian in InterIMAP.
2 comments

New updates coming this week:

Simple and Advanced Searching
I have added a new search system to the library that should make searching within a folder very simple. There are two new objects in the namespace, IMAPSearchQuery and IMAPSearchResult. The SearchQuery object contains many properties that represent the various search options that can be specified.

The actual Search method is defined in IMAPFolder and takes a IMAPSearchQuery instance as a parameter. The search method returns a IMAPSearchResult object that contains a reference to the query that was used, the folder that was searched and a list of IMAPMessage objects that are the results.

To simplify the search process, several static methods have been defined in the IMAPSearchQuery class that take a single argument, and return an IMAPSearchQuery object, which can be plugged directly into the search method like this:

IMAPFolder f = client.Folders[“INBOX”];
IMAPSearchResult sResult = f.Search(IMAPSearchQuery.QuickSearchSubject(“IMAP”));

What the above code basically does is first gets the folder named INBOX, and then calls the Search method on that folder using the one of the quick search static functions to automatically create an IMAPSearchQuery instance to use for the search. The results of the search are stored in sResult and the messages that match the search query can be found in sResult.Messages.

For more advanced searching you can create a IMAPSearchQuery object manually, and specify the terms that you wish to search for. You can specify values for any of the properties that are defined in that class and each will be taken into account during the search.

Currently this system will work only for including all the messages that match the query. There is no support for ‘OR’ or ‘NOT’ style parameters as of yet. The plan i have for this is to add special List<IMAPSearchQuery> properties to the query object, one for ‘NOT’ and one for ‘OR’ where new SearchQueries can be added and processed and included into the main query. I have to experiment with this option and see if it will work well and be intuitive and easy to use.

Local Data Cache
I’ve spent a great deal of time this week working on the local cache system. I’m happy to say that it’s nearly complete. Here is how it currently works:

First, to enable the local cache the developer needs to specify a cache file and cache format in the configuration. This can either be done at runtime, or using the configuration generators.

In the constructor for the IMAPClient class, the configuration is checked to see if a file is defined. if it is, a flag is set, and if autologon is also specified, the system logs in.

Once logged in, the system checks if the specified cache file already exists. If it does, it loads the cache into the object model, and then (if not in offline mode) the system syncs the cache with server (a config option has been added to set the system to auto sync or not). Updates are one-way with the server being the authority. (If the system is running in offline mode, any functions that would change the folder contents or structure are disabled). If the file does not exist, the system then begins the process of building a new cache file. Please be warned that if you are connecting to a large account with many folders and messages this process can definitely take some time (several hours in the worst cases, depending on the location of the server relative to your location, ie. if the server is on a local LAN it will be faster than pulling from the internet).

Once the cache is built, all the data will already be in the object model so no prep work is required to start accessing the messages.

When working in online mode, all of the message and folder copy/move/delete methods will also update the cache when their process completes successfully. This update should not force any special server commands to be run, it should simply serialize the current state of the object model.

InterIMAP Feature Overview May 1, 2008

Posted by atmospherian in InterIMAP.
Tags:
1 comment so far

Note: this post will be updated as the current state of the feature set changes.

Current Features

  • Dynamically loaded message data
  • Caching of message data including attachments to filesystem
  • Supports SSL connections
  • Ability to run in offline mode when using caching system
  • Ability to create and delete folders and copy/move/delete messages
  • Support external configuration files for all options
  • Includes Windows and command-line tools for creating and managing configuration files
  • Provides advanced search system that works in both online and offline modes
  • Ability to save attachments to local filesystem

Planned Features

  • Nested search queries using NOT and OR operators
  • Include simple e-mail client to demonstrate use of the library

InterIMAP Progress Update #4 April 29, 2008

Posted by atmospherian in InterIMAP.
Tags: , ,
3 comments

Here is a break down of this weeks updates to the library:

Configuration Editor (Window app)
I created a simple Windows application that can be used to create and edit configuration files for the InterIMAP library. This should make it much easier to create and manage configuration files.

Local Data Cache Update
So the local data cache has been coming along pretty well. I can now successfully save and load data from the cache and use it to populate the object model upon startup. This is great, however there is much left to be done. For example, there is no synchronization, so if something has changed on the server (folders added/deleted, messages added/deleted), the entire cache would have to be re-generate and saved again. The next step is to design a way to check for changes on the server and changes to the local cache and decide which changes take precedence. Should the local changes be applied first, and then pull the new state of affairs from the server? or should the server be more authoritarian and overwrite the changes made locally when in offline mode?

I implemented a few message management functions on the folder level that allow you to copy, move and delete a message. One problem i ran into though, if the cache is not updated after performing these operations, when i re-ran my test app, messages were loaded that no longer exist on the server and should also not exist in the cache.

Right now the SaveCache method forces every message to be downloaded from the server before everything is saved. But it should behave differently if there is already data from the cache loaded. It shouldnt need to re-download messages that are already stored. Part of the problem i see is that due to the dynamic data loading, most messages by default dont have the content loaded unless the message content has already been accessed elsewhere in the app before the cache is saved.

I will continue to research this and blog more about it as development continues

Bugs Fixed
I Received an email from a developer who was trying to use the SSL functionality to connect to Gmail and got an exception when the system tried to examine the ‘[Gmail]’ folder. The problem was this folder has the \Noselect flag specified which the system wasn’t told to look for and skip that folder if it was found. This has been fixed so the folder processing method will now skip folders that are flagged as not selectable (this applies to EXAMINE as well).

Once that bug was solved, another became readily apparent. After some frustrating debugging and searching i discovered what was causing the ReceiveBuffer method to hang when trying to receive the data from the message body using the SSL connection. I was trying to read from the SslStream object directly instead of the StreamReader object, which both have a Read() method, although slightly different. This has been corrected so you should not have any problems reading data from an SSL connection now.