Recently, eLong has introduced a "five-star companies eLong mail system" and "5 seconds eLong Personal Post" rate mail system, this system uses two sets of eLong, Inc. is the world's largest e-mail with the mail carrier. com co-design and development of the tens of millions of e-mail system: "ComConnect".The author is the principal designer of the system, this article describes the technical characteristics of the system, and database middleware partially achieved.
ELong's "ComConnect" system can run on most unix platforms, including solaris, linux, etc., the use of modular design, the entire system is divided into the following eight modules:
This is the main line of the entire system, through all the other modules in the system.
In large-scale mail system, in order to facilitate the management and deployment of distributed systems, using SQL database needs to store some information such as account information.For example, when users login via POP3, the server will start a program, and SQL database connection and query the legitimacy of the user when the user exits, the program will disconnect the connection with the SQL database
.This large number of users landing, exit, will correspond to the SQL database connection, disconnect operation, reducing the efficiency of the system also increases the load on SQL database.
In ComConnect system, all database operations through "database middleware" modules completed.The module will maintain a connection with the SQL database, and never broken.When a program needs to query the database, it will through the unix domain socket server with the local database is connected middleware, middleware, query requirements will be achieved, and the actual SQL query, and the results returned by unix domain socket to the application.Meanwhile, the inquiry proceeds, the middleware server to the local cache, saved to the memory database BerkeleyDB3 (http://www.sleepycat.com), so the next time the same queries, SQL queries do not have to, because the results have beenis stored in the local cache.This design, on the one hand, to avoid the frequent connection to SQL databases, disconnected operation, on the other hand, the existence of the local cache, but also greatly reduces the load on SQL databases to improve the efficiency of the system response.
This module corresponds to the MX DNS record.
The traditional operation mode of recipient module is to get a message in, save to a local, or forwarded through the SMTP protocol.In ComConnect system, using LMTP (local mail transport protocol) protocol to connect to "prepared to receive the module" and "e-mail storage module", in these two modules, a number of channels to maintain a permanent, when the modules are prepared to receive new messages, it will use existing e-mail content delivery channel to the mail storage module to the background.This design greatly reduces the SMTP server is frequent fork, exit process, thereby improving efficiency.
This module is the most independent of the system modules.In order to improve security, ComConnect system achieved through letters cyrus-sasl user authentication server functions.Certification in part through "authentication modules" to complete.
As with other large-scale mail system, ComConnect system also needs a central database (support for SQL standard) modules.However, unlike other systems, ComConnect system save the contents of the central database is very small, just save the user account information.Moreover, due to "database middleware" and "cache" module exist in the ideal state, there did not need a central database.
This design greatly improves system performance, but also to improve the system scalability.Correspondingly, the other mail systems as much as possible information will be stored in a central database, with the number of users increases, the efficiency will be less and less and eventually become a system bottleneck.
Because stateless nature of HTTP protocol, so when the user needs to be saved after the user's login session (session) information.The traditional session modules are done through the database, so if the user is very large, the load will cause great database, and ultimately the formation of the whole system bottlenecks.ComConnect session handling system, using self-developed special session server processing, system out of a single server to act as session server, session within the definition of a set of protocols to maintain the status of each session, and customers in a certain timeautomatically deleted when the client does not have access to session records to achieve the session garbage collection mechanism.The module's database using an efficient memory database: BerkeleyDB3.Under this design, web server as the client session module, session server module as a server-side session.When the user login, Web server protocol access through the session the background session server, to record the information of the second session.
With the traditional session management mechanisms, this approach reduces the load on the central database, and because of the high-memory database, as well as the simplicity of the protocol session, greatly improving system responsiveness.
Part of a system that requires authentication: Web login, user authentication when sending letters, POP3 login.These certifications are certified by the local module, the module mechanism and "database middleware" like, through the unix domain socket for inter-process communication, and continuous connection with the database, and maintaining the local cache.
This is part of the user to use the Web interface.The module IMAP protocol, and the background of the "mail storage module" to communicate.One of the session, through the "HTTP session module" done.
The best part is the core of the system modules, all of the mail end users are saved in the module.Each user's mailbox is a directory, each message is a file.
Some large-scale mail system to mail header (header) saved to the database, and the message body (body) to a file.This design can improve the user mailbox in the speed of access, especially in the mailbox when a lot of messages, the other, when in actual implementation has the advantages of simple code.But it also highlights the shortcomings, first of all, "prepared to receive the module" to entering the processing of each message to extract the header and body, which reduces the efficiency of receipt of a letter; the other hand, since each messageINSERT trigger on the database operations, thereby increasing the burden on the database; Also, the number of incoming mail with the system increases, the number of records in the database will increase accordingly, and ultimately may be the bottleneck.
Different solutions with the above, ComConnect system established in the file system indexing mechanism, instead of a database index.Each user's mailbox (which corresponds to a directory), there is an index of all messages to the user.Thus, when the user mailbox, the speed can be very fast; In addition, each message into the message, "prepared to receive the module" do not need any treatment, can be transmitted directly through the LMTP channel to the "mail storage module", and "e-mail storagemodule "written directly to the content of the corresponding user's mailbox directory.
These eight modules of the ComConnect the entire system, where each module has a very good extension mechanism, by increasing the number of computers to improve performance.More detailed information, please visit: http://bms.elong.com/ads/
Here is some code database middleware, readers can learn about their operating mechanisms:
dbServer dbserver = db_unknown;
int midInited = 0;
/ Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware
*--------------------------------------------------------------
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp mid_init
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
*--------------------------------------------------------------
backup / bin / conf / data / log / maint / svn / tmp /
void
mid_init (const char * conf)
{
const char fname [] = "mid_init";
const char * server;
if (! conf | |! * conf) {
myconfig_read (MIDWARE_CONFIG_FIENAME);
} Else {
myconfig_read (conf);
}
server = myconfig_getstring ("backend_server", NULL);
if (! server | |! * server) {
syslog (LOG_ERR, "[% s] invalid''backend_server''in conf file", fname);
exit (1); / bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr /var / vmware yes, we exit directly backup / bin / conf / data / log / maint / svn / tmp /
}
/ * Judge what backend server is backup / bin / conf / data / log / maint / svn / tmp /
if (! strcasecmp (server, BACKEND_MYSQL)) {
syslog (LOG_DEBUG3, "[% s] backend server is% s", fname, BACKEND_MYSQL);
dbserver = db_mysql;
} Else if (! Strcasecmp (server, BACKEND_ORACLE)) {
syslog (LOG_DEBUG3, "[% s] backend server is% s", fname, BACKEND_ORACLE);
dbserver = db_oracle;
} Else if (! Strcasecmp (server, BACKEND_BERKELEYDB)) {
syslog (LOG_DEBUG3, "[% s] backend server is% s", fname, BACKEND_BERKELEYDB);
dbserver = db_berkeleydb;
} Else {
dbserver = db_unknown;
syslog (LOG_CRIT, "[% s] unknown backend server:% s", fname, server);
exit (1);
}
midInited = 1;
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware mid_initbackup / bin / conf / data / log / maint / svn / tmp /
/ Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware
*------------------------------------------------------------------------
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp sendCmd
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp - midware protocol implementation on midwared client side
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp ask midware server something via unix domain socket
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp RET:
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp CMD_ERR_SYS: system error
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp CMD_INVALID: null cmd sent to midwared server
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp CMD_OK: ok
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
*------------------------------------------------------------------------
backup / bin / conf / data / log / maint / svn / tmp /
static cmdResult
sendCmd (char * cmd, char * sqlcmd, const char * param2, const char * param3,
const char * param4, char ** reply)
{
const char fname [] = "sendCmd";
int s; int r, iovcount = 0;
struct sockaddr_un srvaddr;
struct iovec iov [6];
static char response [MAX_REP_LEN];
char sockfile [1024];
int start, n;
if (reply) * reply = NULL;
if (! cmd | |! * cmd) {
syslog (LOG_ERR, "[% s] null cmd specified", fname);
return CMD_INVALID;
}
/ * Create socket and connect to midware server backup / bin / conf / data / log / maint / svn / tmp /
s = socket (AF_UNIX, SOCK_STREAM, 0);
if (s == -1) {
syslog (LOG_CRIT, "[% s] create socket failed:% m", fname);
return CMD_ERR_SYS;
}
memset (sockfile, 0, sizeof (sockfile));
strncpy (sockfile, myconfig_getstring ("unixsock_dir", DEFAULT_MIDWARED_DIR),
sizeof (sockfile));
strcat (sockfile ,"/");
strcat (sockfile, SOCKET_FILENAME);
syslog (LOG_DEBUG3, "[% s] socket filename:% s", fname, sockfile);
memset ((char *) & srvaddr, 0, sizeof (srvaddr));
srvaddr.sun_family = AF_UNIX;
strncpy (srvaddr.sun_path, sockfile, sizeof (srvaddr.sun_path));
r = connect (s, (struct sockaddr *) & srvaddr, sizeof (srvaddr));
if (r == -1) {
if (reply) * reply = "Cannot connect to midwared server";
syslog (LOG_ERR, "[% s]: connect:% m", fname);
return CMD_ERR_SYS;
}
/ *
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp connected with the unix-domain socket, next
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp prepare the parameters for midwared server
backup / bin / conf / data / log / maint / svn / tmp /
iov [iovcount]. iov_base = (char *) cmd;
iov [iovcount]. iov_len = strlen (cmd) +1;
/ * Check sqlcmd backup / bin / conf / data / log / maint / svn / tmp /
if (sqlcmd & & * sqlcmd) {
iovcount + +;
iov [iovcount]. iov_base = (char *) sqlcmd;
iov [iovcount]. iov_len = strlen (sqlcmd) +1;
} Else {
goto startsend;
}
/ * Check parameter2 backup / bin / conf / data / log / maint / svn / tmp /
if (param2 & & * param2) {
iovcount + +;
iov [iovcount]. iov_base = (char *) param2;
iov [iovcount]. iov_len = strlen (param2) +1;
} Else {
goto startsend;
}
/ * Check parameter3 backup / bin / conf / data / log / maint / svn / tmp /
if (param3 & & * param3) {
iovcount + +;
iov [iovcount]. iov_base = (char *) param3;
iov [iovcount]. iov_len = strlen (param3) +1;
} Else {
goto startsend;
}
/ * Check parameter4 backup / bin / conf / data / log / maint / svn / tmp /
if (param4 & & * param4) {
iovcount + +;
iov [iovcount]. iov_base = (char *) param4;
iov [iovcount]. iov_len = strlen (param4) +1;
}
startsend:
retry_writev (s, iov, iovcount +1);
/ * Get reply from midwared server backup / bin / conf / data / log / maint / svn / tmp /
start = 0;
while (start n = read (s, response + start, sizeof (response) - 1 - start);
if (n <1) break;
start + = n;
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware whilebackup / bin / conf / data / log / maint / svn / tmp /
close (s);
/ * Marshell the reply for client call backup / bin / conf / data / log / maint / svn / tmp /
if (start <) = 1 {
/ * Failurely got the reply backup / bin / conf / data / log / maint / svn / tmp /
if (reply) {
* Reply = response;
}
return CMD_ERR_SYS;
} Else {
/ * Successfully got the reply backup / bin / conf / data / log / maint / svn / tmp /
response [start] =''0'';
if (reply) {
* Reply = response;
}
return CMD_OK;
}
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware sendCmdbackup / bin / conf / data / log / maint / svn / tmp /
/ Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware
*--------------------------------------------------------------
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp mid_query
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp - exported for end user directly
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp ARG -
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp key: used for cache
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp PRECON:
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp argument''result''must be initialized before this call
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
*--------------------------------------------------------------
backup / bin / conf / data / log / maint / svn / tmp /
midQueryResult
mid_query (char * sqlcmd, const char * key, int useCache, char result [])
{
extern char * cache_mapstr (int);
const char fname [] = "mid_query";
char * reply;
int r = 0;
if (midInited == 0) {
mid_init (NULL);
}
switch (dbserver) {
case db_mysql:
r = sendCmd (REQ_MYSQL_QUERY, sqlcmd, key, cache_mapstr (useCache),
NULL, (char **) & reply);
break;
case db_oracle:
r = sendCmd (REQ_ORACLEL_QUERY, sqlcmd, key, cache_mapstr (useCache),
NULL, (char **) & reply);
break;
case db_berkeleydb:
break;
default:
syslog (LOG_CRIT, "[% s] unknown backend server", fname);
break;
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware switchbackup / bin / conf / data / log / maint / svn / tmp /
/ * Let midware client know what happened backup / bin / conf / data / log / maint / svn / tmp /
if (result & & * result) strcpy (result, reply);
if (r! = CMD_OK) {
syslog (LOG_DEBUG3, "[% s] no reply, system error", fname);
return QUERY_SYS_ERR;
}
if (! strcmp (reply, REP_DB_QUERY_FAIL)) {
return QUERY_FAIL;
} Else if (! Strcmp (reply, REP_DB_QUERY_NOTFOUND)) {
return QUERY_NOTFOUND;
} Else if (! Strcmp (reply, REP_DB_QUERY_MANYRECORD)) {
return QUERY_FOUNDMANY;
} Else {
return QUERY_OK;
}
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware mid_querybackup / bin / conf / data / log / maint / svn / tmp /
/ Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware
*--------------------------------------------------------------
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp mid_insert
backup bin conf config data eshow_sitemap.html generate.sh log maint sitemap.html svn tmp
*--------------------------------------------------------------
backup / bin / conf / data / log / maint / svn / tmp /
midInsertResult
mid_insert (char * sqlcmd)
{
const char fname [] = "mid_insert";
char * reply;
int r = 0;
if (midInited == 0) {
mid_init (NULL);
}
switch (dbserver) {
case db_mysql:
r = sendCmd (REQ_MYSQL_INSERT, sqlcmd, NULL, NULL, NULL, (char **) & reply);
break;
case db_oracle:
r = sendCmd (REQ_ORACLE_INSERT, sqlcmd, NULL, NULL, NULL,
(Char **) & reply);
break;
case db_berkeleydb:
break;
default:
syslog (LOG_CRIT, "[% s] unknown backend server", fname);
break;
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware switchbackup / bin / conf / data / log / maint / svn / tmp /
if (r! = CMD_OK) {
syslog (LOG_DEBUG3, "[% s] no reply, system error", fname);
return INSERT_SYS_ERR;
}
if (! strcmp (reply, REP_DB_INSERT_FAIL)) {
return INSERT_FAIL;
} Else {
return INSERT_OK;
}
} / Bin / boot / dev / etc / home / lib / lost + found / media / misc / mnt / net / opt / proc / root / sbin / selinux / srv / sys / tmp / u01 / usr / var / vmware mid_insertbackup / bin / conf / data / log / maint / svn / tmp /