Friday, 29 June 2018

Types of Queues in Exchange Server


The routing of a message determines the type of queue in which a message is stored. The following types of queues are used in Exchange 2010:

  • Submission queue   A persistent queue that's used by the categorizer to gather all messages that have to be resolved, routed, and processed by transport agents. The categorizer is a component of Exchange transport that processes all inbound messages and determines what to do with the messages based on information about the intended recipients.
All messages that are received by a transport server enter processing in the Submission queue. Messages are submitted through a Receive connector, the Pickup directory, or the store driver. The categorizer retrieves messages from this queue and, among other things, determines the location of the recipient and the route to that location. After categorization, the message is moved to a delivery queue or to the unreachable queue. Each Exchange 2010 transport server has only one Submission queue. Messages that are in the Submission queue can't be in other queues at the same time.

  • Mailbox delivery queue   The mailbox delivery queues hold messages that are being delivered to a Mailbox server by using encrypted Exchange RPC. Mailbox delivery queues exist on Hub Transport servers only. Mailbox delivery queues hold messages that are being delivered to mailbox recipients whose mailbox data is stored on a Mailbox server that's located in the same site as the Hub Transport server. More than one mailbox delivery queue can exist on a Hub Transport server. The next hop for a mailbox delivery queue is the distinguished name of the mailbox store.

  • Remote delivery queue   Remote delivery queues hold messages that are being delivered to a remote server by using SMTP. Remote delivery queues can exist on both Hub Transport servers and Edge Transport servers, and more than one remote delivery queue can exist on each server. Each remote delivery queue contains messages that are being routed to recipients that have the same delivery destination. Remote delivery queues are dynamically created when they're required and are automatically deleted from the server when they no longer hold messages and the configurable expiration time has passed. By default, a remote delivery queue is deleted three minutes after the last message has left the queue. The next hop for a remote delivery queue is an SMTP domain name, a smart host name or IP address, or an Active Directory site name.

  • Poison message queue   The poison message queue is a special queue that's used to isolate messages that are detected to be potentially harmful to the Exchange 2010 system after a server failure. Messages that contain errors that are potentially fatal to the Exchange system are delivered to the poison message queue. This queue is typically empty, and if no poison messages exist, the queue doesn't appear in the queue viewing interfaces. The poison message queue is always in a ready state. By default, all messages in this queue are suspended. The messages can be deleted if they're considered to be harmful to the system. If the event that caused the message to enter the poison message queue is determined to be unrelated to the message, delivery of the message can be resumed. When delivery is resumed, the message enters the Submission queue.

  • Unreachable queue   Each transport server can have only one Unreachable queue. The Unreachable queue contains messages that can't be routed to their destinations. Typically, an unreachable destination is caused by configuration changes that have modified the routing path for delivery. Regardless of destination, all messages that have unreachable recipients reside in this queue.

Queue database files

All the different queues are stored in a single ESE database. By default, this queue database is located on the transport server at
 %ExchangeInstallPath%TransportRoles\data\Queue.

Like any ESE database, the queue database uses log files to accept, track, and maintain data. To enhance performance, all message transactions are written first to log files and memory, and then to the database file. The checkpoint file tracks the transaction log entries that have been committed to the database. During an ordinary shutdown of the Microsoft Exchange Transport service, uncommitted database changes that are found in the transaction logs are committed to the database.
Circular logging is used for the queue database. This means that transaction logs that are older than the current checkpoint are immediately and automatically deleted. Therefore, the transaction logs can't be replayed for queue database recovery from backup.

Exchange Server 2013 Mail Flow

Microsoft reduced the number of server roles in Exchange 2013 to just two in order to “increase simplicity of scale, hardware utilization and failure isolation”:

Mailbox Server role: 

which includes the Client Access protocols, Hub Transport service, Mailbox databases and Unified Messaging, and also handles all activity for a given mailbox.

Client Access Server role

which provides authentication, redirection and proxy services (it doesn't do any data rendering). The Client Access server is a thin and stateless server and there is never anything queued or stored on it. The Client Access server offers the usual client access protocols: HTTP (Outlook Anywhere, ActiveSync and Web Services), POP, IMAP and SMTP, but not MAPI (all MAPI connections are encapsulated using RPC-over-HTTPS)!

Note that the Edge Server is not present in Exchange 2013. This is because its services are now provided by the Client Access Server (although you can still use an Exchange 2007/2010 Edge server as a gateway with your Exchange 2013 environment).

In this new architecture, the Client Access server and the Mailbox server roles are not as dependent on one another as in previous version of Exchange because all processing and activity for mailboxes occurs on the Mailbox server where the active database copy of a mailbox resides. All data rendering and data transformation is performed local to the active database copy, eliminating concerns of version compatibility between the Client Access server and the Mailbox server.

Transport Pipeline:

Mail flow takes place through the Transport Pipeline which is a collection of services, connections, components and queues that work together to route messages. With Exchange 2013, the transport pipeline is now made up of 3 different services:

1. Front End Transport service

which runs on all Client Access servers, acts as a stateless proxy for all inbound and outbound external SMTP traffic. This service does not inspect message content but it can perform basic e-mail filtering based on connections, domains, senders and recipients. It only communicates with the Transport service on a Mailbox server and does not queue any messages locally.

2. Transport service 

runs on all Mailbox servers and is almost identical to the Hub Transport server role in previous versions of Exchange. It handles all SMTP mail flow for the organization, performs message categorization and content inspection. Unlike previous versions of Exchange, the Transport service never communicates directly with a mailbox database, which is now handled by the Mailbox Transport service. The Transport service routes messages between the Mailbox Transport service, the Transport service and the Front End Transport service. This service does queue messages locally.

3. Mailbox Transport service 

also runs on all Mailbox servers and is made of two separate services:

a. The Mailbox Transport Delivery service

receives SMTP messages from the Transport service and connects to the mailbox database using an Exchange Remote Procedure Call [RPC] to deliver the message.

b. The Mailbox Transport Submission service 

connects to the mailbox database using RPC to retrieve messages and submits them over SMTP to the Transport service. The Mailbox Transport service also does not queue any messages locally.
As Exchange 2013 no longer has an Edge Server role, e-mail messages from the Internet are received and sent to Exchange using a third-party e-mail gateway, an Exchange 2007/2010 Edge server or through the Exchange 2013 Client Access server as Microsoft intends it to be. In this last scenario, these e-mails enter the Exchange organization through a Receive connector in the Front End Transport service and are then routed to the Transport service on a Mailbox server.
While Exchange 2007/2010 Hub Transport servers were not configured out of the box to accept e-mails from the internet, the new Client Access server comes with a Receive Connector named “Default Front end <server_name>” that is already configured to allow “Anonymous Users” to connect to it:

Transport service on Edge Transport servers 

This service is very similar to the Transport service on Mailbox servers. If you have an Edge Transport server installed in the perimeter network, all mail coming from the Internet or going to the Internet flows through the Transport service Edge Transport server. 



The following figure shows the relationships among the components in the Exchange 2013 transport pipeline.



Mail flow documentation:


Topic
Description
Mail routing describes how messages are transmitted between messaging servers.
Connectors define where and how messages are transmitted to and from Exchange servers.
Accepted domains define the SMTP address spaces that are used in the Exchange organization. Remote domains configure message formatting and encoding settings for messages sent to external domains.
Transport agents act on messages as they travel through the Exchange transport pipeline.
Transport high availability describes how Exchange 2013 keeps redundant copies of messages during transit and after delivery.
Transport logs record what happens to messages as they flow through the transport pipeline.
Moderated transport requires approval for messages sent to specific recipients.
Content conversion controls the Transport Neutral encoding format (TNEF) message conversion options for external recipients, and the MAPI conversion options for internal recipients.
Delivery status notifications (DSNs) are the system messages that are sent to message senders, for example, non-delivery reports (NDRs).
Delivery Reports is a message tracking tool that you can use to search for delivery status on email messages sent to or from users in your organization's address book, with a certain subject. You can track delivery information about messages sent by or received from any specific mailbox in your organization.
This topic describes the size and individual component limits that are imposed on messages.
You use the Queue Viewer in the Exchange Toolbox to view and act upon queues and message in queues.
The pickup and replay directories are used to insert message files into the transport pipeline.
This topic describes the considerations for using an Edge Transport server from previous versions of Exchange in Exchange 2013.


Messages from external senders

Messages from outside the organization enter the transport pipeline through a Receive connector in the Front End Transport service on the Client Access server and are then routed to the Transport service on the Mailbox server.
If you have an Exchange 2013 Edge Transport server installed in the perimeter network, messages from outside the organization enter the transport pipeline through a Receive connector in the Transport service on the Edge Transport server. Where the messages go next depends on how your internal Exchange servers are configured.

·         Mailbox server and Client Access server installed on the same computer   In this configuration, the Client Access server is used for inbound mail flow. Mail flows from the Transport service on the Edge Transport server to the Front End Transport service on the Client Access server, and then to the Transport service on the Mailbox server.

·         Mailbox server and Client Access server installed on different computers   In this configuration, the Client Access server is bypassed for inbound mail flow. Mail flows from the Transport service on the Edge Transport server to the Transport service on the Mailbox server.

Messages from internal senders

SMTP messages from inside the organization enter the transport pipeline through the Transport service on a Mailbox server in one of the following ways:
·         Through a Receive connector.
·         From the Pickup directory or the Replay directory.
·         From the Mailbox Transport service.
·         Through agent submission.

The message is routed based on the routing destination or delivery group. If the message has external recipients, the message is routed from the Transport service on the Mailbox server to the Internet, or from the Mailbox server to the Front End Transport service on a Client Access server and then to the Internet if the Send connector is configured to proxy outbound connections through the Client Access server.
If you have an Edge Transport server installed in the perimeter network, messages that have external recipients are never routed through the Front End Transport service on a Client Access server. The message is routed from the Transport service on a Mailbox server to the Transport service on the Edge Transport server.

Transport service on Mailbox servers

Every message that's sent or received in an Exchange 2013 organization must be categorized in the Transport service on a Mailbox server before it can be routed and delivered. After a message has been categorized, it's put in a delivery queue for delivery to the destination mailbox database, the destination database availability group (DAG), Active Directory site, or Active Directory forest, or to the destination domain outside the organization.

The Transport service on a Mailbox server consists of the following components and processes:
·         SMTP Receive   When messages are received by the Transport service, message content inspection is performed, transport rules are applied, and anti-spam and anti-malware inspection is performed if they are enabled. The SMTP session has a series of events that work together in a specific order to validate the contents of a message before it's accepted. After a message has passed completely through SMTP Receive and isn't rejected by receive events, or by an anti-spam and anti-malware agent, it's put in the Submission queue.

·         Submission   Submission is the process of putting messages into the Submission queue. The categorizer picks up one message at a time for categorization. Submission happens in three ways:
o    From SMTP Receive through a Receive connector.
o    Through the Pickup directory or the Replay directory. These directories exist on Mailbox servers and Edge Transport servers. Correctly formatted message files that are copied into the Pickup directory or the Replay directory are put directly into the Submission queue.
o    Through a transport agent.

·         Categorizer   The categorizer picks up one message at a time from the Submission queue. The categorizer completes the following steps:
o    Recipient resolution, which includes top-level addressing, expansion, and bifurcation.
o    Routing resolution.
o    Content conversion.

Additionally, mail flow rules that are defined by the organization are applied. After messages have been categorized, they're put into a delivery queue that's based on the destination of the message. Messages are queued by the destination mailbox database, DAG, Active Directory site, Active Directory forest or external domain.

·         SMTP Send   How messages are routed from the Transport service depends on the location of the message recipients relative to the Mailbox server where categorization occurred. The message could be routed to one of the following locations:

o    To the Mailbox Transport service on the same Mailbox server.
o    To the Mailbox Transport service on a different Mailbox server that's part of the same DAG.
o    To the Transport service on a Mailbox server in a different DAG, Active Directory site, or Active Directory forest.
o    For delivery to the Internet through a Send connector on the same Mailbox server, through the Transport service on a different Mailbox server, through the Front End Transport service on a Client Access server, or through the Transport service on an Edge Transport server in the perimeter network.

Transport service on Edge Transport servers


The components of the Transport service on Edge Transport servers are identical to the components of the Transport service on Mailbox servers. However, what actually happens during each stage of processing on Edge Transport servers is different. The differences are described in the following list.

  • ·         SMTP Receive   When an Edge Transport server is subscribed to an internal Active Directory site, the default Receive connector is automatically configured to accept mail from internal Mailbox servers and from the Internet. When Internet messages arrive at the Edge Transport server, anti-spam agents filter connections and message contents, and help identify the sender and the recipient while the message is being accepted into the organization. The anti-spam agents are installed and enabled by default. Additional attachment filtering and connection filtering features are available, but built-in malware filtering is not. Also, transport rules are controlled by the Edge Rule agent. Compared to the Transport Rule agent on Mailbox servers, only a small subset of transport rule conditions are available on Edge Transport servers. But, there are unique transport rule actions related to SMTP connections that are available only on Edge Transport servers.

  • ·         Submission   On an Edge Transport server, messages typically enter the Submission queue through a Receive connector. However, the Pickup directory and the Replay directory are also available.

  • ·         Categorizer   On an Edge Transport server, categorization is a short process in which the message is put directly into a delivery queue for delivery to internal or external recipients.

  •        SMTP Send   When an Edge Transport server is subscribed to an internal Active Directory site, two Send connectors are automatically created and configured. One is responsible for sending outbound mail to Internet recipients; the other is responsible for sending inbound mail from the Internet to internal recipients. Inbound mail is sent to the Transport service on an available Mailbox server in the subscribed Active Directory site.