Skip to main content

Durability and Reliability

1 min read

Many people tend to either confuse Durability and Reliability or assume they are the same. This is a quick write up to clarify that these are two fundamentally different yet critical concepts.

Message reliability focuses on the delivery of messages. Message reliability means a message needs to be delivered at least once with possible duplicates or at most once with no duplicates. Reliability also dictates whether messages must be delivered in the same order they were sent or can be delivered out of order.

Message durability focuses on the management of messages. Message durability dictates whether a message can be simply kept in-memory or if it must be saved to persisted storage to avoid message loss (in-memory messages are lost during system, network or power failures). Durable messaging also enables operations to resolve issues in a much easier manner than relying solely upon reliable messaging. When a durable messaging solution experiences an exception or outage it is much easier to determine the state of the transactions and restore/replay messages from a known state without losing any messages.

Note: I've been using this post to test webmentions.

Service Bus Refresher

12 min read

Re-post of an earlier blog entry, testing integration of my blog with Medium. Feedback/flames always welcome.

 

tldr;

Service Bus is a general term for an integration infrastructure that enables different systems or components to communicate through a shared interface, traditionally implemented as event-driven message queues.   Microsoft supports the service bus pattern across a number of products but this entry will only focus on two of them: Azure Service Bus and Service Bus for Windows Server.  These two products are sometimes simply referred to as "Service Bus" but it's important to realize their differences:

 

  • Azure Service Bus is a generic, cloud-based messaging system for connecting just about anything—applications, services, and devices—wherever they are. Connect apps running on Azure, on-premises—or both. You can even use Azure Service Bus to connect household appliances, sensors, and other devices like tablets or phones to a central application or to each other. 

 

  • Service Bus for Windows Server (SBWS) is a set of installable components that provides the messaging capabilities of Azure Service Bus on-premises. Service Bus for Windows Server enables you to build, test, and run loosely-coupled, message-driven applications in self-managed environments and on developer computers. 

 

Azure Service Bus and Service Bus for Windows Server are wire compatible via BrokeredMessages - this means messages generated by Azure Service Bus can be routed over Service Bus for Windows Server or Service Bus for Windows Server over Azure Service Bus.

 

Azure Service Bus and Service Bus for Windows Server share some common capabilities but also have fundamental differences.  See the sections below for more details on the differences and scenarios on which to use when.

 

Important Note: Service Bus for Windows Server lags behind the SDK support for Azure Service Bus due to the product team's "cloud first" approach. At the time this guidance was written Service Bus for Windows Server SDK was several releases behind the Azure SDK.  See the "Service Bus Releases" section of the Service Bus for Windows Release Notes for more information.  

 

Comparison of Azure Service Bus (ASB) and Service Bus for Windows Server (SBWS)

BrokeredMessages, Queues and Topics

  • ASB and SBWS both support BrokeredMessages.
  • ASB and SBWS both support a publish and subscribe pattern usingqueues andtopics.
    • With queues, each message sent to the queue is consumed by a single receiver.
    • With topics, messages are made available to each subscription registered with the topic. Each subscription logically maintains its own queue of messages. Subscriptions can also be configured with filter rules that restrict the set of messages passed to the subscription queue to those that match the filter.

Installation and support

  • ASB: Since ASB is a cloud-based service there is no software to install and support is provided by the Azure Team. OS and application patching and updating is also handled by the Azure Team. Storage is provided by and managed by the Azure Team. Backups and business continuity of ASB itself is managed by the Azure Team.
  • SBWS: SBWS runs on-premises and must be installed and configured on a Server 2013 or Server 2008 R2 machine (you can also install it on Win7 or Win8 for individual developer usage). SQL Server 2012 or 2008 is also required (Enterprise or Express versions). OS and application patching and updating must be handled by a local support team. Storage is provided by SQL Server and must be managed by a local support team. Backups and business continuity must be managed by a local support team.

Message Relays

  • ASB: ASB’s relay service supports direct one-way messaging, request/response messaging, and peer-to-peer messaging.  The Service Bus relay service enables you to build hybrid applications that run in both an Azure datacenter and your own on-premises enterprise environment.
  • SBWS: SBWS does not support relayed messaging.

Security

  • ASB: Shared Access Signature authentication (SAS) is the preferred way to implement authentication in Azure Service Bus.  SAS enables applications to authenticate to ASB using an access key configured on the namespace or the entity with which specific rights are associated. You can then use this key to generate a SAS token that clients use to authenticate to Service Bus.  See MSDN for more information on SAS tokens
  • SBWS: Solutions deployed on-premises using SBWS can use either Shared Access Signature authentication (SAS) or  Windows integrated security (Active Directory).  Applications can use SAS in scenarios in which they do not need to manage the notion of an authorized "user".  SBWS includes a Service Bus Security Token Service (SBSTS) which is fully integrated with the Windows security model. SBSTS can issue self-signed Simple Web Tokens (SWTs) using Windows identities. 

Quotas/runtime settings

  • ASB: Options regarding message and queue sizes are fixed due to the multi-tenant nature of ASB.  ASB Queue/Topic sizes can be up to 5 GB.  Messages can be up to 256K.  See here for specifics on ASB quotas.
  • SBWS: SBWS provides much greater freedom for configuring runtime settings and message and queue sizes.   SBWS Queue/Topic sizes can be up to 10 GB.  Messages can be up to 50MB.    See here for specifics on SBWS quotas.

Network latency

  • ASB: Network and security latencies may impact performance, especially when moving messages from on-premises to cloud or cloud to on-premises. Throughput could also be impacted by unexpected spikes in unrelated internet traffic.
  • SBWS: Administrators have much more control over their own network’s traffic and routing, thereby enabling more efficient bandwidth usage and throughput.

Addressing schemas

  • ASB: ASB Namespaces are fixed – all endpoints have a Service Bus postfix added to the URL to ensure uniqueness.
  • SBWS: SBWS uses fully qualified domain names (FQDN) or a mapped DNS entry to represent service namespaces.

Instances

  • ASB: In ASB there is only one "instance" of the service available and it is managed by Microsoft.
  • SBWS: With SBWS it is possible to have several on-premises SBWS servers via self-hosted servers or third-party providers.

Notification Hubs

  • ASB: Azure Notification Hubs provide an easy-to-use infrastructure that enables you to send mobile push notifications from any backend (in the cloud or on-premises) to any mobile platform.

 

  • SBWS: SBWS doesn't support Notification Hubs (although Notifications can be sent over SBWS).

Event Hubs

  • ASB: Event Hubs are highly scalable publish-subscribe event ingestors that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications.
  • SBWS: SBWS doesn't support Event Hubs (although Events can be sent over SBWS).

 

What to Use When:

 

Use Azure Service Bus when:

  • Building cloud applications or services and requiring a scalable messaging system between components
  • Crossing cloud and on-premises boundary
  • Message transformation or service orchestration are not required
  • Support for B2B integration is not required (AS2, X12, EDIFACT, …)
  • Relaying small messages (256 KB or less), which may contain the payload itself or may be notifications for the availability of larger payloads to process from Azure Storage Blob (see http://msdn.microsoft.com/en-us/library/hh690942.aspx)
  • You need to support AQMP in a cloud or hybrid architecture
  • Distributed Transactions (DTC) is not required
  • You need to build hybrid applications that run in Azure datacenter and on-premises without having to open a firewall connection or require intrusive changes to a corporate network infrastructure. (See note below regarding Service Relay vs. Hybrid Connections)
  • You need to support mobile push notifications from any backend (in the cloud or on-premises) to any mobile platform.
  • You need to integrate with and analyze large volumes of events on a near real-time basis (e.g. IOT, RFID and similar scenarios)

 

Use Service Bus for Windows Server when:

  • Publishers and Subscribers are all on premises.
  • Message transformation or service orchestration are not required
  • Support for B2B integration is not required (AS2, X12, EDIFACT, …)
  • You need to support large Queues/Topics or (up to 50 MB) or Messages (up to 10 GB)
  • You need to support AQMP on-premises
  • Distributed Transactions (DTC) are not required

Use BizTalk (on-premises BizTalk Server or Azure BizTalk Services) when:

  • Brokered application-to-application message-based integration with data mapping and diverse communication mechanisms are required
  • You need to support service orchestration (on-premises) or workflows (cloud)
  • You need to support Content-Based Routing (CBR)
  • Dealing with long-running processes or configurable business rules
  • Implementing B2B standards (e.g. EDI, AS2, X12, EDIFACT others) for multiple trading partners
  • Integration with popular LOB systems (such as SAP) is required 
  • Visibility into integrated processes via BAM is required
  • You need simple, configurable business rules implemented by a broker

Use MSMQ when: 

  • You need to support simple asynchronous communications between on-premises Windows applications
  • Senders and receivers are not running at the same time
  • Message level logging is required
    • MSMQ makes a useful lightweight local transactional queue to support durable messaging when on-prem (especially if you’re using HTTP)
  • You aren't deploying into Azure (MSMQ isn't available in Azure)

 

A note on Service Bus Relays and Hybrid Connections

When connecting from Azure to on-premises Azure Service Bus Relays and Hybrid Connections initially seem to provide similar capabilities.  Service Bus Relay relies upon Service Bus and WCF while Hybrid Connections relies upon BizTalk Services. Here are some simple guidelines on when to use one over the other. 

 

Use Service Bus Relay when:   

  • You need to build hybrid applications that run both in Azure and on-premises
  • You need to enable a hybrid architecture at a lower cost than using BizTalk Services
  • You need to connect to a WCF service using WCF relay bindings
  • You need support for ACS security
  • You need to configure your own listening component or a WCF Service to use a relay binding
  • You need to avoid installing or using a special "agent" to configure and connect to on-premises web services
  • You can only support web service integrations via ACS
  • You need to consume web services from on-premises (not within Azure)

 

Use Hybrid Connections when:   

  • You need to consume on-premises resources (not just web services) from Azure, not from on-premises
  • You need to connect via any port
  • You need to connect to something other than a WCF service (e.g. an application or a database)
  • You need to support Shared Access Signatures
  • You need to integrate with non-.NET technologies in a hybrid architecture 
  • You need to connect to on-premises resources from Azure using BizTalk Services 

 

Scenarios

Service Bus for Windows Server

 

Azure Service Bus

Performance and Scaling

Azure Service Bus (ASB)

 

Service Bus for Windows Server (SBWS)

  • SBWS supports both scale out (add more compute nodes and databases) as well as scale up (add more memory and CPU to the existing nodes). These two main scenarios lead to different approaches: When a Service Bus farm is planned to support many topics and queues each with low\medium throughput, a scale out approach is recommended as you can keep adding databases and nodes. However, when high throughput of small numbers of topics and queues is expected, it is recommended to add more CPU and memory to your existing hosts and database servers.
  • For more details see MSDN for SBWS Performance Considerations

Business continuity   

Azure Service Bus (ASB)

 

 

Service Bus for Windows Server (SBWS)

 

Security

 

Manage

  • Deployment:
    • Azure Service Bus:
    • Service Bus for Windows Server:
      • SBWS runs on-premises and must be installed and configured on a Server 2013 or Server 2008 R2 machine (you can also install it on Win7 or Win8 for individual developer usage). SQL Server 2012 or 2008 is also required (Enterprise or Express versions). OS and application patching and updating must be handled by a local support team. Storage is provided by SQL Server and must be managed by a local support team. Backups and business continuity must be managed by a local support team.
      • See MSDN documentation on Planning and Configuring SBWS
      • See MSDN documentation on Managing Service Bus for Windows Server
      • See MSDN documentation on using Service Bus Explorer with Service Bus for Windows Server

 

 

 

Troubleshooting:

 

Tools and Scripts

 

References and Resources