HI Kenny,
This is what I was afraid of. Biggest bottleneck in this chain is email generation speed. This is due to the fact, that each email is built up for specific recipient and than ran through multiple processes.
Speed of email generation of course depends on few variables such as not only personalization, but also overall amount of macros in the text as well as links creation and links tracking.
Generally most costly is macro resolution as it is a text scanning operation where each occurrence of {% %} pattern has to be found and macro resolver is called on expression which evaluates macro and replaces its content with resolved result. Of course personalization is next on the list as each personalized item means different content is created. And lastly ammount of links, especially tracked links as it means another resolution, that needs to be evaluated with special context (in order to create new link) and last are UTMparameters which are injected in each email.
These are all our implementation of public IIssueTextFilter interface:
DefaultIssueTextFilter
private readonly IssueTextMacroResolver mIssueTextMacroResolver;
private readonly IssueTextLinkToAbsoluteConverter mIssueTextLinkToAbsoluteConverter;
private readonly IssueTextTrackingLinkConverter mIssueTextTrackingLinkConverter;
private readonly IssueTextTrackingImageInjecter mIssueTextTrackingImageInjecter;
private readonly IssueTextUtmParametersInjector mIssueTextUtmParametersInjector;
So this mostly depends on actual machine processing power. In this case you could scale up in azure via web farms and use a single server as a processing and sending machine with tapping to our service
Service<IIssueSender>.Entry().Send(IssueInfoProvider.GetIssueInfo(parentIssueId), mailoutTime);
or creating custom one to send emails.
I will try to explain few more available methods:
EmailQueueManager.GenerateEmailsInternal: The list of unsubscribed emails is loaded as some subscribers may be subscribed as part of a group and have opted out of the email campaign individually. Next the generator updates the email status to "Preparing data" indicating that the individual emails are being prepared. If the email is a variant, the parent email's status is also updated to "Preparing data". The generate method then it checks if bounce monitoring is enabled and what the bounce limit is. Next the individual emails for direct subscribers are generated (more on this later). If the email is a variant, the A/B test is retrieved and the number of individual emails per variant is calculated and saved to the A/B test info. Finally, the "Newsletter.Emails.RemoveDuplicates" query is executed to remove duplicate emails.
EmailQueueManager.GenerateContactEmails: The contact emails are generated by first executing the "Newsletter.SubscriberNewsletter.SelectSubscriptions" query passing a where clause filtering the subscribers to the specific email campaign with an approved or null subscription and a type of contact with bounces under the limit to get the contact details for individual subscribers. The contact IDs are filtered to only unique IDs and then the subscriber details are retrieved 10,000 records at a time and saved to a data table in memory. Next all the contact groups that are subscribed are retrieved and the existing in memory data table is updated to add the subscriber details for each of the subscribed contact group's members. If there are any subscriber details in the in memory data table after loading these two sets of data, the necessary data is bulk inserted into the "Newsletter_Emails" table.
EmailQueueManager.GenerateBasicEmails: The direct subscribers emails are generated by executing the "Newsletter.Emails.CreateEmails" query, passing in the appropriate parameters.
NewsletterSender.Execute: The "Send email campaigns" scheduled task, which runs once a minute by default, runs an asynchronous thread to process all new and failed emails in the email campaign queue.
ThreadEmailSender.Run: When a mail out task is executed, the task will get the email info and clone it so some temporary updates can be made to it without affecting the original. Additionally, the thread will retrieve the associated email campaign, email template, and site info for the email or variant. If email is not enabled for the site and the "Generate campaign emails if emails are disabled" option in the Settings application under On-line Marketing/Email Marketing is disabled the mail out task thread exits. Otherwise, the thread will continue and update the email info status to sending. If the email has an email template associated with it, the template is retrieved and the template macros are resolved (e.g. editable text regions values are inserted for the placeholders, any macros in text regions are not resolved at this time). This is then saved to the cloned email info's body. Next the email body is sent through a filter to resolve the email campaign and email macros while leaving any unresolved context macros (e.g. firstname) in place to be resolved in the future. The filter also converts links to absolute URLs, adds the UTM parameters, converts links to tracking links (if applicable), and inserts the tracking image (if applicable).