, 7 min read

Considerations on a Newsletter Program

Original post is here eklausmeier.goip.de/blog/2024/02-11-considerations-on-a-newsletter-program.


1. Statement of the problem. This blog does not offer any newsletter functionality. If a reader is interested to know whether I have posted new content, he must either use an RSS feed or directly visit this site. WordPress offers the possibility of getting notified of new posts automatically. I.e., a user can easily subscribe for new content.

On my old WordPress blog, https://eklausmeier.wordpress.com, I had 79 subscribers. From their e-mail names, I would suspect that some of them were not really interested in my actual content but were a little bit spammy. Nevertheless, many seemed to be legitimate.

There are a lot of professional newsletter services on the market. For example:

  1. https://www.mailjet.com
  2. https://buttondown.email
  3. https://mailchimp.com
  4. https://omnisend.com

There are many more.

These solutions should be distinguished from mailing list software.

2. Data model. Initially, I thought of a single file used to store all information. Something like this: Handling of subscription-file: Read into a PHP hash table, change whatever needs change, and if there is a change required, e.g., new subscriber, then move the old file, and write a new file from the hash.

However, this file needs some protection using flock() to guard against simultaneous writing to it. After some thought it seems more advantageous to use a simple SQLite file, i.e., a database, which already handles concurrency out of the box.

A single database table suffices. Henceforth this table is called subscription.

Nr. Column type nullable Example or meaning
1 email text not null primary key, e.g., Peter.Miller@super.com
2 Firstname text null e.g., Peter
3 Lastname text null e.g., Miller
4 registration date not null date of registration, e.g., 06-Feb-2024
5 IP text not null e.g., 84.119.108.23, IP address of web client during initial subscription
6 status int not null 1=in-limbo
2=active
3=inactive
4=bounced during registration
5=bounced
7 token text not null e.g., uIYkEk+ylks=
computed with
$token = base64_encode(random_bytes(8));

State diagram for status is as below.

graph LR A(1=in-limbo) --> B(2=active) B --> C(3=inactive) A --> D(4=bounced during registration) B --> E(5=bounced)

Create script for SQLite is like this:

drop table subscription;

create table subscription (
    email       text primary key,
    firstname   text,
    lastname    text,
    registration    date not null,
    IP          text not null,
    status      int not null,
    token       text not null
);

The following SQL statements will be used:

  1. During sending out the newsletter: select email, firstname from subscription where status=1
  2. New subscriber: insert into subscription (email,firstname,lastname,registration,IP,status,token) values (...)
  3. Checking correct token: select token from subscription where email=:m
  4. Updating status column: update subscription set status=:s where email=:m

The following columns could be added to better cope with malicious users.

Nr. Column type nullable Example or meaning
8 lastRegist date null date of last registration, relevant only for multiple subscriptions for the same e-mail
9 lastIP text null last used IP of the web client, when used for multiple subscriptions

3. Sketch of solution. Here are considerations and requirements for a simple newsletter software.

  1. Programming this application in PHP is preferred as this can be installed on many hosting providers, which offer PHP, e-mail, DNS, etc.
  2. Have one single database table, called subscription, see above.
  3. Periodically reads incoming e-mails for new subscribers or unsubscription requests.
  4. New subscribers add an entry to the subscription table.
  5. Subscription requests will generate a random token, which is sent to the e-mail address.
  6. Unsubscribe requests set the status column to inactive in the subscription table.
  7. During deployment of a new post on the static site, or by manual start, send an e-mail to all recipients on the subscription table, which are active.
  8. The IP address of the registrating web client is stored. With this we can defend against flooding of e-mail addresses, which all bounce. For example, this IP address can then be blocked in the firewall of the web-server.

The token does not need to be overly confidential. Its purpose is to defend against funny/stupid/malicious actors, who want to unsubscribe people against their will.

Handling of e-mails: For reading e-mail you can use imap_headers(), for sending imap_mail(). Also see Sending email using PhpMailer with Gmail XOAUTH2, and Gmail Email Inbox using PHP with IMAP.

Subscribing to the mailing list works with an empty e-mail that states Subscribe in the subject line. For unsubscribing you send Unsubscribe in the subject line and the token in the body part. These two operations are also supported by a simple web-form, which essentially asks for the e-mail address and the token from the user and then sends the confirmation e-mail and sets the status in the subscription table.

Reading e-mails is done every 20 minutes, e.g., controlled by cron. The reading process then analyses the subject field for Subscribe and Unsubscribe. This process also checks for any bounces. In case of a bounce the status flag is set to either bounced or bounced during registration. No distinction is made betweeen hard or soft bounces.

A subscription request makes an entry in the subscription table and sets the status column to in-limbo. The sender receives an e-mail, which he must confirm by e-mail or web form. Once the confirming e-mail is received or the web form is used to confirm then the status column is set to active. If a new subscription request is made with an already existing e-mail address then a new token is generated and sent, and the status remains its previous status, e.g., it might remain active or in-limbo.

If a malicious user subscribes to multiple e-mail addresses, which he does not own, then all these e-mail addresses are set to in-limbo. If the legitimate user now wants to subscribe, he can do so without fuss, because new tokens are sent out for any subscription requests. This prevents that e-mail addresses are blocked, which are not confirmed.

4. Web forms. The HTML form for processing subscribe and unsubscribe requests looks very simple:

First name:
Last name:  
E-mail address:
Token:   (only required for unsubscribe)
               

Changing your e-mail address is done by subscribing to the new address, and then unsubscribing from the old one.

If you have lost or deleted the token for unsubscribing, then simply subscribe again with the same e-mail address. A token will be sent to you, which you then can use for unsubscribing.

While the e-mail address is mandatory, the first and last name are optional.

The actual e-mailing can be done with below simple HTML form:

Greeting:   Firstname will be taken
Content:  
                   

The following e-mails are sent depending on the circumstances:

  1. Once a user has entered his name and e-mail on the HTML form, he will be sent an e-mail to confirm his e-mail address with the generated token.
  2. If the user has unsubscribed from the mailing list, he will receive a confirmation e-mail, which confirms that he has unsubscribed. If the token is wrong then no e-mail will be sent.
  3. The actual content is sent to all members stored in the subscription table, which are active. I.e., this is the whole purpose of maintaining this e-mail list.

5. Effort estimation. I expect the whole code for this to be no more than 1kLines of PHP code. I expect the following PHP programs/files:

  1. Handling the web form.
  2. Run through cron and checking for new subscription or unsubscription requests. Checking for bounces.
  3. Configurations for user-id, password, and hostname for e-mail host.
  4. Sending an e-mail to each recipient in the subscription table, either by using a web form, or via command-line, taking a text file as input.

Possible problems ahead due to hosting limitations:

  1. If you want to use Google Mail as mail provider you will encounter their limit of 500 mails per day.
  2. Yahoo seems to have a limit of 500 mails per day.
  3. Outlook also has a 500 mails per day limit.
  4. IONOS imposes a 500 mails per hour limit.
  5. Hetzner similarly restricts to 500 mails per hour.
  6. Amazon SES has a limit of 200 mails per day

To counter above limits somewhat, you can split your e-mails into batches, i.e., send 500 e-mails the first hour, then another 500 mails the next hour. For this you need an additional table, which stores the batch-number, and the message text to be sent. Obviously, you will not actually send 500 e-mails, but rather 450 or so, to cope for the confirmation mails for new subscribers or unsubscribers.

I am quite surprised that a Google search didn't reveal any program, which already does something similar. The most resembling is this phpList.