Self-Hosted Mail Server on a VDS: PTR, SPF, DKIM, DMARC

A self-hosted mail server on a VDS gives you full control over mailboxes, logs and policies, but it only works if the DNS and IP reputation basics are right. Install your mail stack (iRedMail, Mailcow or Zimbra), set a matching PTR (rDNS) record, publish SPF, DKIM and DMARC, and warm the IP up slowly. That combination is what keeps your messages out of the spam folder.

This guide walks through the full path on a VDS, including a Turkey location server if your users or data are in the region. The examples use documentation addresses (192.0.2.25, 2001:db8::25) and example.com. Replace them with your own.

Before you install anything

Deliverability problems are usually decided before the first message is sent. Check these items first:

• A clean, static IP. Look up your address on public blocklists before you commit to it. A previously abused IP is hard to fix.
• Outbound port 25. Confirm it is open for your plan. Many networks restrict it by default.
• A real FQDN. Use something like mail.example.com, with A (and AAAA, if you use IPv6) records pointing at the server.
• Dedicated resources. Mail stacks with antivirus and anti-spam engines are memory-hungry. Don't share the VDS with unrelated workloads.
• A fresh OS. Mailcow, iRedMail and Zimbra all expect a clean install. Don't layer them over an existing web stack.

Set the hostname so it matches your future PTR record:

hostnamectl set-hostname mail.example.com
echo "192.0.2.25 mail.example.com mail" >> /etc/hosts

Choosing between iRedMail, Mailcow and Zimbra

If you want the fastest route to a working, maintainable server, Mailcow is a good default. iRedMail suits people who prefer native packages. Zimbra makes sense when you need its collaboration features.

Mailcow install

Install Docker and Docker Compose first, then:

cd /opt
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
./generate_config.sh      # enter mail.example.com when asked
docker compose pull
docker compose up -d

Log in to the admin UI over HTTPS, change the default admin password immediately, then add your domain and mailboxes.

iRedMail install

On a supported, clean Debian/Ubuntu or RHEL-compatible system:

tar xjf iRedMail-x.y.z.tar.gz
cd iRedMail-x.y.z
bash iRedMail.sh

The installer asks for the storage path, backend and first domain. Reboot when it finishes.

Zimbra install

Download the installer for your OS release, extract it, and run:

./install.sh

Zimbra validates DNS during installation. If the MX or hostname lookup fails, fix DNS first rather than ignoring the warning.

Set the PTR (rDNS) record

Reverse DNS is the first thing many receiving servers check. The PTR for your IP must resolve to your mail hostname, and that hostname must resolve back to the same IP (forward-confirmed reverse DNS).

On Nixcon VDS services you can set the PTR from the control panel. Enter mail.example.com as the PTR value for 192.0.2.25 (and for the IPv6 address, if you send over IPv6). Then verify:

dig -x 192.0.2.25 +short
# mail.example.com.

dig mail.example.com A +short
# 192.0.2.25

Your SMTP banner and HELO name should use the same hostname. In Postfix:

myhostname = mail.example.com
smtp_helo_name = mail.example.com

A mismatch between HELO, PTR and A record is one of the most common reasons for rejection.

Publish MX, SPF, DKIM and DMARC

MX and SPF

example.com.        IN MX  10 mail.example.com.
example.com.        IN TXT "v=spf1 mx ip4:192.0.2.25 ip6:2001:db8::25 -all"

Keep SPF to one record per domain and stay under the 10-DNS-lookup limit. If you also send through a third-party service, add its include: mechanism to the same record, not a second one.

DKIM

Generate the key in your stack, then publish the public part:

• Mailcow: Configuration → ARC/DKIM keys → add a key for the domain, then copy the TXT record.
• iRedMail: run amavisd-new showkeys (the command name can vary by distribution) to display the public key.
• Zimbra: /opt/zimbra/libexec/zmdkimkeyutil -a -d example.com prints the record to publish.

The record looks like this:

dkim._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkq..."

Use a 2048-bit key if your DNS provider handles long TXT strings correctly. Some panels require splitting the value into multiple quoted strings.

DMARC

Start in monitoring mode, read the reports, then tighten:

_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=s; aspf=s"

After a few weeks of clean reports, move to p=quarantine, then p=reject. Don't jump straight to reject on a new domain.

Optional but useful

• MTA-STS and TLS-RPT to enforce and monitor TLS for inbound mail.
• A valid TLS certificate for the mail hostname (Let's Encrypt works fine; Mailcow handles this automatically).

Test before you send real mail

# SMTP and TLS check
openssl s_client -starttls smtp -connect mail.example.com:25

# Send a test message with authentication
swaks --to test@example.net --from you@example.com \
      --server mail.example.com --auth LOGIN --tls

Send to a few external mailboxes and inspect the headers. You want to see spf=pass, dkim=pass and dmarc=pass in the Authentication-Results line. Online mail testers that give you a one-time address are also handy for a quick score.

Keep your mail out of spam

Correct DNS gets you to the starting line. Reputation keeps you there.

1. Warm up gradually. A new IP that suddenly sends thousands of messages looks like spam. Start with low volumes to engaged recipients and increase over weeks.
2. Send only to people who want it. Use confirmed opt-in for lists and honor unsubscribes quickly. Include a List-Unsubscribe header on bulk mail.
3. Secure your accounts. A single compromised mailbox can burn your IP. Enforce strong passwords, enable rate limits per user, and watch the queue for sudden growth.
4. Close open relay and weak auth. Test that unauthenticated relaying is rejected.
5. Monitor feedback. Read DMARC aggregate reports, check bounce messages, and register with major mailbox provider postmaster tools.
6. Separate transactional and marketing mail. Use different subdomains or IPs so a campaign problem doesn't affect password-reset emails.

Common mistakes and fixes

• PTR points to a generic hostname. Update it in the panel to your mail FQDN and confirm with dig -x.
• Two SPF records. Merge them into one TXT record.
• DKIM selector mismatch. The selector in the signature header (s=dkim) must match the DNS name dkim._domainkey.
• Port 25 blocked outbound. Messages sit in the queue with connection timeouts. Check with nc -vz gmail-smtp-in.l.google.com 25.
• IP on a blocklist. Identify the cause, fix it, then request delisting. Don't just keep sending.
• No backups. Back up mail storage and your config. Mailcow ships a helper script (helper-scripts/backup_and_restore.sh).

Why a VDS in Turkey (or elsewhere)

Mail servers benefit from stable networking and a predictable IP. A VDS on a Turkey location server in İzmir keeps mail close to users and data in the region, while Frankfurt and Fremont points of presence help with reachability for international recipients. Nixcon operates its own network, AS40368, and you can check routes, ping and traceroute from İzmir, Frankfurt and the US on the public looking glass.

If you have your own ASN and IP space, a BGP-enabled VDS lets you announce your own block and send mail from addresses whose reputation you control. For servers that attract attacks, see the DDoS protection and firewall options.