Don't Let the Robot Touch the Mail

There is a four line permission rule on the screen in front of me, and I have rewritten it three times.

data "aws_iam_policy_document" "certmanager_route53" {
  statement {
    actions   = ["route53:GetChange"]
    resources = ["arn:aws:route53:::change/*"]
  }

  statement {
    actions   = ["route53:ListHostedZonesByName"]
    resources = ["*"]
  }

  statement {
    actions = ["route53:ListResourceRecordSets"]
    resources = [
      "arn:aws:route53:::hostedzone/Z010939671A5NCHBBBCH",
      "arn:aws:route53:::hostedzone/Z05575802ULDZNGHPUH6L"
    ]
  }

  statement {
    actions = ["route53:ChangeResourceRecordSets"]
    resources = [
      "arn:aws:route53:::hostedzone/Z010939671A5NCHBBBCH",
      "arn:aws:route53:::hostedzone/Z05575802ULDZNGHPUH6L"
    ]

    condition {
      test     = "ForAllValues:StringLike"
      variable = "route53:ChangeResourceRecordSetsNormalizedRecordNames"
      values   = ["_acme-challenge.*"]
    }
  }
}

The job is boring. A certificate robot needs to prove it owns two domains, so it needs to write one record into DNS, wait for the certificate authority to read it back, then clean up after itself. Standard. Every cluster on earth does this.

The easy version is one line. Let the robot write anything it wants.

Here is why I am not writing the easy version.

An hour ago, into those exact same two zones, I put the company mail.

The mail exchange record that says where inbound messages land. The DKIM keys that sign every outgoing message. The DMARC policy that tells the entire internet to quarantine anything that does not carry that signature.

So the robot renewing a certificate at three in the morning is one careless permission away from stepping on the record that decides whether an invoice reaches a client.

And mail does not fail loudly. It does not bounce. It just... stops arriving.

So the four lines stay. The robot may change one kind of name and one kind only, the challenge record, the temporary one, the one nobody's inbox depends on. Everything else in that zone is off limits to it, permanently, by policy, not by good manners.

That is the shape of the whole day. This repository woke up as a one line project description and went to bed running two companies worth of email, a real staff directory with single sign on, and a stack of application addresses pointed at a live cluster.

All of it written down. All of it repeatable.

Well. Almost all of it.

Because one of those application addresses is not a fixed value at all. It is a question the code asks the internet, every single time it runs.

And the internet is allowed to change its answer.

The Honest Answer Is Almost Everything

Welcome back.

I left you looking at four lines of permission text and a certificate robot that wanted to write into the same two zones as the company mail. I'll come back to those four lines at the end. They're the smallest thing built today and the most considered. But they only make sense once you know what else went into those zones. The honest answer is: almost everything.

So let's start where the day started. This repository began as one line. A heading, with the project's name after it, no trailing newline. That's the entire contents.

# limitless-cloud

By the end of the day it runs the email for two companies, a staff directory people can actually log into, and roughly twenty named addresses pointed at a live cluster. The interesting part isn't the volume. It's the order things were done in, because the order tells you what the person at the keyboard was worried about.

It Was the Room

The first thing written wasn't infrastructure at all. It was the room. A development container definition — a small file that says: when somebody opens this project, give them Ubuntu, Terraform, the Terraform language server, and the cloud provider's command line tool. Then one more thing, my favorite line in the file. It writes a shell alias so that typing tf runs Terraform.

"postCreateCommand": "echo alias tf=\\'terraform\\' > ~/.bash_aliases"

Two characters instead of nine. On a day where you run that command four hundred times, that's not laziness. That's someone who already knows what today is going to feel like. And everyone who opens this project gets the same tool versions, so the bug where it works on one machine and not another never gets to exist here.

The Provider Is Pinned

Second thing written: the provider is pinned. Version five, region set to Northern Virginia. Which sounds like bookkeeping, and isn't — an unpinned provider means the plan you ran last month and the plan you run tonight are not the same program, and the difference shows up as a surprise in the middle of a change you were confident about.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

Room first. Tools pinned. Then work.

Built Twice, Written Once

Now, the shape of the problem. Two companies, living in one cloud account. The consultancy, and a second brand — a shorter, stranger domain name that ends in dot glass. And nearly everything built today has to be built twice, once per company, without the two of them touching.

So the project opens with a single variable that describes both. For each company: the domain name, the identifier of its zone — the container holding all the naming records for that domain — and a short alias used by the mail system.

variable "route53_zones" {
  type = object({
    glass = object({
      domain     = string
      hostedzone = string
      alias      = string
    })
    limitless = object({
      domain     = string
      hostedzone = string
      alias      = string
    })
  })

  default = {
    glass = {
      domain     = "madeof.glass"
      hostedzone = "Z010939671A5NCHBBBCH"
      alias      = "madeofglass"
    }
    limitless = {
      domain     = "limitlessinteractive.com"
      hostedzone = "Z05575802ULDZNGHPUH6L"
      alias      = "limitlessllc"
    }
  }
}

Everything downstream loops over that one structure. Which means "did we do this for both companies?" is never a question you have to check. You can't do it for one and forget the other, because you never wrote the names down twice in the first place.

A Confession in Capital Letters

Then the mail. And the mail file opens with a confession.

A comment, at the top, in capital letters. The mail service has to be created by hand, because there's no provider support for it.

### WORKMAIL needs to be manually created due to lack of terraform provider

I want to sit on that, because I think it's the most professionally mature line written today. The whole promise of this style of work is that the infrastructure lives in the file. You read the file, you know what exists. So the worst thing you can do is have a gap and not say so — because six months from now somebody reads this end to end, believes they've seen everything, tears it all down, rebuilds it, and discovers that the mailboxes didn't come back.

A tool boundary you've written down is a known limitation. One you haven't is a trap.

Two Hundred and Fifty Six Addresses

And around that manual step, everything that can be automated is. The mail service needs a directory behind it — a real corporate directory, the kind that holds employees and groups and passwords. Each company gets one, and each directory needs somewhere to live.

That means a private network per company. Both small — two hundred and fifty six addresses each, the smallest block the provider will accept, and about two hundred and fifty more than a directory service will ever need. Each is carved into two subnets, in two different availability zones. Separate buildings, separate power.

data "aws_availability_zones" "available" {
  state = "available"
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

resource "aws_vpc" "limitless" {
  cidr_block = format("10.1.0.0/24")
  tags = {
    Name = "limitless-activedirectory-vpc"
  }
}

resource "aws_subnet" "limitless" {
  count = 2

  vpc_id            = aws_vpc.limitless.id
  cidr_block        = cidrsubnet(aws_vpc.limitless.cidr_block, 4, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
}

Nobody is asking for high availability on a staff directory of this size. But the service requires two zones before it will start, and the reason is that a directory outage doesn't degrade gracefully. It just means nobody can log in.

And the project doesn't hardcode which two. It asks the provider for the available zones, then filters out the ones that require you to opt in — because your account can see those, but can't place anything in them until somebody enables it, and the failure you get is not a friendly one. That's someone who got burned by exactly that, once, and decided never again.

Ignore All Changes

Now, the directories need administrator passwords. Here I want to slow down, because the choice is unusual and I think it's correct.

Each password is generated — twenty eight characters, upper case, lower case, digits, explicitly no punctuation. And then it's given an instruction that overrides everything: ignore all changes.

resource "random_password" "limitless_ad" {
  length      = 28
  special     = false
  min_lower   = 1
  min_numeric = 1
  min_upper   = 1

  lifecycle {
    ignore_changes = all
  }
}

So the password is generated exactly once, on the first run, and after that the tool is forbidden from ever looking at it again. Change the length, change the character rules, change your mind entirely — no plan will ever propose regenerating it.

Which sounds like giving up. It's the opposite.

If that password regenerates, the directory it belongs to gets replaced. A replaced directory means the mail system on top of it loses its backing store — and the mail system is the one part of today's work created by hand, which cannot be rebuilt by running the project again. So a one-character edit to a password rule, made casually, on a Tuesday, would destroy the mailboxes of two companies.

That instruction is a fuse. It costs nothing, does nothing on any normal day, and exists solely to make one specific catastrophe unreachable.

The missing punctuation is the same instinct. Symbols get passed through shells and through automation that quotes things inconsistently. Letters and digits survive the trip.

Four Kinds of Record

So the directories are up. Now the actual mail — which, in practice, means writing down a set of public promises about who is allowed to speak for these two companies.

Four kinds of record go into each zone, and they do genuinely different jobs.

The Mail Exchange

The mail exchange record is the delivery address. Somebody just typed your address into their mail client — where does that message physically go? Here, the provider's inbound gateway. That one record is the difference between having email and not having email.

resource "aws_route53_record" "workmail_mx" {
  for_each = var.route53_zones

  zone_id = each.value.hostedzone
  name    = ""
  type    = "MX"
  ttl     = 300
  records = ["10 inbound-smtp.us-east-1.amazonaws.com."]
}

The Sender Policy

Then the sender policy record, plain text at the root of the domain: a list of who's permitted to send mail claiming to be from here. It ends with a soft qualifier — a tilde, followed by "all" — meaning anything not on the list is suspicious, but don't hard-fail it. That's cautious, and it's cautious because the internet is full of legitimate mail you forgot about.

resource "aws_route53_record" "workmail_spf" {
  for_each = var.route53_zones

  zone_id = each.value.hostedzone
  name    = ""
  type    = "TXT"
  ttl     = 3600
  records = ["v=spf1 include:amazonses.com ~all"]
}

The Signing Keys

Then the signing keys. Three per company, published as alias records that hand off to the provider's signing infrastructure. Every outgoing message gets a cryptographic signature, and these records are how the receiving end fetches the public half and checks it. Three, not one, because that's how you rotate keys without an outage.

resource "aws_route53_record" "workmail_dkim_limitless" {
  for_each = toset([
    "orf5c2cchlkzjys24rcu5tpk2ogt4an7",
    "op3dez7a2r4npmamjylo4ejcd2tiytbn",
    "q6ehf2rdshg2wz7yhx75ww5bhjq5yifa",
  ])

  zone_id = var.route53_zones.limitless["hostedzone"]
  name    = format("%s._domainkey", each.value)
  type    = "CNAME"
  ttl     = 3600
  records = [format("%s.dkim.amazonses.com.", each.value)]
}

The One With Teeth

And then the fourth record, which is the one with teeth.

It's called DMARC, and it's a standing instruction to every mail server on earth about what to do when a message claims to be from these domains and doesn't check out. Version one. Policy: quarantine. Percentage: one hundred. Report failures.

resource "aws_route53_record" "workmail_dmarc" {
  for_each = var.route53_zones

  zone_id = each.value.hostedzone
  name    = "_dmarc"
  type    = "TXT"
  ttl     = 3600
  records = ["v=DMARC1;p=quarantine;pct=100;fo=1"]
}

So: don't reject it outright, but don't put it in the inbox either. Put it in the junk folder. And apply that to every single message — no sampling, no gradual rollout.

Most organizations creep up on that over months. Publish in monitoring mode, watch the reports, find the four systems nobody remembered were sending mail, then tighten. This project skipped the creep and went straight to enforcement on day one. Defensible — the two things it protects are small enough that you can hold the whole sending surface in your head. But the rule is now live and absolute. Anything that sends mail as these companies and can't produce a matching signature disappears into a junk folder.

Hold onto that. It comes back.

The Invisible From

One more piece of plumbing, and it's the one people skip. Every message has two "from" values: the one you see in your mail client, and an invisible one the servers use between themselves. Failures come back to that second one, and by default it belongs to the provider, not to you. Each domain here gets four extra records so that it belongs to you.

resource "aws_ses_domain_mail_from" "sender_domain" {
  for_each = var.route53_zones

  domain           = aws_ses_domain_identity.sender_domain[each.key].domain
  mail_from_domain = format("bounce.%s", aws_ses_domain_identity.sender_domain[each.key].domain)
}

resource "aws_route53_record" "ses_domain_mail_from_mx" {
  for_each = var.route53_zones

  zone_id = each.value.hostedzone
  name    = aws_ses_domain_mail_from.sender_domain[each.key].mail_from_domain
  type    = "MX"
  ttl     = "600"
  records = ["10 feedback-smtp.us-east-1.amazonses.com"]
}

resource "aws_route53_record" "ses_domain_mail_from_txt" {
  for_each = var.route53_zones

  zone_id = each.value.hostedzone
  name    = aws_ses_domain_mail_from.sender_domain[each.key].mail_from_domain
  type    = "TXT"
  ttl     = "600"
  records = ["v=spf1 include:amazonses.com ~all"]
}

Now: People

Right. That's the mail. Now: people.

Because a mail system without accounts is furniture. Somebody has to be able to log in.

So there's a directory of real humans, defined as structured data — given name, family name, work address. Each one becomes a login. Username is the first name in lower case, display name is the full name properly capitalized, and the email address is generated from the same source rather than typed in.

resource "aws_identitystore_user" "user" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.limitless.identity_store_ids)[0]
  for_each          = var.iam_users

  display_name = format("%s %s", title(each.value["given_name"]), title(each.value["family_name"]))
  user_name    = lower(each.value["given_name"])

  name {
    given_name  = title(each.value["given_name"])
    family_name = title(each.value["family_name"])
  }

  emails {
    value   = format("%s@limitlessinteractive.com", lower(each.value["given_name"]))
    primary = true
  }
}

Which eliminates a whole category of afternoon. Nobody's address will ever have a typo in it, because nobody's address was ever typed. Then a group, called Internal Users. Everybody in the directory lands in it automatically.

And then the part that determines what that group can actually do. A permission set, named for the mail service, described as "access to the mail service," granted to the group. And exactly one capability attached: full access to the mail system. Nothing else. Not read access to the storage. Not the ability to list servers. Not view-only on the billing page.

resource "aws_identitystore_group" "internal_users" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.limitless.identity_store_ids)[0]
  display_name      = "Internal Users"
  description       = "Users with a basic level of access"
}

resource "aws_ssoadmin_permission_set" "workmail" {
  instance_arn = tolist(data.aws_ssoadmin_instances.limitless.arns)[0]
  name         = "WorkMail"
  description  = "Access to WorkMail"
}

resource "aws_ssoadmin_managed_policy_attachment" "workmail_full" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.limitless.arns)[0]
  managed_policy_arn = "arn:aws:iam::aws:policy/AmazonWorkMailFullAccess"
  permission_set_arn = aws_ssoadmin_permission_set.workmail.arn
}

So the full sentence today's identity work writes is: everyone in this company can log in as themselves, with a session that expires, and that session can do precisely one job. That is the correct shape, and it is genuinely rare to see it done on the first day rather than retrofitted after an audit.

The Account That Doesn't Follow the Rules

Sitting above all of that is one account that doesn't follow any of those rules. A single administrative user, with a permanent key, holding every permission in the account.

resource "aws_iam_user" "superadmin" {
  name = "superadmin"
  path = "/limitless/"
}

resource "aws_iam_access_key" "superadmin" {
  user = aws_iam_user.superadmin.name
}

resource "aws_iam_user_policy_attachment" "superadmin_access" {
  user       = aws_iam_user.superadmin.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

It exists because it has to. You cannot use a login system to build the login system — something has to have authority before there's anything to grant authority with. It's written down, with a comment above it explaining exactly what it is and how to get the secret out. I'm going to leave it there. That account is going to get an episode of its own, and it deserves one.

The Cluster. And the Robot.

Which brings us, finally, to the cluster. And to the robot.

There's a Kubernetes cluster, and the things running on it need certificates so that browsers will trust them. A certificate authority won't issue one until you prove you control the domain.

The way you prove it, for a wildcard certificate, is the naming system challenge. The authority gives you a random string. You publish it as a record at a specific name under your domain. The authority looks it up. If it's there, you've proven control, and you delete the record.

Certificates expire every ninety days, so this happens on a schedule, unattended, forever. Which is why it's a robot and not a person. And the robot needs credentials that let it write into your zones.

Now hold that next to what we just put in those zones.

The delivery record for two companies' mail. The signing keys. The sender policy. And the quarantine rule that tells the entire internet to junk anything that doesn't match.

The easy version of the robot's permission is one line: allow it to change records. Every tutorial you'll find writes it that way, because it works and because narrowing it is genuinely tedious.

Four Separate Statements

This project wrote four separate statements instead.

The first lets the robot check on the status of a change it submitted. Read only, and it applies to any change — which is fine, because a change identifier is meaningless without the change behind it. The second lets it find a zone by name. That one has to be unrestricted, because you can't look something up by name if you already have to know its identifier. It grants nothing except the ability to ask.

The third lets it read the records in a zone — and here the scope tightens. Not any zone. Two specific zones, named by identifier.

And the fourth is the one that matters. Permission to actually change records, in those same two zones, with a condition attached.

The condition says: for every name in this request, the name must begin with the challenge prefix. Not most of them. Every one — which closes the trick where you sneak a forbidden name in alongside a permitted one.

condition {
  test     = "ForAllValues:StringLike"
  variable = "route53:ChangeResourceRecordSetsNormalizedRecordNames"
  values   = ["_acme-challenge.*"]
}

So what the robot can do is now exactly one thing.

It can create and destroy the temporary challenge records, in two named zones, and nothing else. It cannot touch the mail records. It cannot repoint the website. It cannot delete the signing keys. Not because it's well-behaved — because the permission to do it doesn't exist.

That distinction is the whole discipline. The failure mode we're preventing isn't a malicious robot. It's a configuration mistake, at three in the morning, in software that has to be given write access to the most load-bearing records the business owns.

And mail failures are the worst kind to debug, because they're silent. A quarantined message doesn't bounce. It arrives nowhere and tells nobody. You find out when a client asks why you never replied to something you never received.

Sixteen Names, and Zero

Last piece of the day, and it's a list. The cluster's configuration includes every application address that should point at it. Sixteen names on the dot glass domain, and — this is the part I like — zero on the consultancy's domain. An empty list, written out explicitly.

variable "routes" {
  default = {
    glass = {
      domain     = "madeof.glass"
      hostedzone = "Z010939671A5NCHBBBCH"
      cnames = [
        "sso",
        "minecraft.public",
        "registry.lab",
        "alerts.mon",
        "grafana.mon",
        "prom.mon",
        "fetch.news",
        "xmit.news",
        "screen.news",
        "conf.news",
        "shorts.news",
        "music.news",
        "movies.news",
        "books.news",
        "library.news",
        "ai.home",
      ]
    }
    limitless = {
      domain     = "limitlessinteractive.com"
      hostedzone = "Z05575802ULDZNGHPUH6L"
      cnames     = []
    }
  }
}

That's not an oversight. That's someone saying, in code: the client-facing company deliberately runs nothing on this cluster, and I want the next person to see that I meant it.

And the sixteen are a portrait. Single sign on. A container registry, in a lab. Metrics, alerting, and dashboards. Feeds and reading tools. Music, movies, books, a library. Something for the home. And a Minecraft server.

You are looking at somebody's actual life. The monitoring is real work. The book library is not. And they're in the same list, at the same level of care, which is how it goes when the person running the infrastructure is also the person using it.

There's also one record in that file that's commented out. An address named "ingress," fully written, then disabled. The fossil of a plan that changed, still sitting there as a note to a future self.

# resource "aws_route53_record" "ingress" {
#   for_each = var.routes
#
#   zone_id = each.value.hostedzone
#   name    = "ingress"
#   type    = "A"
#   ttl     = 120
#   records = [data.external.current_ip.result.ip]
# }

That's a Real Day

So. Let's total up the day.

A repository that was one line this morning now describes: a reproducible development environment, a pinned provider, two private networks, two corporate directories, the complete mail configuration for two companies, a staff directory with single sign on scoped to one product, a tightly bounded credential for a certificate robot, and about twenty application addresses pointed at a live cluster.

All of it written down. All of it repeatable. One documented exception, flagged in capital letters at the top of the file where it matters.

That's a real day.

Don't Change This

Except for one thing. And I've been putting it off.

All twenty of those application addresses need an address to point at. So the project has to answer the question: what is the public address of the machine running this cluster?

And the way it answers is not by writing the number down.

There's a block in the cluster file with a comment above it that says, simply, don't change this.

What it does is shell out. It runs a small script that calls a public service on the internet and asks it one question: what address do you see when I talk to you?

Whatever comes back gets read into the project and written into every one of those naming records.

# Don't change - gets current IP
data "external" "current_ip" {
  program = ["bash", "-c", "curl -s 'https://ipinfo.io/json'"]
}

Which means the public address of two companies is not a fact stored in this repository. It's a question, asked fresh, from whatever desk happens to be running the command.

Run the project from the office, and the office is the website.

Run it from a laptop on hotel internet, and every one of those twenty names now points at a hotel.

Nothing errors. Nothing warns. The plan looks clean, because from the tool's point of view the answer changed and it's simply keeping the records accurate.

The comment says don't change this. It's the one line in the whole project that's going to have to.