Cloud Access Management Explained
Security groups, IAM roles, firewall rules, and VPCs: there's a method to the madness
Have you ever launched a PostgreSQL database in AWS or GCP, tried to access it, and your connection timed out? Or you try to hook up Fivetran, Census, Hightouch, Mode, or Looker and those connections also time out? Ah, the classic whitelisting issue—some security setting somewhere isn’t allowing access to the database (or the database’s broader network) from relevant IPs.
Networking is a complex beast. Visualizing how different services talk to each other can be extremely abstract, but is important in the world of security. Exposing a database to the entire internet with access only guarded by a password carries a lot of risk.
Although it has high market share, AWS is praised by software engineers for its granular configuration settings but dreaded by many others due to the steep learning curve of those same configuration settings. Networking on GCP is easier to get started with, but controlling the complexity of the connections is more difficult.
To my fellow analytics professionals, we don’t have to dread thinking about networking. There’s an argument for us not having to think about networking so deeply at all, but that’s a post for another day.
In the meantime, let’s unpack how some of the fundamental networking components work.
Networking 101
As my goal here is to look at AWS networking concepts, I’ll be referencing AWS resources names, but always tying them back to their GCP counter parts (sorry Azure, I’m not as familiar with you).
Consider you have two separate databases for two separate use cases, and let them be RDS instances. Let database A be used to hold third-party data, exposed to the internet, while database B is used for internal user data that should be highly protected and deployed into a private network.
VPC stands for “virtual private cloud”, a network you maintain, with more details in AWS’s documentation as they are much more qualified to go into the depths of private networks than I am. At a high level, it’s your own little piece of the cloud that you control access to.
Each VPC can have different subnetworks, or subnets. A public subnet can be accessed from the broader internet, while a private subnet can only be accessed from resources within the same VPC (or other trusted and peered VPCs).
To illustrate this, consider an Airflow webserver deployed on AWS resources (ex EC2, ECS, or EKS) that exposes access to all IPs. If Airflow is deployed on a resource in a public network, going to its IP and port 8080 will show the login screen. If deployed into a private network, giving all IPs access won’t help—access to the resource from the internet is disabled and the UI won’t load. Deploying into a private network is a security measure that can sometimes be considered extreme, other times a tactical one especially when dealing with confidential data.
So, database B should certainly be in a private network given the high level of security needed around the type of data being stored. Database A could also be in a private network, but for illustration purposes, let’s say we accept the third-party data being exposed in a public network. After all, some tools (looking at you Segment) can only connect to databases in a public network.
Now we need to give access to both databases to: internal employees connected to a VPN, self-deployed automation tools (like self-hosted Airflow/Dagster/Prefect), and third-party tools.
How do we do that? With security groups and IAM roles.
Wearing security hats and IAM shoes
Consider each database to be a person. A person (usually) only wears one pair of shoes. The comfort level of the shoes dictates how far the person can and wants to walk in them. And sometimes, shoes (and IAM roles) aren’t required (like at the beach). Let an IAM role to a service be like shoes to a person. The service usually only has one role, with policies attached to it dictating what is has access to. The IAM role gives permissions to the AWS API which includes creating or deleting other services.
Additionally, people can wear hats—not only one, but several at once. When skiing, you might wear a beanie and a helmet. Let a security group to a service be like a hat to a person, except that it’s fairly common to have multiple security groups. Security groups grant network(s) access to the service, and can be easily taken on and off of a resource. For comparison, shoes might need to be slowly untied and untangled—oftentimes changing the role on a service involves recreating the service altogether.
Between AWS and GCP, AWS IAM roles are more or less like service accounts in GCP. AWS security groups, generally more at the service level, are like VPC firewall rules in GCP that are controlled at the network level. More on AWS and GCP comparisons here.
Bringing this back into the networking world, an EC2 instance is created with a role that could allow it to create or delete other EC2 instances from within it. An RDS instance doesn’t necessarily need a role, unless monitoring happens in CloudWatch. Security groups are more often used for granting access to networks, like granting access to IPs on a particular port.
Given the databases outlined for this particular use case, one possible architecture in AWS is:
A VPC for all the services to be launched into
An IAM role with a policy granting access to CloudWatch
An RDS instance launched into a public subnet, representing database A for third party data
A security group on database A (RDS instance) with a list of
postgresql-tcpingress rules (accessing the database) for third-party services like Fivetran or Looker IPsA security group on database A with a list of
postgresql-tcpingress rules for internal access from the VPN IP to easily queryAn EC2 instance, serving as the VPN, launched into a public subnet. This EC2 instance must be launched with an IAM role, giving it the ability to do things like associate a particular static IP address
An RDS instance launched into a private subnet, representing database B for internal more confidential data
A security group on database B allowing a
postgresql-tcpconnection from the private subnet the VPN EC2 instance is running on
Final Thoughts
Lingo is helpful. Resources assume roles, and do what the roles allow them to do within AWS. On the other hand, security groups are attached, detached, and maybe attached again to resources. These security groups (specifically their ingress rules) give other networks access to the resource.
This isn’t the only possible architecture. There might be other best practices in GCP. Networking is easy if all resources are in public subnets and exposed to all access, but of course that approach exposes security risks. Consider what the minimal viable security setup is for a particular project and run with it.
Thanks for reading! I love talking data stacks so please reach out. I’d love to hear from you.