Security Token — How We Automate Firewall Reviews

by Rémi CATTIAU

You might have read about some of the recent data leaks which came from unsecured ElasticSearch servers directly connected to the internet.

This is why you get a thorough firewall review In security norms & regulations like PCI/DSS. In a public Cloud, updating security groups while testing and then forgetting about it, can happen really fast. Separation of duties is a way to prevent this from happening, but as you probably know, problems usually don’t have a unique cause.
The Nuxeo security team motto is: “automation, automation, automation”. And so, it seems obvious for us to face this new security challenge in an automated way. In this blog, I’m going to cover how we automated our firewall reviews and what we think about doing next in terms of automated security.

We already process all cloudtrail events in real-time, so we added a token at the end of every security group rule that is proof of acceptance from the security team. In our production environment, we enforce these tokens; any rule that does not contain or match a token authorizing the rule is removed by our automation.

A security group rule has several attributes: Protocol, Type, Target, Port, Description.

We decided to use the Description to add the token so if the rule is:

As we are deploying each customer in its own separate VPC, we came up with regex to validate the full deployment:

We use the regex to allow as target all the core-app-sg.

The tokens also have regex to validate which security group they are applied to.

In our custom Security Operations Center (SOC), people can request new tokens and once the request is validated by a Security team member, the token is delivered.

We have then the capabilities to see where a token is used within our AWS environments.

Those tokens are digitally signed to make them impossible to alter from the database (DynamoDB), here is a scraped export of one:

{ "_creationDate": "2020-06-17T16:37:42.056Z", "_lastUpdate": "2020-06-17T16:37:42.056Z", "data": "{\"GroupName\":\".*-alb-sg\",\"TokenType\":\"SG_RULE\",\"Target\":\"140.82.112.0/20\",\"Type\":\"INGRESS\",\"Description\":\"Github webhooks\",\"ToPort\":\"443\",\"IpProtocol\":\"TCP\",\"login\":\"uanon\",\"ts\":1592301098594}", "publicKey": "-----BEGIN PUBLIC KEY-----\r \nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA33UkG8UoMs+t4iw8/Th/\r \nFFpziS8MUxgKV/Trcs6kqRxz8Ytfj+F30+ylJ4uuiMxlPBZLAGPCCKOCIXUKJNvG\r \nH3AvW9zxf19+OLC0z/qdkCpf5x4LwBy9XCxwy7997YT/429Ah3RZF0GHaAR1Fwbp\r \nOkj8AlX2zv0FV9zx6i2k9eKmbsoD0a0dM7wCJq1p1So9Rj63aJXba4lDu44jLvuq\r \nZJu9Ty0cnDmsXqcR57ULByERqnx8nMCbvPWTq/Ya+UyGQuhbE7Lce3gJ5Wv3m9qU\r \npuVVJn4zGnksIl4mJZyShqDM7t4ETYstkghfOpJDI5ubuidWGOPXJIhmQ9DMtwFp\r \nox7T/q1MdosKej8VZanXQ9ELVemXhfpnnOjQK88Pc0HxbJK5nZAVaiXrb5Audi1R\r \nWRqdXeBMt9d/v9Ugl85VrOtbrlbbF7hK2L/UvxFaI1aoeXvHSkrwFtXDeNkvM8nf\r \nZp7Z0WEwjrpF7zcHE7mneZhmNHUxOQGBSbNHsQM6HNNOcczgx0RqcfbN63YA0gh6\r \nob6lbkU11qa6rYpWmwmKYj2GVEQ5TOJwFbDSH7+BwLhxRqGpro2Zx3AGXMxv/ECe\r \nlZaKHPAHllwoVrAuPS2VoqhqczkMoatouIzuMnbpBo9S3kohZVGmvmtqIghWPiW1\r \nilD0AF5Xf8BiDTNKZi2i8qECAwEAAQ==\r \n-----END PUBLIC KEY-----\r \n", "signature": "028a286c6e81b5f74f0397e30f1fcc9e9a7b2281148eceb85d67ddc83ba8e13059c8b6835cf7c23b247afd67613fa2de892f5a282f3a27ab5c21cafcdbe30d7679b4a2ba4b77bef6342c1cb5c027ef1a3b0f376c0e147564f3bb272b6086edaf6162f7fa83febe793c9fdea3b3babf34455a6c0e206dc965ffccccfe72e39a5ec633d6f7d695da1740fea1a0e1ecb30169e5a1168f4dc41d02b7fba3d46948d68f8c0425e25429a27da52da5acf8ed07a8aaf36fbc912d99c9b1aba2c4aeb994dd50c4f190144d0a69269a0c09fb5425547e1c19afa74d2c523fc8178262dbad4f379b09036bed70469becce171548d20d051616a78b9539eb2999039332fe989511530d5b53765e350587fc0b0a9ffc513c2a85c634da64f0fdaacdb82ba1d2658dcd6d2cb4c629e75cdec74ac457ac8bd0abc1e6c2ee0821c3b5a4f150f8b7b00dcd4bdb11255ecb90063396a793c6040fd5c5b6f8b223ae8457c15e918c0ec4722cc0acf5567e1abaa7cbb2c72f722d4dd10b122b1085886b3e4d398eea2f251a2857a208c34ba4b042d427444b3bfe6d2bc8548b7d179f793762f61de3298e079767eaab2fbffccebb36d5a5d271603e962408eb088648cc6ccdcc509650ff53c3d63ce3077906a40037642ae1c70c811b58d60a0ab24362e69a07ef8b36b0dbf0719615bf1c7fa1f39d4c8177fef3159fdbddc3f19c2d3dbc95a0d24b4f", "signedDate": "2020-06-17T16:37:39.515Z", "signer": "usecurity", "type": "SG_RULE", "uuid": "75_paKzzzeWSjq.h81cjWA" }

So whenever a SecurityGroup is modified, we do analyze each rule to verify they contain a token, if they don’t, we delete the rule to avoid any invalidated traffic. Also, as soon as we revoke an approval, it will remove all matching rules within the environments.

One of the goals of our team is to simplify as much as we can those processes, so if we identify a rule that matches a token but doesn’t contain one, we will add it on behalf of the user. We can then let them know that the process wasn’t respected, but work can continue seamlessly.

The process has been in place for almost 2 years now. We’re very happy with its implementation and the requirements are better understood by our teams.

But I have to admit, I was also responsible for an outage of an hour when we deployed the first time. Let me tell you what happened: in one of the first deployments, a bug sneaked in, letting the automation think the tokens were not legitimate, which led to a removal of all security group rules. We quickly identified the issues and our production team recovered within an hour. I could try to pretend it was some chaos engineering to ensure the disaster recovery process was working well but I’m too honest for that. We learn from our mistakes.

What’s next? We will soon be adding some DNS resolutions within token to allow some rules to include DNS name, so our system will help our teams to allow our office provider inside their system. We will also backport it as an IvoryShield module, let us know if you want to know more about it.

I hope this helped you understand how we manage our firewall review and I hope to share more of our security processes with you in the future, to continue showing you the Nuxeo Platform is best suited for organizations that have deep and complex regulations environments.

Originally published at https://www.nuxeo.com on October 20, 2020.