A Brief Analysis of Serverless Security
While scrolling through news last weekend, I came across the largest-ever cyber attack (DDoS) that hit the popular code hosting website GitHub. It was the distributed denial of service that originated from over a thousand different autonomous systems (ASNs) which peaked at 1.35 Tbps.
This could be you tomorrow. Or me too! With bad actors exploiting our vulnerable systems, at Simform we prepare ourselves with the best exercises to handle whatever comes our way in delivering software development services to the clients! During our journey towards Serverless Architecture, we’ve found how serverless application security is different from the traditional cloud.
Serverless can be a dangerous place unless you prepare yourself with the best practices. There are bullies out there who desperately wants to break into your system. DDoS attackers, ransomware distributors and all kinds of cyber creeps preying on our databases and poorly configured serverless functions.
In this blog, we will study the serverless (AWS Lambda) security issues & concerns and how you can strengthen them. Along with the application of continuous security and monitoring exercises as a defensive perimeter.
- Serverless Security: How Different it is from the Cloud?
- AWS Lambda Security: Common Issues & Best Practices
- What to do Next?
Serverless Security: How is it Different from Cloud
We, at Simform, have observed many aspects that differentiate the Serverless security landscape from cloud security. And that’s what I want to discuss first.
No physical servers to manage
According to the statistics today, many successful exploits in the world today are due to a problem with hygienic conditions of the servers. Most attacks exploit vulnerabilities that have not been patched despite being known for months.
Symantec actually thinks that by 2020, 99% of the vulnerabilities that are exploited in the world have been known for over a year. It is sort of an inherent technical challenge and when you deal with service, management is not your responsibility. So, your only responsibility is your data and hence you only care about choosing a platform that you trust to do it.
Since functions are a service, their security becomes the concern of your service provider. Patching servers and their up-to-date management are one of the core competencies and it will be managed rightfully by your service provider.
Flexibility = higher data at risk!
Since functions are stateless, you move a lot of data across the network which has more opportunity for that information to leak.
Though you have the benefit of granular accessibility and flexibility, it exposes the risk of data leakage. So there are things which makes it better and worse at the same time. But here’s how we tackle it.
Before moving to that, you need to measure answers to a certain question. What sort of data you’re transmitting across your network and how are you interacting with it? How well is your service provider protecting the data? Are you communicating over the secure channel? Is it something installed over your VPN or you’re communicating over HTTP?
It is really easy to tamper with the data one or the other way. Most of the time, people rely on DNS for service authentication which isn’t the best practice.
Encryption, definitely, is your primary friend here. Classify data processed, stored or transmitted by an application. Encrypt all data in transit with security protocols such as TLS with PFS. Or you may enforce encryption using directives like HTTP Strict Transport Security.
Plus, you have to minimize functions that can access each data store. Do not let every function access to each data store. Use of separate credentials for the different set of functions is recommended. Remember, once you add permission, it is hard to remove it and hence it is better to go slow and precise.
More to that, you should monitor, monitor and monitor, about which we will discuss in the next point.
Monitoring of unused functions
We know monitoring can be a cumbersome task. It is recommended to monitor what’s going on inside a function and also its activity amongst the other functions. When you have the combination of easy deployment with minimal cost, and just like permissions, as I mentioned before, once you’ve added a function it is really hard to remove.
With a lack of precise monitoring solutions, function tracking is not reliant. When you have hundreds of such functions, every one of these is at a risk with an open opportunity for the attacker to come in, penetrate your system and move ahead. Although the operation cost is extremely low, one should keep in mind the total cost of ownership.
While this challenge is ingrained in the serverless ecosystem, the existing monitoring solutions are built for an end-to-end app, plus, their logical interface isn’t designed to handle the fine granularity of serverless architecture. Thus it is better to rely on simple tools like this open-source project: list-lambdas by Epsagon.
It’s a Python script which lists Lambda functions across all AWS regions. It also includes some relevant information such as the function’s memory settings, invocation time and last modified time.
Thus it identifies a function that hasn’t been used for a long time is most likely outdated and may have wrong permissions or contain old external libraries.
Security perimeter and granularity
As we discussed already, Flexibility is great, but at the same time, there is more opportunity for attackers to do things that you did not intend to do and your system is at higher risk.
The other aspect is the loss of the perimeter. In a traditional application, you know that you have a bunch of functionalities tied up together and you have a security perimeter around it. With serverless architecture, each function has its own perimeter and you need to build it technically and perceptually.
This means you have to monitor their input and output and worry about securing AWS Lambda and its code dependencies. However, this enforces a higher responsibility on the developers and testing automation engineers to exactly control and monitor what each function can and cannot do.
A suggested practice would be to test each function separately for its security flaws by unit testing and test-driven development. Since functions are invoked through outside API, you should exercise strict constraints on their authentication.
Multi-Tenancy- a low-priority threat
Multi-tenancy means your functions are running on a single physical server which is shared between multiple users.
Although, ensuring that no security problems occur is a principal priority for any service provider. However, recalling the opinion of Mike Roberts from his elaborate article on serverless architecture, due to possible hosting application errors, one tenant may potentially access other tenants data.
The chances of this scenario to occur are very low but low-probability concerns are still concerns and one should not neglect them.
Benefit of Statelessness
Typical attackers find some loopholes that would advance in a step in the operating stages. They then disclose some malicious agents in that spot which would continue forward.
For example, the Sony breach and Panama Papers were high profile vulnerability breaches and data leaks which were actions of an agent installed on the vulnerable system which leaked information out slowly and gradually.
However, with functions, we are naturally stateless and therefore we cannot enter a bad state at all. Before the malicious agent would compromise with the server, it will get reset very quickly as it is in operation for that particular invocation. Due to this, attackers will have to repeat it again and again which has fair chances of risk detection.
Ory Segal in a recent webinar with Rupak Ganguly from Serverless Inc. discussed the problems that lie with third-party dependencies and too many permissions. He suggests having a gatekeeper which monitors if there are possible vulnerabilities before you deploy your code. As he says that there is no reason for something to pass the build and get too close to the cloud.
Further, he says, with the benefit of the stateless characteristics, knowing that the function’s code is immutable is very reassuring and important as a developer. Due to the ephemeral nature, malicious code attempts are prone to detection. In the near future, we will see behavioural models based on the best practices which will be able to detect abnormal function behaviour and thus alarm us.
DDoS is a billing issue
What is DDoS? The Distributed Denial of Service attack means some stream of requests will be sent to your server. These requests will try to find the area where these requests will take the proportionate amount of time to process. Attackers make it process over a prolonged time which ultimately will keep your server busy.
Since you are running on a definite amount of computing resources, this takes your server down. Once it’s occupied processing these requests, it is not serving other requests. But with functions, since autoscaling is available and you have a much broader elasticity, it is hard to take down your servers.
However, the serverless platforms are not infinitely scalable. They too have computed resource limits, for example, the default limit of concurrent function execution is 600 for AWS Lambda. The other thing is your bill. You still need to pay for the executions and if the attackers found a way to make you charge the hefty amount, it will be a great loss.
AWS Lambda Security: Common Issues & Best Practices
#1. Function event data injection
Injection vulnerabilities and errors occur when an untrusted hostile data is sent to the interpreter and is eventually executed.
In the case of serverless architecture, function event data injection is vulnerable to multiple points since functions are event triggered and can be invoked by multiple types of event sources. Some of them are:
- HTTP API calls
- Database events
- Stream processing events (e.g. AWS Kinesis)
- IoT devices
- Push notification, emails, etc.
Serverless functions can be triggered from such event sources which may contain malicious code. When these user-supplied event sources are not validated, they are vulnerable to the attack landscape. Not only that, with the help of event sources, the execution of the function can be manipulated.
Solution: Keep your data separate from commands and queries. Make sure that the code is running with minimum permissions required for successful execution. Use a safe API to invoke your function, which avoids the use of the interpreter entirely.
Use SELECT LIMIT and other SQL commands (if your function is dealing with SQL database) to prevent the mass disclosure of records in case of SQL injection.
[ORDER BY expression [ ASC | DESC ]]
LIMIT number_rows [ OFFSET offset_value ];
#2. Broken authentication
Chances are, you might be consuming event sources from multiple sources, like NoSQL Database events, Serverless database, IoT devices, SMS notifications, etc. you might be exposing APIs from these services on the public domain. How does your system authenticate these endpoints?
For example, the trigger for your functions is Kinesis stream which process or analyze streaming data. Without proper authentication, the system is exposing an unauthenticated event trigger for the functions. May result in enabling the attacker to bypass application logic and hamper function and its execution.
Solution: Incorporating an extensive authentication system which exercises control and authentication over these APIs highly critical. You need to make sure that every endpoint is authenticated.
We store user’s active tokens in the database and authenticate them against every request. This will let you exercise token authentication, invocation count and expiration time limit. If you’re just starting with the serverless architecture, AWS Cognito is an easy solution which will suffice your needs.
If at all, user authentication is not an option, developers are recommended to use secure API keys or SAML assertions.
#3. Over-privileged IAM roles
Grant IAM role access according to the “Least Privilege Principle”. For example, a function is analysing the data from the database, then it should be given “read-only” access and nothing more than that.
Since serverless architecture has hundreds and thousands of functions, some organisations opt for giving the same security parameter and grant access.
In a system, where every function has access to everything, it is a serious security loophole.
Solution: There are no one-size fits all approach to this methodology. What we’ll advice is to figure out how you’re going to set security fences for each function in your application. Here are some of the resources which can be handy for you:
#4. Insufficient logging & monitoring
The majority of the breached organisations are notified by someone other than their own staff, according to Mandiant’s M-Trends 2017 report.
If you look at some of the most popular security breaches in the recent past, you will notice that none of them was reported in real time. This shows how insufficient monitoring exercises are prevailing amongst the administration.
While many Serverless vendors also provide logging and monitoring, these are not sufficient to provide full security to the system or trail back the data breach, if it happens. And hence, it is assertive for the development team to fabricate a logging logic which will suffice the need of their particular organisation.
Solution: Here are some of the report examples that you should generate on a regular basis for effective monitoring and logging, recommended by SANS Essential Categories of Log Reports. You can also set alarms on AWS CloudWatch so that suspicious activity on any of the below-mentioned report is notified to you.
- A report containing all login failures and successes by user, system and business unit.
- Login attempts (success & failures) to disabled/service/non-existing/default/suspended accounts.
- All logins after office hours or “off” hours.
- User authentication failures by the count of unique attempted systems.
- VPN authentication and other remote access logins.
- Privileged account access.
- Multiple login failures followed by success by the same account.
#5. Protect Your Keys
As soon as the applications grow, there arises the need for how to store the app’s secrets. These secrets include API keys, database credentials, encryption keys and other sensitive configuration settings.
One of the common practice is to enlist all of them in a single plain text document and store it with the software project. Such files can be easily compromised by hardcoding a script and if your software is hosted on the public library, anybody can access it.
Solution: We, at Simform, have kept a “Key Person” who is responsible to execute the following tasks to make sure so spillage of app’s secret.
- First and foremost, rotate the keys on a regular basis. Even if you’re hacked, this ensures that access to hackers is cut-off.
- Every developer, project, project and component should have separate keys.
#6. Vulnerable third-party dependencies
In an article titled, Gathering Weak npm Credentials, the author discusses how he obtained direct publish access to 14% of npm packages (including popular ones). The estimated number of packages potentially reachable through dependency chains is 54%.
One should keep in mind that even the most secure functions can become prone to attacks while using code from the third-party libraries.
Solution: Should you stop relying on those dependencies? No! Let’s discuss what you can do about it.
- Remove unnecessary dependencies, unused features, components, files and documentation. On client and server-side both, continuously monitor version of frameworks, libraries and their dependencies.
- Components should be obtained from official sources with signed packages to reduce the chance of malicious components.
- Create security patches for the older versions of libraries and its components.
Adnan Rahic, Developer Advocate at Dashbird says that the practice of serverless security tends to come with time. When you’re young and eager to build new software, security takes a back-seat ride. You build the app and hope you don’t get hacked. As a developer matures, this mindset changes.
First and foremost, step one is to learn general best practices in programming. Validating input is the most important, by far! Learning that input your system takes can be malicious regardless of who sends it. Parse it correctly, don’t cut corners here.
Once that step is done, the following crucial steps are permissions and secrets. Using AWS KMS is a must if you want to manage secrets and keys in a proper manner. While permissions can be managed with ease through IAM Users and Roles. Have in mind this is most crucial for S3 buckets. Secure your buckets immediately, because they are the most vulnerable part of the whole system.
Once you have these key steps down, the rest is just keeping it all working flawlessly. Here’s where monitoring tools come in to play. Then you need to learn how to utilize Dashbird, Cloudwatch, IOPipe, Thundra, or whichever tool that gets the job done for you.
What to do Next?
Security should be viewed as an integral part of the application development life cycle. Thought there are certain issues in serverless applications which makes security a bit difficult.
But these shouldn’t stop you from adopting this technology in your custom software development practices. Recently serverless ecosystem has matured and many 3rd party tools have come up to ease the security and monitoring pains.
Do you have any more suggestions? Feel free to comment or connect with me on Twitter @RohitAkiwatkar to take the discussion further.