RSS
 

Archive for the ‘Server Management’ Category

How to Hide your Servers Origin IP Address

23 Apr 2018

Overview

One project I maintain is frequently targetted by DDoS attacks. The power behind each attack ranges from weak attacks (100 Mbps) to very strong attacks (100+ Gbps). This project is essentially a side project and as a result I cannot afford to set up expensive DDoS mitigation solutions with failover servers. I suspect there are many that also face the same situation. A long time ago I decided to do everything I could to hide my servers origin IP in hopes of preventing my cheap web server from being bombarded with malicious traffic. The results from these changes have been fantastic!

Below is a summary of all the changes I made to hide my servers origin IP address for this project.

CloudFlare

Cloudflare Logo

One of the first changes I made was moving over my website to CloudFlare. CloudFlare provided me with many benefits (and all for free!). Not only did CloudFlare mask my servers origin IP by routing requests from users to my server, they also reduced my overall bandwidth costs and provided fast file caching.

The primary reason I signed up to CloudFlare was to mask my servers origin IP. Instead of pointing my domain to my origin server, I instead point my domain towards CloudFlare’s servers. CloudFlare then forwards user request to my origin server and my origin server replies to CloudFlare with a response. Finally, CloudFlare sends the response back to the user. During the initial connection between the user and CloudFlare, CloudFlare can handle malicious traffic it detects and may present users with a CAPTCHA to verify they are human. If the checks fail the request is denied and the traffic never reaches your origin server.

CloudFlare Overview

Firewall Whitelisting

Now that CloudFlare is set up and working, our web server should only be contacted by CloudFlare (and perhaps other services you use). SSH into your server and configure your firewall to only allow CloudFlare IP’s inbound/outbound access. CloudFlare’s IP ranges are available here and rarely change.

HTTPs

Another change you should definitely make is using HTTPs only for the connection between your server and CloudFlare. Make sure you use Full (strict) mode under the Crypto > SSL section in the CloudFlare dashboard. If you didn’t already know, you can get a SSL certificate for free and automate renewal easily using Let’s Encrypt. So there is no real reason to not make this change.

Cloudflare Crypto SSL Dashboard Very Strict

DNS Record

DNS Records can leak the origin IP if the origin IP appears in any records. In my case, I had a A  record called server pointing to the origin IP which had to be changed.

Mail Server

Hosting mail services on the same box as your web server is another problem. Mail servers will not trust mail that fails to pass certain anti-spam checks. Emails that are missing various headers like  X-Originating-IP  will likely not be trusted. Servers that don’t have the correct mail DNS records pointing to the origin server will not be trusted. Servers that fail a reverse DNS lookup will not be trusted. Typically a mail server will lookup the name of your mail server mail.example.com, retrieve its IP and compare it to the connection IP, then perform a reverse DNS lookup on that IP (using the PTR record) and ensure it points to the mail server name.

One solution here is to not run a mail server.
Another is to send your emails knowing they will be untrusted by mail servers (meaning they will always be sent to the spam folder or might not reach their destination at all).

However, if you need proper trusted mail services you must use an external mail service. Unfortunately, almost every external mail provider will include your origin servers IP address in the  X-Originating-IP header field to deter people from using their services to deliver spam. This means an attacker simply needs to get your web server to send an email (via the external mail provider) to retrieve the origin IP by inspecting the RAW email message.

Raw Email Message Origin IP

After some research, I discovered that Amazon Simple Email Service (or Amazon SES) did not include the  X-Originating-IP header field in outgoing mail. However, to ensure email services were not being used for spam, Amazon required you to verify your domain. Furthermore, Amazon monitor your bounce back rate and your complaint rate. If either metric creeps too high then your account status may be in jeopardy which may lead to service termination.

Outbound Requests

A very common issue is that may websites make outgoing requests to user provided endpoints. Making a request to a server you deem safe is fine (i.e. trusted APIs, another server you own) but user input SHOULD NEVER BE TRUSTED! In my case, users on my website could create an account and upload an avatar. Alternatively, they could enter the URL of an image on the web and my web server would fetch the image, store in locally and then use it as that users avatars. The problem here is that its not CloudFlare making the outgoing request, its your origin server. Therefore, an attacker could set up their own web server, enter a link to an image they are hosting, then look at their web server logs to see which IP just accessed their image. That IP is your origin server IP and is now leaked!

The simplest solution here was to not provide the alternative method.

Leaking Client IPs

One additional consideration to make is protecting the IP of your users. If users on your website can make posts and share content that is retrieved separately by the client over HTTP rather than sent by the origin server in the RAW HTML (i.e. images specified in an <img> tag), then they can steal the IP of any user that views that content. A user will load a page with the content on it, they will receive the RAW HTML payload from CloudFlare, next their browser will fetch any external references (i.e. to images in <img> tags) using the clients connection, thus leaking the clients IP to the external server hosting the content.

One solution to this is to use an image proxy which routes all images through your web server.
I ended up using camo as my image proxy which is open source and very easy to set up.

With an image proxy, all links to images are replaced with link that looks like this:

The image URL is then decoded (using a shared secret key) to retrieve the original external image which is then provided to the client via the origin server.

This protects the clients IP but we’ve just leaked the origins server IP.
To solve this, you will need to use a whitelist of trusted external websites that can be used to source images.
For example, you may only allow images be posted from imgur.com.
This may be a slight inconvenience to users but it protects both their IP and the servers origin IP.

Software Vulnerabilities

This is one is pretty obvious but stay up to date with security patches for any software you use and for the box itself.

Changing your Origin IP One Final Time

After making all the above changes, it will likely be necessary to change your origin IP by requesting a new IP from your server provider. This is because there are many websites out there that store historical DNS record information. Some in particular target CloudFlare protected websites. This change is especially required if you were already being attacked as the attackers will already have the servers origin IP and thus your new leakage prevention solution won’t matter.

Keep in mind that clever attacks may still find your new origin server IP using your old origin server IP. If your webserver displays your website when accessed directly by IP, an attacker could assume a newly allocated IP might be on the same /24 or /16 subnet. It would take no time at all to write a web scraper in Python to iterate over all IP’s in that range to search for a server that is responding to HTTP/HTTPS requests and has your logo (or any other identifying information) in the HTML source code. One way to help stop this is by disabling direct IP access to your website or by displaying a simple forbidden page for all direct IP accesses.

Impact

After making all of the above changed, I noticed a massive improvement. Prior to making the changes I received one medium size attack every 3 months. These attacks would take down my website and leave it down for the majority of the day until the attack stopped. After making these changes, I have received almost no significant attacks over the past 3 years. I did receive 1 very large attack that caused some disturbance but the web server still remained online throughout the attack. The best part is I’m only paying $20 for my server a month which is amazing given its overall traffic and the frequency of attacks. On the other hand, a proper DDoS mitigation solution with failover servers would cost hundreds a month and is only really viable for business solutions rather than side projects.

Overall, I’m very happy with the results.

 
1 Comment

Posted in Server Management

 

Fix PHP 500 Internal Server Error when using date()

31 May 2016

For a particular project I was working on, I set up a website with a really simple script that used the php date() function. However, the use of the date() function resulted in a 500 internal server error. After some testing, I determined this was caused because of a configuration in php.ini. There was an invalid value for the date.timezone setting.

This is what my php.ini file looked like:

Now if date.timezone is set to an invalid timezone (one that does not exist) then it produces a 500 internal server error. However, Australia/Sydney is a valid timezone and should not result in that behaviour.

I then wrote a little script to see what timezones PHP was recognizing:

This resulted in the following output:

Note that Australia/Sydney is missing from the list. Finally I realised that the timezone file for Australia/Sydney was missing on the box (for some unknown reason).

I was using Centos 6.7 and needed to obtain and replace the missing Sydney timezone file so it was available here:

After making this change and restarting php and apache, the issue was solved!

 
No Comments

Posted in Server Management

 

Fixing Mixed Content warnings using CRONjobs

02 Dec 2015

So if you, like myself, have a HTTPs only website you may have noticed that your green bar, green label or security shield (image below) disappears if your webpage fetches an image from another website over HTTP and not HTTPS.

Chrome Secure Green Lock

HTTPS Secure Chrome Lock

Now this isn’t an issue in itself which is why all modern browsers only produce a small warning in the console.

Console Warning Message

Mixed Content Message in Console

Unfortunately, most browsers also remove the secure label which is unfortunate as most business websites want to display their secure logo for customer reassurance reasons if nothing else. Personally, I just think it looks cool so I like to keep it green.

Easy (Obvious) Solution

The obvious solution is obviously  to host the image yourself or move the image to a website that supports HTTPS (like imgur.com).

However! The real issue is when the image is being produced by some API and you do not have access to the source code of the script producing the image.

Cronjobs!

Okay so in this case, we are using some API which regular updates an image.
We want our CRON job to run daily (or whenever required based on your needs) and to download that image and store it locally, so that your website has access to it (over HTTPs!)
This makes all the mixed content errors disappear.
I came up with the following bash script in my case:

Simply save this as some_name.sh and add a CRONjob to run the script at some interval (like daily at 3AM when your server isn’t being used much).

Check out this post on how to make CRONjobs:
http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/

 
No Comments

Posted in Server Management

 

How to get an A+ on Qualy’s SSL Labs Server Test (Apache)

27 Nov 2015

The SSL Server Test by Qualy’s SSL Labs is an easy way to determine how secure your SSL set up actually is.

You can run the test at: https://www.ssllabs.com/ssltest/

This is the score for this domain/server:

MoBeigi.com SSL Server Test

How to get an A+ score on an Apache HTTP server

The default Apache configuration for websites running HTTPs leaves your set up vulnerable to a variety of attacks. So you will need to modify the configuration file for your SSL enabled website.

First navigate to the httpd.conf file and open it in your favourite text editor. In my case this file was located at: /etc/httpd/conf

Navigate to the VirtualHost line that corresponds to the SSL enabled website.

Here is where we add all on configuration options. I’m not going to explain what each option does but do research that if it interests you.

The important configuration options we set are: SSLProtocol (disable SSLv2 and SSLv3), SSLHonorCipherOrder (Beast attack), SSLCipherSuite (support wide range of secure protocols) and HTTP Strict Transport Security. Obviously, replace the placeholder paths, server name (example.com) and file names.

Thats it!
Now restart your apache server ( service httpd restart ) and run the test.

 
No Comments

Posted in Server Management