As most of you know by now the repo has been down for a few days due to some security concerns. In this post I will try to explain why I took such measures and more importantly what I did to fix things. Let me start by saying that none of my personal info was leaked. The info shared was that of a repo server I was building. It was in no form secret or private data!

The incident:

Last Thursday the new 64-bit version of ArmA rolled out, and with it a load of new bugs and mod updates that we had hoped wouldn't be needed. As a result I decided on Friday that I would add a new repo server and set up some form of balancing between the 2. As I was in the process of configuring the new setup I had asked the one of the CM's to test some things for me. During this test 2 things happened. Firstly the CM's got so excited they all decided to start downloading, not what I asked for guys. This caused some frustration as I was trying to do some log tests. Second I noticed someone was actively trying to circumvent the blockades I had put in place during the testing.

Now this in itself I don't mind, if anything I applaud people who test our infrastructure. It helps me make things more resilient and allows you guys to enjoy a safe environment to share your stories in. However! I do not want some kid to go on non FK channels and boast about his escapades. Not only is this a gross breach of trust it is also against everything I stand for myself. What ever he found he should have reported to me, and no one else. Sadly for him he didn't really find anything useful. The info he gathered was something anyone could have found with some simple techniques. Hell I am going to share the same details with you guys right now.

The new repo setup:

In the past we had one repo server that would host a simple web server, this would allow arma3sync to connect via http which it used to download the mods. This box had reached its limits with regards to network speeds and caused painfully slow downloads. To expand this setup I would need a new server and a way to "balance" the load, however due to the amount of data we send out a true load balancer was not an option. Instead I have opted for redirect technique. What does this mean? It means we now have 3 web servers for the repo, one front end and 2 back end servers. In detail:

The front end (a3sync.fuckknows.eu) hosts a small webserver that takes an incoming request and redirects you to another url. It does this by picking a random number between 1 and 4, based on that number it knows what repo to redirect you to. This is not real load balancing, however it does allow us to share load over the 2 repo's. It also allows us to add more servers fairly quickly if we decide to expand in the future.

The back end hosts (repo1.fuckknows.eu & repo2.fuckknows.eu) are web servers that hold the actual repo files. They operate standalone from the front end and allow you to directly connect to should you so desire. They do nothing more but serve files that arma3sync downloads.

Al in all this is not really rocket science, it is a fairly simple setup that allows for easy management and  expansion.

The future:

What happened last Friday in detail? On Friday evening I changed the a3sync url, instead of pointing directly at repo1 it would now point at the front end server. I had disabled this web server while I did the initial setup and synchronization of the new repo server. During a short period of time I brought it online to perform some tests, in these 5 min tops several people connected and tried to download mods. Once I disabled the front end I expected load on the repo servers to drop again, however that did not happen. A quick look at the log files told me someone was bypassing the front end and when I blocked repo1 that same IP jumped to repo2. I did this testing several times and every time the same client IP popped up. That IP was also using a tool other then arma3sync to download with. I responded by banning that IP from the servers and thought nothing more of it until the screenshot posted by Linnet reached me.

What did I do to prevent this? Even tho I can not make the setup 100% obscured due to the way arma3sync works, I can limit the way's people connect. One of the things I can do is limit the allowed user agents to the web servers, anything other then arma3sync gets a 403 error. I have also increased the monitoring of the log files, repeated attempts to access the repo without arma3sync will get an IP ban. This ban will get propagated to all FK servers in a matter of minutes.

Again I encourage people to try and find issues with our infrastructure, however I do ask you do it in a thoughtful manner. I am not asking you to try and ddos our servers because any monkey can do that for a few bucks. I am asking you to find holes in our security and reporting them to me. If I find anyone exploiting our servers or sharing private info I will not hesitate to report you to the police and your ISP. Needles to say such an action will also result in a ban from our community.