Recommend
528 
 Thumb up
 Hide
262 Posts
1 , 2 , 3 , 4 , 5  Next »  [11] | 

Geekdo» Forums » News

Subject: BGG is being targeted by a distributed denial of service (DDoS) attack rss

Your Tags: Add tags
Popular Tags: 200_yahoo_boots [+] [View All]
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmbmbmb
As people who regularly use the site have noticed, we were having some serious issues delivering pages in a timely fashion most of last week. We have determined that these issues were caused by an ongoing DDoS attack. We have been able to stop the damage for now, but why the attack happened and if the attack will continue in a way that bypasses our current measures is unknown at this time.

Here is the story:
The attack started on Monday January 15. 2018 at 1am CST. The DDoS attack created an unprecedented amount of traffic directed at BoardGameGeek, however as it started in the middle of the night for the United States our servers handled it. Problems started around 9am EST when the BGG servers started adding the normal Monday morning traffic (one of our busiest times during the week) on top of the traffic from the DDoS attack. Reports from site members unable to connect and use BGG started coming in almost immediately. My first instinct was that we had a spike of visitors due to some mainstream media source - like getting mentioned on The Today Show or linked on a popular website - things that have caused similar traffic spikes in the past. When it was clear that this was not the case Daniel Karp (our Senior Software Developer) and I spent the rest of the day investigating the problem, looking for the usual culprits in the system for causing slowdowns. By Monday evening the problems started to subside but we still had not found the source of the problem and occasional "weirdness" like random disconnects, timeouts, etc were still occurring.

Tuesday morning was met with a return to the problems from Monday. By the end of that second day we were resigned to thinking we were just receiving a new level of daily visitors and running up against an internal limitation on our network. (Of course, we were receiving a new level of daily visitors....just not in the way we had thought). After we changed some settings in response to the new traffic level things improved a bit. Problem solved! Or so we had thought.

Wednesday started with BGG being completely non-responsive. We didn't know it at the time, but the changes we made on Tuesday enabled the botnet to wreak greater havoc on our systems. Unfortunately, prior engagements I had scheduled prevented me from being able to further diagnose the problem during the day. That night we tried disabling various systems that we suspected were causing the problems, but this offered no relief because the problem was coming from outside. Wednesday ended up being a loss of BGG services for almost the entire day.

I woke up Thursday extremely groggy from sleep deprivation due to the previous few late nights of trying to solve the BGG issues. Dan and I tried some new strategies, but nothing helped so Thursday afternoon we solicited Twitter for advice from the community. Anna Filina (@afilina) responded and got straight to helping us systematically go through our entire system stack and asking questions we hadn't considered. She really knew her stuff! With Anna’s help, by the end of Thursday we had identified some key data that lead us to uncover what we now know - that BGG has been the target of a sustained DDoS attack from a substantial botnet. Following the discovery of the true source of our problems the rest of the pieces started falling into place and we were able to devise a way to mitigate enough of the damage that was being caused to get the site close to normal operations.

We are still being targeted. As of the time of my writing this article there are still ~275,000 separate IP addresses sending us 2000+ page requests each minute in an effort to choke all the real users out from being able to access BGG.

I was initially hesitant to release this information. But ultimately I want you all to know what has been going on. We still don’t know who is doing this, or why. And we don’t know if they will change-up their efforts in response to this info being released. If anyone reading this has any information that can help us track down the cause of this attack please email us at Contact@boardgamegeek.com

So, long story short, I guess we’ve leveled up! BGG’s first DDoS attack! Sorry for the frustration it has caused, and be assured that we are doing everything possible to keep BGG up and running despite this attack on our site.
  • [+] Dice rolls
BoardGameCo
United States
Charlotte
North Carolina
flag msg tools
badge
Avatar
mbmbmbmbmb
Wow, that's intense. Congrats on reaching the big leagues and best of luck resolving.

43 
 Thumb up
0.05
 tip
 Hide
  • [+] Dice rolls
Tim Norris
United States
IN
flag msg tools
badge
Avatar
mbmbmbmbmb
My money is on the Russians!


I knew something was up. I kept having issues trying to gain access to the site all week. I mean, it's not a huge surprise given my Russian isn't very good, comrade.
25 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Alex Despres
United States
Arlington
Texas
flag msg tools
Saints are 4-4!
badge
Avatar
mbmbmbmbmb
This attack is being perpetrated by someone who literally hates fun!
75 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Corey Mayo
United States
Schertz
TX
flag msg tools
Have you paid your dues, Jack?
badge
Yes sir, the check is in the mail.
Avatar
mbmbmbmbmb
  • [+] Dice rolls
Jason Hurd
United States
Gordon
Nebraska
flag msg tools
Avatar
mbmbmbmbmb
AlexDespres wrote:
This attack is being perpetrated by someone who literally hates fun!


Or by some closet Monopoly fan who was actually traumatized by 3-day Monopoly games with house rules but suppressed the memories...!

On a serious note... I can't conceive of why someone would go to so much trouble in an attempt to take down a site like BoardGameGeek. (I mean... Come on. Do something meaningful with your life!)
65 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chad Sims
United States
Pennsylvania
flag msg tools
mbmbmbmbmb
I am sorry. I am betting it is my wife. She is really sick of seeing me on here.
  • [+] Dice rolls
The Steak Fairy
United States
Columbia
South Carolina
flag msg tools
Games? People still play games??
badge
Specious arguments are not proof of trollish intent.
Avatar
mbmbmbmbmb
Yeah, I think we all knew that the RSP swamp draining was going to have repercussions. Sorry for your troubles.
44 
 Thumb up
5.00
 tip
 Hide
  • [+] Dice rolls
CW Lumm
United States
Hampden
Maine
flag msg tools
badge
Avatar
mbmbmbmbmb
Aldie, given the traffic benchmarks you've been mentioning lately, I'm shocked it took this long, but congrats anyway, and thanks for keeping the hearth warm. I haven't noticed any problems with service, and it's probably a tribute to your team's diligence.
15 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chris Schenck
United States
Dayton
Ohio
flag msg tools
GO BUCKS!
badge
Stop touching me!
Avatar
mbmbmbmbmb
Fortress Ameritrash recruiting is getting really aggressive!
29 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ed G.
United States
Fort Wayne
Indiana
flag msg tools
Avatar
mbmbmbmbmb
Maybe it's the HATE backers. Or the HATE haters.
29 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
that Matt
United States
Ann Arbor
Michigan
flag msg tools
I'm a quitter. I come from a long line of quitters. It's amazing I'm here at all.
badge
I can feel bits of my brain falling away like wet cake.
Avatar
mbmbmbmbmb
Phew! I was worried the hamsters were getting restive.
10 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Crazy Adam
Canada
Toronto
Ontario
flag msg tools
Congratulations. You have just discovered the secret message.
badge
Please send your answer to Old Pink, care of the Funny Farm, Chalfont.
Avatar
mbmbmbmbmb
Aldie wrote:
...sending us 2000+ page requests each minute...


That could have been me hitting the 'Refresh' button while things were hinky. Sorry about that.
63 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
that Matt
United States
Ann Arbor
Michigan
flag msg tools
I'm a quitter. I come from a long line of quitters. It's amazing I'm here at all.
badge
I can feel bits of my brain falling away like wet cake.
Avatar
mbmbmbmbmb
Ursus_Major wrote:
Maybe it's the HATE backers. Or the HATE haters.

Or the HATE hater haters.
23 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Woo-Hoo Gamer
United States
West Virginia
flag msg tools
Looking for resources in how to grow in friendship, unity, and/or intimacy in marriage? - Ask me!
badge
More Woo-Hoo Gamer reviews coming throughout 2018!
Avatar
mbmbmbmbmb
BGG and DDoS...

Dogs and cats, living together — mass hysteria!!!!!
20 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Justin Steinberg
United States
North Miami Beach
Florida
flag msg tools
badge
Avatar
mbmbmbmbmb
Hopefully someone on here amongst all us gamers has an idea towards a solution for this. Good job and good luck on the work Aldie!
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Andrew Bonneau
United States
Springfield
Massachusetts
flag msg tools
Avatar
mbmbmbmbmb
Timing is too perfect, has to HATE radicals revolting.
4 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Dan Cristelli
United States
Milton
Vermont
flag msg tools
badge
Avatar
mb
Ouch. We went through this with a few sites at work, which led us to use Cloudflare as our CDN. They have been excellent at helping us prevent future attacks and mitigate any that might happen.
27 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
James C
United States
McLean
Virginia
flag msg tools
Avatar
I appreciate the update. I wish I had a clever joke, but I don't.
19 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Abraham Drucker
United States
San Francisco
California
flag msg tools
MOAR GAMES
badge
Damn Dirty Ape I Love You
Avatar
mbmbmbmbmb
I work at a major FI, and from what I've heard from the fraud team, this is the new normal.

Sucks though.
9 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
kat costa
United States
Paradise
California
flag msg tools
badge
Who dares disturb the Hobnob Goblin?
Avatar
mbmbmbmbmb
BGG's 18th birthday is right around now. You're all grown up, little site. Registering to vote, moving into a dorm, getting your first DDoS attack... the adventure is just beginning.

mbmbmbmbmbmbmb
  • [+] Dice rolls
Serious? Lee
United States
Coppell
Texas
flag msg tools
badge
Lost in thought.
Avatar
mbmbmbmbmb
Are those 200 boots upset about yahoo messenger again?

mb
61 
 Thumb up
3.27
 tip
 Hide
  • [+] Dice rolls
Drew
United States
Dallas
Georgia
flag msg tools
badge
Avatar
mbmbmbmbmb
I'm now waiting for someone to create the "I was there for DDoS attack" microbadge
  • [+] Dice rolls
Zed TwoEggs
United States
Texas
flag msg tools
Avatar
mb
freechinanow wrote:
Aldie wrote:
...sending us 2000+ page requests each minute...


That could have been me hitting the 'Refresh' button while things were hinky. Sorry about that.
yeah, as a werewolf, I can account for times like 3:01/4:01..
18 
 Thumb up
0.02
 tip
 Hide
  • [+] Dice rolls
Haoran Un
Australia
Marsfield
NSW
flag msg tools
mbmbmbmbmb
I don't know if any of this is helpful to you, but I'm a senior dev at a highly-trafficked media house, and these are the two most helpful things we've done to mitigate DDoS.

1. We set ourselves upbehind CloudFlare (but there are other competitors in this space, like Akamai, and AWS offer similar services), which protect against DDoS. They're able to detect identical traffic and squash a lot of stuff automagically.

2. We turned on rate-limiting, which limits any one IP address being able to hit us more than 10 times a second.

We pay for this service, but you could roll your own, more easily than you'd expect:
https://aws.amazon.com/blogs/security/how-to-configure-rate-...

Good luck!
  • [+] Dice rolls
1 , 2 , 3 , 4 , 5  Next »  [11] | 
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.