A Big Snafu and a Big Apology


Boy, did things go wrong over the weekend. An email test went bonkers, and sent many of our customers an email with an inaccurate message about their Boingo account being switched from monthly Unlimited to AsYouGo. Everyone received the same message about the service change with additional account information that was personalized to each individual recipient.

If you received this email message (or many, many messages), please disregard it. We’re terribly sorry for the confusion and disruption. If you want to check your account details, please visit https://my.boingo.com/selfcare and enter your username/password to securely access your information.

Many of you have taken to Twitter and Facebook to alert us of the problem and ask for clarification. Below are the most frequently asked questions and answers about our embarrassing email snafu. Hopefully, this clears up any confusion and gives you peace of mind. If you have additional questions, please leave us a post comment.

  • Have my account details been compromised?
    No, we would never send out highly confidential information via email communications. The email you received was personalized with some of your account details but did not include billing information (credit card number). In fact, our marketing email database does not store (or have access to) any credit card numbers.
  • Has my account switched from Boingo Unlimited to Boingo AsYouGo?
    No, there have been no changes to your account. The email you received about the service change is inaccurate. To check the status of your account, please visit our self-care portal: https://my.boingo.com/selfcare.
  • Does Boingo intend to eliminate the Boingo Unlimited monthly plan?
    No, we have no intention of eliminating our popular monthly plan. We have many customers who rely regularly on a Boingo connection; we would never force them into a pay-as-you-go-plan designed for those with ad-hoc needs.
  • What went awry with the email testing?
    We were testing out different email templates that – as you know – went into live production to many of you, indiscriminate of the type of Boingo service you have. If only the stars were aligned and the template you accidentally received was our “Thank You For Being a Loyal Customer” template instead of the one meant for those who actually want to change their Boingo account!
  • What is Boingo doing to ensure that this email mistake doesn’t happen again?
    We’ve no doubt learned some important lessons from this mistake. One of several measures we are taking is to heighten the wall between our test and production environments. We’re revisiting and re-engineering the sandboxes that these systems exist within to better protect the integrity of church (test) and state (prod).

We sincerely apologize for this big, fat error. Thank you for your understanding.

This entry was posted in Boingo and tagged , , . Bookmark the permalink.

81 Responses to A Big Snafu and a Big Apology

  1. Dave G says:

    Wow. A lot of angry people out there. Try focusing your rage on something important, people. You got an email with no sensitive data. You get virus embedded pranks fired at you daily. Leave these innovators alone. The giant word “TEST” in the subject line led me to calmly check my account using the excellent self service tools offered, see nothing had changed, and figured it was an error. I see lots of these things. No biggie. I say thanks for being pioneers in convenient WIFI access.

    You want to win more friends from your fans who spend a lot of time in the airport? Negotiate a Boingo user rate/”roaming” plan for GOGO inflight wireless. Next to the huge savings I reap for my company as a Boingo user (vs. $12 -$25 PER NIGHT for subpar, non-VPN friendly hotel proprietary connections), GOGO is the biggest productivity gain I’ve had in a long time. Not losing contact with my customers for 3-5 hours every couple of days reduces backlogs like crazy. Seems like a nice complement to your reach.

  2. Baochi says:

    Hi Dave,

    Thanks so much for your understanding — much appreciated.

    As for Boingo Wi-Fi on planes: we’ve been trying to work with the various in-flight Wi-Fi providers on securing a roaming deal so Boingo customers can connect with their Boingo accounts. Still working hard to negotiate but no luck yet. We’re optimistic that something will work out…hopefully sooner rather than later. Stay tuned and keep the feedback coming.

  3. Jim says:

    Happy Long-Time Boingo User

    I too received the erroneous messages, seven or eight all told, but am puzzled by some of the above comments. Not one of the mesages I received had any accurate customer information, not even the expiration dates for my CC. I initially assumed SPAM, and quickly checked with Boingo, and was immediately informed by a voice announcement about the error. I now assume that the supposed customer data was generated by the system that was testing the messages, and that no real customer data was used other than name and email address. Certainly possible to receive a message with your actual CC dates, but such evidence as I have makes it seem unlikely that such an occurance would be anything but a coincidence. Regardless, thanks to Boingo for keeping us informed, and for recently lowering the rates for the unlimited monthly account (and making it easy to switch to the lower rates!).
    Jim

  4. Kevin A says:

    I don’t know about other people, but I kinda figured out there was a snafu when I received 5 emails in a row and all had the subject line “TEST Ad Hoc Template” in the subject line. Things happen, and I thank Boingo for following up with us.

  5. Baochi says:

    Hi David: again, very very sorry for the spate of emails you received. I understand the *misconception* that we are forcing our unlimited customers to a pay-as-you-go plan. But that is not the case: we have NO plans to eliminate Boingo Unlimited and/or force our customers into a plan they did not opt for. Our unlimited customers are extremely valuable to us for a number of reasons, including: they are our power users and thus our most loyal customers, and their extensive experience with Boingo enables them to give us very insightful feedback on our service. If you look back over the last few years, you will see that our pricing has gone down — not up. As you keep your eye on us, you will see that we mean what we are saying. Thanks.

  6. Baochi says:

    Thank you for reaching out to us, Chris. Again, very sorry for the email disruption. These last few days, our customer care center — which usually averages only a 5-minute wait — has been inundated with calls and emails as a result of this email snafu. So like you, several of our customers were disconnected and for that, I’m again really sorry. Regardless, I will pass along your feedback to our customer care team. Thanks so much for your understanding.

  7. Baochi says:

    Vincent: thanks for your understanding. I am reaching out to you via email.

  8. Vincent says:

    Thanks again for your offer is greatly appreciated and not expected at all !!! Either way like I have said before I will continue to be a loyal customer of boingo. I hope a deal can be reach with GOGO that will really be something I will really looking forward too as I am a frequent flyer and this will really increase my productivity.

  9. Nikki N. says:

    I received the erroneous email and immediately sent a response to support@boingo.com about it four days ago. I never received a response, not even an automated “we got your email!” response. I was about to send another, more frustrated, email about it when I decided to just check your site instead. I read the apology and I understand there was a problem, but what I don’t understand is why I wouldn’t have gotten some sort of response after FOUR DAYS from your supposed 24/7 support line?

    Coming from someone who works on a help desk this seems a bit unacceptable to me. I imagine there are other people out there as well still wondering why they got that email and haven’t stumbled onto your site.

  10. Kirk says:

    Dissapointing to not receive the apology/explanation email until almost 5 days after the original email. I was on the road for business when I received the initial email, and I only received 1 of them. I expect that myself (like many others) didn’t blink in changing service providers quickly in order to keep our traveling operations running smoothly. It only took me about 12 minutes to uninstall the Boingo desktop tool and sign up with another provider. I can’t imagine I was the only one. I’m sure that for many folks who utilize Boingo simply to surf the web when at the local coffee shop, they were interested in getting to the bottom of the email, possibly by calling customer service. I wasn’t interested in calling customer service to voice my conmplaints about a plan change, I doubt I was alone. Quicker and easier to just find a new one.

    It sounds like lessons have been learned from the top down from this incident. Probably a good reminder for many of us that a single stroke of the keys can have significant effects on our businesses. You often won’t have a second chance at easily capturing those opportunities again. Awhile back, I got in the habit of always adding a destination email address after I finished composing my emails. Maybe we can all embrace it.

  11. Baochi says:

    Kirk: I am sorry to have lost you as a customer because of this big email mistake. We’ve been trying to communicate as quickly as possible: our social media channels (Twitter, Facebook, Blog) were updated immediately on Saturday when the errant emails went out, and the front page of our site was updated with a note from our CEO. Before sending out an apology email to you all, we needed to first diagnose the emailer problem, since it was what started the trouble in the first place. That’s why it took us a few days to get the apology emails out to you all. Sorry again for the disruption, and we hope you’ll consider us in the future.

  12. Chris says:

    One has to wonder why production data ever was migrated into the test environment. Anything with valid customer information, even scrambled, should not be utilized in your test regions. Valid email address can still e-mail. A big no no if you are following formal testing procedures. Keep Prod and Test separate, logical and physical, including the data. Its the only way to avoid these embarrassing moments. Stuff happens, but creating a bed of test data to utilize should help everyone sleep better. That way the “system can’t decide” to communicate to your valued customers without your consent. I hope you gave that system a timeout.

    Seed the test data with sufficient test accounts so you can view the e-mails, but not your customers. Conditioning that test data just needs to be part of your test plans.

    Good job on the communication. I’m glad my account wasn’t compromised, which was what I first assumed upon receiving the couple e-mails from you on the week end.

    Better luck next test.

  13. B Fredricks says:

    Received the email…..you have to be kidding this was an “accident” or “testing” snafu…..I’m not aware of too many tests being conducted that would include the narrative it did. Appreciate the apology, but the verdict is out if the “story” is true…….

  14. Tor Vincent says:

    Give me a break people! He who is without sin let him cast the first stone. I am so damn tired of people bitchin and moaning about an “accident”. Bloody hell… do you know what forgiveness is? How many times does someone have to apologize. They have clearly explained the issue… and their plan for remediation. Quit putting Boingo on the cross!

    No I don’t use Boingo. I came here from a Twitter link because I wanted to see what happened. To err is human. Would hate to think how each of you arrogant pricks would feel when you made a mistake and nailed you to a cross. Did you mean to do it? GIVE IT A REST and be thankful that Boingo busted their ass to fix and inform unlike so many other companies that just sweep it under the carpet. You know who you are.

    ***Exiting Soap Box***

    Great communication Boingo. Not being a customer and seeing the great over communication you supplied and the excellent support even on the social networks makes me want to shout from the mountain tops…. USE BOINGO…. THEY CARE! Congratulations… I think you just earned my respect and business. Cheers to YOU!

  15. Lou Wyandt says:

    Just wanted to say you have a great site and thanks for posting!

  16. Ewan says:

    Well done Boingo for coming clean on the issue, I look forward to continuing to use the service when I travel.

    Ewan

  17. Amanda says:

    I also received the error emails and figured (intelligently, unlike some people) that they were in fact errors. Thank you for actually admitting to the mistake and coming clean.. I wish most companies did the same. I hope nobody got fired over it.. mistakes happen.

  18. Baochi says:

    Hi there: I totally understand the confusion and want to make sure to clarify: the “story” about eliminating Unlimited is NOT true. That email was intended for customers who do want to cancel or change their account to AsYoGo. Unfortunately, the test email went out to an indiscriminate pool of our customers. We’re very, very sorry for the disruption and hope you will see in time that we have no intention of forcing our loyal monthly customers into a plan designed for ad hoc use.

  19. Pingback: The art (or not) of the apology // Brett's Waste Blog

  20. Baochi says:

    Nikki: I’m sorry you didn’t receive an explanatory email from us for four days. We’ve been trying to communicate as quickly as possible: our social media channels (Twitter, Facebook, Blog) were updated immediately on Saturday when the errant emails went out, and the front page of our site was updated with a note from our CEO. Before sending out an apology email to you all, we needed to first diagnose the emailer problem, since it was what started the trouble in the first place. That’s why it took us a few days to get the apology emails out to you all. Additionally, our call center lines and email inboxes have also been backed up. Definitely not an excuse, but just wanted to explain to you the delay in getting back to you directly. Again, we’re so sorry for this disruption.

  21. Baochi says:

    Michael: we are PCI DSS compliant, and we are actively addressing the exact issues you raised. Thanks so much for remaining a Boingo customer.

  22. Tara says:

    This stuff happens and it is not a big deal. However, I think you should apolgize for your mis representation of how your unlimited service works. It is bogus that is per airline and you don’t do a good job of making that know prior to subscribing. THat is where you should focus your apologies.

  23. George Curnew says:

    I just want my name and any information about me purged from your database. I doubt I’m the only former customer that has ZERO level of comfort that Boingo has its IT act together. Even before this incident, there was no chance I’d ever use Boingo again; my experience with you guys was a one-time use in a U.S. airport at some ludicrous rate for an hour of pretty lousy service.

    I suggest sending everyone who got the email (you do know who you sent it to, right?) something that asks if they want their info to be purged.

  24. Baochi says:

    Tara: I’m really sorry you feel like we misrepresent our Unlimited service. We’re very open about which locations we support through our location directory, which is available on our website: http://boingo.jiwire.com/.

  25. Matt says:

    My actual question is: Why do you keep my information in the first place? I closed this account in June 2009 (because I was billed twice for a only one one-time access and reimbursed only after I complained about it). This is the same irresponisible handling of data as we find it in so many places (facebook etc).

  26. Mike C. says:

    My unlimited account was in fact cancelled when this happened and I have contacted support today to see why.

    I will wait and see!

  27. Baochi says:

    Hi Mike,

    Send me your username and I will investigate: bnguyen@boingo.com.

    Best,
    Baochi

  28. Baochi says:

    Hi Matt: by law, we must keep records of our customers, past and present, for a few years. We assure you that your personal information was not sent to anyone but you. In addition, the errant email you received did not include highly confidential information. Again, we are so sorry for the inconvenience.

    Best,
    Baochi

  29. Pingback: Social Axcess » Five Key Topics Every Enterprise Blogs Must Have

  30. Pingback: Case study: Boingo social media crisis - Colt Interactive

  31. Pingback: The Makings of a Great Social Media Crisis Plan: Boingo