A Big Snafu and a Big Apology


Boy, did things go wrong over the weekend. An email test went bonkers, and sent many of our customers an email with an inaccurate message about their Boingo account being switched from monthly Unlimited to AsYouGo. Everyone received the same message about the service change with additional account information that was personalized to each individual recipient.

If you received this email message (or many, many messages), please disregard it. We’re terribly sorry for the confusion and disruption. If you want to check your account details, please visit https://my.boingo.com/selfcare and enter your username/password to securely access your information.

Many of you have taken to Twitter and Facebook to alert us of the problem and ask for clarification. Below are the most frequently asked questions and answers about our embarrassing email snafu. Hopefully, this clears up any confusion and gives you peace of mind. If you have additional questions, please leave us a post comment.

  • Have my account details been compromised?
    No, we would never send out highly confidential information via email communications. The email you received was personalized with some of your account details but did not include billing information (credit card number). In fact, our marketing email database does not store (or have access to) any credit card numbers.
  • Has my account switched from Boingo Unlimited to Boingo AsYouGo?
    No, there have been no changes to your account. The email you received about the service change is inaccurate. To check the status of your account, please visit our self-care portal: https://my.boingo.com/selfcare.
  • Does Boingo intend to eliminate the Boingo Unlimited monthly plan?
    No, we have no intention of eliminating our popular monthly plan. We have many customers who rely regularly on a Boingo connection; we would never force them into a pay-as-you-go-plan designed for those with ad-hoc needs.
  • What went awry with the email testing?
    We were testing out different email templates that – as you know – went into live production to many of you, indiscriminate of the type of Boingo service you have. If only the stars were aligned and the template you accidentally received was our “Thank You For Being a Loyal Customer” template instead of the one meant for those who actually want to change their Boingo account!
  • What is Boingo doing to ensure that this email mistake doesn’t happen again?
    We’ve no doubt learned some important lessons from this mistake. One of several measures we are taking is to heighten the wall between our test and production environments. We’re revisiting and re-engineering the sandboxes that these systems exist within to better protect the integrity of church (test) and state (prod).

We sincerely apologize for this big, fat error. Thank you for your understanding.

About Melody Demel

Melody is Senior Manager, PR for Boingo Wireless, where she is responsible for media relations, brand building and thought leadership initiatives. Melody is a die-hard Nebraska Cornhuskers fan and retired NCAA DI college field hockey player.
This entry was posted in Boingo and tagged , , . Bookmark the permalink.

81 Responses to A Big Snafu and a Big Apology

  1. Gary Massey says:

    You are incorrectly stating “….the email you received was personalized with some of your account details but did not include billing information.” That is not true, rather you have only sent out ‘partial billing information, as you have sent out my CC expirations data…and I am only a one time customer(so no need to store it for me it the first place)

    So it is a bit ingenous to say you ‘did not’ send billing information, rather – it was just not complete billing information. In any case, I am still a bit concerned that (1) you have sent my information out to others (not addressed in your ‘mea culpa’), and (2) that you are storing my billing information int he first place(now that you have shown how secure it is, especially).

    So is it possible to have my billing information removed and if not, why not? ….or do you store it forever…

  2. Mark Davies says:

    Yes, I got the email. Yes, I was confused. Yes, I appreciate you sending this email and publishing this post.

    Shit happens! It’s not the end of the world.

  3. Baochi says:

    Hi Gary,

    Thanks for giving us a chance to further clarify.

    You’re right: the email blast, in addition to communicating an inaccurate message about account changes, included basic account information that was personalized to each individual recipient. First, please rest assured that your specific account information was sent to you and only you. Second, providing credit card expiration in an email is commonly (and legally) included between a business and its customer because it is the one detail that does not compromise the person’s personal security. We would *never* send out emails providing credit card numbers and billing addresses.

    Please feel free to contact me directly re: your Boingo account: bnguyen@boingo.com. Again, I am so sorry for this disruption, and I aim to clear up any confusion our customers may have.

    Sincerely,
    Baochi Nguyen

  4. Baochi says:

    Thanks so much for your understanding and support, Mark. We sure hope this doesn’t signal the end of the world! 😉

  5. R Craig says:

    Although you seem to be making jokes on this – and while it may not be the end of the world, it is a bit of a concern that the information you have stored could be a compromise in respect to privacy acts that currently exist in some countries – especially as it relates to using customer data for purposes other than explicitly stated (especially when customers have opted out of email marketing campaigns.

  6. Twitter Comment


    We’re taking your questions re: our weekend email snafu. Please ask away. We’re here on twitter. FAQs: [link to post]

    Posted using Chat Catcher

  7. Pingback: Blowing Things Out of Porportion: Boingo Makes a Boo Boo » Techcraver.com | Craving Tech, Craving Life!

  8. Baochi says:

    Robert,

    I’m sorry, it was not my intention to be glib. I assure you we all here at Boingo are taking this matter very, very seriously.

    We definitely respect the laws of privacy and spam. To clarify, your specific account information was sent to you and only you. Credit card expiration data was included in the inaccurate email, but *not* your credit card number and billing information.

    You’re right that some of you received these errant emails despite being opted out of marketing email communications. To reiterate: nobody should have received these test emails. The reason why we store opted-out emails at all is that by law, we *must* contact a customer if something is amiss with their payment information.

    Again, please accept my apologies for coming across as flippant — I’m sorry. On behalf of Boingo, I’m also very sorry about the disruption this has caused.

    Best,
    Baochi

  9. Twitter Comment


    @LeighBlaize Totally agree. We did not intentionally send so many emails; we’re so sorry! Pls see FAQ for more details: [link to post]

    Posted using Chat Catcher

  10. Twitter Comment


    RT @boingo: Have a question about our email fiasco from the weekend? Pls read our FAQs: [link to post]. More questions? Tweet away; we

    Posted using Chat Catcher

  11. Twitter Comment


    Have a question about our email fiasco from the weekend? Pls read our FAQs: [link to post]. More questions? Tweet away; we’re here.

    Posted using Chat Catcher

  12. Twitter Comment


    @cnet We have an FAQ re: our weekend email fiasco: [link to post]. Pls note we have *no* intention of eliminating Boingo Unlimited.

    Posted using Chat Catcher

  13. Twitter Comment


    RT @boingo: We’re taking your questions re: our weekend email snafu. Please ask away. We’re here on twitter. FAQs: [link to post]

    Posted using Chat Catcher

  14. Twitter Comment


    @wlanmac You all definitely deserve as many “I am sorrys” as you need 🙂 We have FAQs posted on the email fiasco: [link to post]

    Posted using Chat Catcher

  15. Twitter Comment


    @innov8tr We assure you, no plans to change our monthly subscribers to an AsYouGo plan. Pls read our FAQs: [link to post]. So sorry!

    Posted using Chat Catcher

  16. Twitter Comment


    @WirelessWeek We’ve posted an FAQ about our weekend email fiasco: [link to post].

    Posted using Chat Catcher

  17. mlegoff2 says:

    This is unbelievable — I check my usage and I haven’t used Boingo since the weekend I was on business in South Africa — today, however, I have just been charged 4x times of $19.95 each time!!! My credit card company called me and I had to cancel my card in the event that this was identity theft — I now have a business trip to go on without a valid credit because of BOINGO!!!! Please cancel my subscription and I will continue to tell everyone I know —- just unbelievable!!

    M.

  18. Baochi says:

    Hi Michael,

    I’m reviewing your account right now and will send you a direct email as I want to ensure your confidentiality.

    We *do not* charge a customer’s credit card for As You Go accounts unless a connection has been made. That said, let’s rule out anything that may have gone awry.

    Baochi

  19. Richard Beirne says:

    While I appreciate the “clarification” of the error, I can’t help but wonder about your explanation. Would Starbucks put out an email about using dirt in their coffee? I doubt it. Why would you compose such a message to begin within you did not intend to make

  20. Baochi says:

    Hi Richard,

    Your comment got cut off. But let me address what I think you are saying:

    First, while this email error is not something we are proud of, it is hardly comparable to your analogy of Starbucks putting dirt in their coffee. Let me reiterate: there is nothing nefarious going on here. Nothing has changed about our Wi-Fi service and operations. Nothing has changed about our service subscriptions (except for our new one-hour plan via iTunes — yet another option for our customers).

    We’ll explain our mistake and repeat our apologies for as long as we have to, but I do not appreciate the suggestion that our service/product have been sullied — or that we have admitted to lowering the quality of our service. We are simply owning up to our mistake and trying to make ourselves as transparent as possible. That is why we have been so public about our apology.

    The email test template that you received (by error) was designed for customers who contact us requesting to change from a Boingo Unlimited monthly plan to the AsYouGo one. The idea is that an email confirmation — similar to the one we mistakenly sent out to our customers — would be automatically sent to the customer for final consent and approval. We have zero plans to switch our monthly customers over to a per-connection plan. Frankly, that would make little sense for many reasons I have enumerated here on the blog post as well as in my other comment replies. The long and the short of it is: Boingo monthly customers are among our most loyal ones. Why on earth would we want to (or plan to) dishonor that and risk losing their business as well as their valuable input?

  21. tony says:

    you should make amends by providing your local customers more than just an apology…throw in a few free months.!!!!

  22. "Really Happy Long Time Customer" says:

    Dear Boingo,

    Hey mistakes happen and you gotta laugh at the worst (or best) of them. I can only imagine the scrambling going on as the email was going out en masse. There seem to be a lot of angry people out there looking for an excuse to take it out on someone, perhaps sitting in an airport on their trusty connection checking emails, annoyed they got delayed another hour.

    I’d like to point out a few reasons I *LOVE* Boingo cause it sure sounds like you guys could use some kudos right about now…

    1) Your unlimited plan price has remained steady and low over the years. Heck, you could’ve accidentally sent out an email saying it was going up by ten bucks and I’d have said “ah, it’s worth it…” and not had a second thought.
    2) The number of locations your plan is available at has steadily grown to nearly every airport, hotel, Starbucks, and other imaginable hot spot so I almost never pay any more one-time usage fees to other providers.
    3) Customer service has always been great. I’ve had to talk to your folks a few times over the years and the professionals on the phone have always been exceptionally knowledgeable, professional, and courteous. My problems were always resolved competently and immediately.
    4) Despite an errant email, your marketing is slick and cool; you’ve got a great name, and your installable app to find hotspots is great.

    I’d also like to point out that you guys have handled this crisis beautifully, in stride, and a whole lot more calm, cool, and collected than some of these respondents handle some disruptions (okay, there wasn’t even a disruption!). I recall when ATA Airlines went out of business ovenright and codeshare partner Southwest Airlines got stuck with an immediate crisis for handling thousands of displaced Hawaiian travelers. Their VP of Marketing stayed online all weekend, night and day, responding via blog to all those angry upset passengers and customers. They did a beautiful job: responsive, honest, and forthright. While we miss our Hawaiian connection, I’m still a loyal customer because they did the right thing. And, frankly I think that’s the best thing any customer could expect.

    Thanks for staying great, taking the blows, and keeping things going. You rock!

    – “Really Happy Long Time Customer”

  23. Jeffrey T Carder says:

    Yes, I received the “weird” email on Saturday-not one time but 12X !

    My response was one of disbelief and “what the hell are they thinking”. Why anyone would want to pay as you go vs. monthly access is beyond me. One cross country trip in 2 or 3 airports and you see the falacy in “pay as you go”. Honestly, I was ready to cancel Boingo on the spot. I am so grateful the email was not legit and Boingo is still available on a monthly access plan. I have been very happy with the service and want to continue. Thanks for the explanation. Perhaps a late “April fools ?”

  24. Baochi says:

    Thank you 🙂 Your words mean a lot — in fact all the positive responses here on this blog have really been uplifting in the midst of our collective dismay. Thank you for remaining a loyal customer.

  25. Richard M. Davis says:

    I cannot understand the mentality (or lack thereof) of many people who delight in an opportunity to excoriate you for this error.

    I am another very happy, long-time Boingo customer who is extremely pleased with the service I have and do receive from your company. Thanks for all you do! Accidents happen and you were completely forthright in communicating the error.

    Don’t worry about a few Sad Sacks who are never happy unless they are reaming someone out.

    Thank You!

  26. Baochi says:

    Thank, Jeffrey. We’re really happy you gave us the chance to explain. Thank you for remaining a Boingo customer.

  27. Baochi says:

    Richard, thanks for taking the time to let us know you’re still a happy customer. It’s really nice to hear. 🙂

  28. David says:

    Thank you for sending out the email to explain, though it really should’ve been sooner. I’m being charged for unlimited service, but because of this delay I was functionally unable to use it from Saturday (when I got the email telling me not to connect or I’d be charged $7.95 a day) through Wednesday (when I got the second email which said the first email was a mistake). With all due respect, a prorated refund for that part of the month is in order for those of us who were thus affected.

    That said, I’m with Richard… mistakes happen. It’s silly to say that we should all get “a few free months” – we were only stiffed for a few days. And it’s ludicrous and childish for people to take out their frustrations on the service reps who came here to explain what’s going on. That’s just as low as yelling at the cashier because you think the price is too high.

    To Baochi et al, thank you for your kindness in talking to us about it. Many companies would have just left it at sending out the form letter, and never responding to the individual customers.

  29. Stephanie G. says:

    I was a recipient of these e-mails and thought it was some confusion as I just updated my account to unlimited service. While I had yet to follow up and see what happened, I appreciated the e-mail I just received explaining the error.

    It’s embarrassing to admit your mistakes – especially to a very large base of customers. However, I have a lot of respect for Boingo because you owned up to it and addressed the issue in a timely fashion. (as any respectable company should)

    Thanks!

  30. Darren Schreiber says:

    Hey there!

    I just thought I’d share a funny story with you. In 2006 I joined a business-to-business (NOT SPAM) email marketing company. We had a client who was very mad at us for messing up their email deliveries on a regular basis. I came to the rescue and offered to help get their troubles all sorted out and guarantee sending of their email. So what did I really end up doing? I accidentally sent a test message to 60,000 of their clients, and even worse, I was using my own personal email address in the reply-to field of the test! Needless to say, I got an awful lot of spam back and some very upset customer feedback.

    So I feel your pain, and I’ve been in your shoes – AND you are forgiven!

    It’s just email – nobody died. Better luck next time.

    – Darren

  31. Mark B says:

    I just wanted to take a second and say thank you for the quick post regarding the boo boo. Being a Net Admin, I have certainly been in that sort of position before (albeit not on such a large scale!). Sitting in front of a keyboard saying ‘uh oh’. It was confusing but am happy that all is well and nothing was compromised.

    Keep up the good work folks! And I personally do appreciate the good spirits in which the post was intended 😉

    -mark

  32. Jim F says:

    Hi Boingo,

    As a frequent global traveller and user of WiFi hotspots, I’m with “Really Happy Long Time Customer” on this issue. Thanks for quickly stepping up on this issue. Yes, I was a bit surprised and dismayed when I first receieved the slug of emails over the weekend, though I quickly realised that something must be wrong when they kept Boingoing into my inbox. Conspiracy theorists aside reference a forced move to “pay as you go” in the future, you’ve done a great job of growing the business, and more importantly of providing reasonably priced access for WiFi and wired users all over the world.

    Boingo hot-points:

    1. Great pricing and plans (anyone who has paid for WiFi in Australia or the UK will understand this point),
    2. A very large and growing list of hotspot locations,
    3. An ever expanding Roaming partner list,
    4. And most importantly, a SOLID, reliable, courteous and professional customer support staff (others could learn from you).

    Thanks for keeping quality service and customer support at the top of your priority list as you’ve grown the business. Keep up the good work. In the future, just make sure your programmers do a better job of testing template changes in a QA environment before migrating them to production. ;o)

    -Jim

  33. Sean Williams says:

    I am sorry, but I am still unclear how a professional company in this day and age “tests” with live customer data.

    Where you testing in a production environment? If so, that is sloppy and dangerous on many levels.

    Do your test environments contain live unscrubbed customer data? If so, then you are just asking for trouble. It is not a matter of if, but when. Perhaps this was the when.

    At the end of the day the question is, “Who has access to customer information at Boingo?”

    I’m not sure I know the answer to that based on your explanation. I can tell you this, it should not be the person testing emails. That is very poor quality control.

  34. Baochi says:

    Hi Sean,

    No problem. We’re happy to explain as much as necessary.

    You are right: we’ve learned for this experience that the wall between our test and production environments isn’t high enough. We’re revisiting and re-engineering the sandboxes that these systems exist within to better protect the integrity of church (test) and state (prod).

    That said, highly confidential information such as credit card numbers and passwords are not available — ever — in our email database. In fact, only a select number of support agents have access to our customer database so that they can assist a customer with account details. As a member of the marketing team, I can verify that none of us have access to your highly confidential data.

    The account information that you did receive was personalized and sent to you and you alone.

    Does this clarify things for you? I am more than happy to answer any remaining questions you have.

    Best regards,
    Baochi

  35. Twitter Comment


    @anattendantlord @Nonworkingmonke To explain: the crazy amount of emails you received was the result of a glitch: [link to post].

    Posted using Chat Catcher

  36. Baochi says:

    Thanks, Jim, for your kind words. We’re so happy to hear that you find your Boingo service valuable. You’re right to observe that we are constantly expanding our network footprint so that our customers have the option of connecting in as many places as possible. It’s also very rewarding to hear of your positive experience with our Customer Care team, especially since world class customer service is a key focus here at Boingo.

    As for the email test gone bonkers, there’s no doubt we’ve learned some very important lessons. You can bet that there have been some pretty intense discussions about re-engineering the sandboxes.

    Thank you for remaining a loyal Boingo customer. Please reach out to us anytime.

    Baochi

  37. Baochi says:

    Thank you, Mark, for your understanding. Let’s just say that at the moment we realized what had happened with the email glitch, we weren’t merely thinking “uh-oh.” 😉

  38. Baochi says:

    Yikes! What a story! Thank you for sharing. We’re glad you understand that this was all a very big mistake and nothing more. Thank you for your patronage. Hopefully, you’ll get only good news from us going forward.

  39. Baochi says:

    Stephanie, you’re right: we’re embarrassed. Even more so, we feel awful for this disruption to our loyal customers. It’s understandable that so much confusion and anger have ensued, and we’re focused on clarifying the situation as much as we need to.

    Thank you for your kind words and for being a loyal Boingo customer.

  40. Michael says:

    Please don’t bother to reply. I see a number of apologies here. I’m hoping lesson learned. I’ve enjoyed the Boingo service on a limited basis. I also got the email a BUNCH of times and my first thought was that someone else compromised your system – not that you compromised your own. Getting your apology email containing the phrase “the system decided….” was not encouraging – especially from the president of the company.

    You’ve definitely given me two reasons to think twice about conducting business over your service. 1) the problem and 2) your failure to recognize what goes through a guy’s mind when it’s clear that a system that I’ve sent important stuff through – is not in control of the people I pay to secure it.

    I’ll be looking for alternatives. There are many.

  41. David Landau says:

    Seeing the email didn’t give me security concerns, but it did make me doubt the intelligence of your company. I have subscribed for a couple of months and was about to use your service on a trip I was making when I decided not to go near the service, due to the email. It left the impression that you felt the flat rate was too advantageous to your customers, and so you were going to charge on a per-use basis as a way of hiking your revenues. I still suspect you have that thought in the backs of your minds. Meanwhile, I have yet to use the service successfully. In part this has to do with a casual attitude on my part; I just haven’t tried, or tried hard enough. But rest assured I will be watching you very closely, just as many others will after the spectacular errors you committed.

  42. Chris says:

    I was traveling when this snafu happened. I did try to call several times when I first got the email to your customer service number and was disconnected by your phone system each time (which was very frustrating as it heightened my concern as to what had happened). I did call the next day to hear that your opening message on your phone system had been changed to address the issue.

    I would also investigate why your phone system disconnected me as I am sure it did with many people.

    I have no issues with Boingo and consider it a great service as I am a business traveler. Just wanted to let you know about the phones.

    Thanks!

  43. Twitter Comment


    A Big Snafu and a Big Apology:[link to post]

    Posted using Chat Catcher

  44. Vincent says:

    Hi I also would like to say that I am one of those people that got the unintended email. I was travelling and I stop using my account and try to contact boingo. I did hold for almost 30 to 40 minutes on the weekend but nobody pickup the phone to clarify this.I also try to email back to ask for clarification. I got 4 or 5x this email. When I got home the day after I phone again and got the voicemail message indicating the email we got is an error. I understand that people makes mistakes and I respect that Boingo own up to their mistakes (which not much company does that unless there is no other choice ) See Toyota !!. Anyway I just wanted to say I will still continue to use boingo as I respect their business model and what they are doing. But if there is some compensation that can be given to affected customers that would be even better !!

    Thanks again for providing us the frequent flyer the connection we need to work / chat effeciently !!

  45. Shelley Beck says:

    I am a marketing chick and always consider such situations from a marketing/image/PR/customer relations standpoint and I have to say that your apology was what public apologies from companies and some individuals should be: openly admit the mistake, acknowledge the issue(s) it may have caused, a sincere offer of your best efforts to resolve said issues…all with a touch of humility and/or humor.

    I’d give you an A- (only because the ‘data dump’ reference was avoidable jarigon)

    Auto makers, politicians, CEO’s, philandering husbands: Take note!

  46. Twitter Comment


    Crisis communications done right. [link to post]

    Posted using Chat Catcher

  47. Twitter Comment


    A Big Snafu and a Big Apology [link to post]

    Posted using Chat Catcher

  48. Michael Walter says:

    Let me start by saying this happens and it certainly won’t impact my future decisions to use boingo. That said what have we learned? Being a technologist (CTO) I can only conceive of 2 scenarios that would have allowed this to happen.

    1) You allow your dev/marketing team to test in you production computing environment.
    2) You allow your test environment to access the internet, don’t obfuscate your customer data in the test environment and/or use the same environment for testing and production computing.

    The first scenario is a common mistake and should lead to better separation of duties (you are PCI DSS compliant are you not?) and moving these types of tests to a non exposed environment. The second scenario should lead to a reduced exposure for your test environment and better security for the data in it (you didn’t send out security sensitive data but please tell us that data doesn’t exist in a looser security environment in a non-obfuscated form).

    So with that being said, my question is what is Boingo doing from the technical perspective to ensure that this type of scenario (or worse) is impossible or at least far more difficult to accomplish in the future?

    Thanks,

    Michael

  49. E Shelton says:

    I got the “test” email. But I didn’t get a subsequent email explaining the problem. However, when I called the customer support #, the outgoing message was great. Thanks for that.

  50. Dean Bain says:

    Thanks for the mistake you made. I implement systems and you’ve provided me with an example that I will use on the importance of segregating test and live environments. Again thanks for the mistake