O2 Wireless Box III / Thomson TG585n Security Flaws

Dear O2, Please fix my router...

Brief summary

28 August 2009, plus major update on 10 April 2010: The O2 Wireless Box III (a customised Thomson TG585n router) is an ADSL modem and wireless router used by O2 Broadband customers. Two weeks ago, I discovered a serious security vulnerability that allows remote attackers to access a home user's private network and view/change settings on the router.

I have tried to tell O2 about this problem several times, but they insist that the O2 Wireless Box III is secured to a standard that is acceptable for home use.

I strongly disagree, which is why I have published this article — how can a wireless router's security be deemed acceptable when it allows a remote attacker to gain unfettered access to your home network?

In particular, a remote attacker can:

Motivation for this Research

I have been an O2 Broadband customer for a few months now. I was looking through O2's terms and conditions for O2 Home Broadband and noticed that one of my obligations was to:

6. take whatever steps you consider necessary to backup and protect any data on your IT systems, including taking additional measures over and above any measures included within the Equipment to protect your IT systems from viruses, trojans, malware and other threats to your infrastructure;

("the Equipment" refers to the O2 Wireless Box III in this instance)

So, I must take whatever steps I consider necessary to protect my own computers from threats? Fair enough. But to do that, I really ought to check that the O2 Wireless Box III router they have given me is secure...

Seeking Permission to Test the Router

As 'my' O2 Wireless Box III is merely leased to me during my broadband contract, it remains the property of O2. Just to be on the safe side, I emailed O2 Broadband customer services to check that I would be allowed to test the box for security problems.

I made it clear that I was not intending to physically open the box, or do anything that would leave it damaged (they told me I would have to pay �50 if I broke it). After clarifying the nature of the tests I intended to carry out, I was given the all clear to go ahead.

The Discovery

I had a metaphorical poke around with the box and found that a number of good practices had been employed to defend against cross-site request forgery (CSRF) attacks. In particular, a nonce is used to ensure that all configuration changes originate from the router's own HTTP configuration interface. However, after a bit more poking, I found a design flaw which allows this protection to be bypassed.

This flaw allows remote attackers to take almost full control of the router, including stealing the wireless encryption key (even if the most advanced WPA2 setting was enabled) and forwarding external ports to internal IP addresses.

I have made several attempts to contact O2 about the vulnerabilities, but no progress has been made; in fact, they insist that the O2 Wireless Box III is "secured to a standard that is acceptable for home use". Unbelievable!

Trying (and Failing) to Report the Problem to O2

I considered this security problem very serious; in essence, it can allow anybody in the world to gain access to my own home network. I therefore decided to contact O2 to report the problem and to ask for my router to be fixed or replaced.

O2 do not make it easy to report security problems at all. I could not find any relevant email addresses or phone numbers on their website.

I phoned the O2 Broadband customer services number and reported the problem to a chap who was very helpful, although he did not fully understand the problem I was describing. He confirmed that my O2 Wireless Box III had the latest firmware update (8.2.L.0 - apparently, O2 is able to remotely apply firmware updates to all of their Wireless Boxes) and he arranged for someone technical to call me back to discuss the issue the following evening. Nobody called me.

After waiting a week for this phone call, I decided it was never going to happen, so I dropped a quick email to customer services as well (unnecessary text and headers omitted for brevity):

From: Paul Mutton
To: O2 Broadband

I believe the O2 Wireless Box III is vulnerable to cross-site request forgery attacks so I would like to have it replaced with a different router. I phoned about this a week ago, as it seemed rather serious to me, and I was told that someone technical would call me back, but nobody has called me.

Sadly, O2's only response was to offer me a Wireless Box II as a replacement. I didn't fancy downgrading my router to a slower one which is only available on cheaper contracts, and they said they won't be able to pay for another router on top of this. Incidentally, I subsequently discovered that the O2 Wireless Box II is also affected by the same problem. O2's reply made no mention whatsoever of the security problem I had reported.

So I sent another email:

From: Paul Mutton
To: O2 Broadband

I don't want a Wireless Box II, as it does not support N wireless. I really don't think I should have to pay for a replacement router if the one you have provided me with is so insecure. One week ago, I phoned you to inform you about a serious security problem I found with the Wireless Box III, and I wouldn't be at all surprised if this affects every O2 Broadband customer who uses one. I was told that someone would phone me back to find out more and nothing has happened. Who can I report security problems to, because this route clearly isn't working!

(Note that this email was sent in plain text; I have highlighted the last bit just to point it out. Please excuse using "who" instead of "whom" :-)).

O2's response to this email astounded me (irrelevant bits removed):

From: O2 Broadband
To: Paul Mutton

As per your terms and condition you are responsible for your own security.

We provide you with a modem free of charge which is encrypted and secure to a level we find acceptable.

If you feel that this isnt secure, then you are welcome to purchase and use your own modem, however we will not be able to pay for this replacement.

Wow. Secure to a level they find acceptable? I sincerely hope the rest of their infrastructure is more secure than this!

Besides, they arguably do not provide the modem "free of charge" - it's obviously subsidised by my monthly payments to O2 Broadband. As it's a fundamental component of the service I'm paying for, I feel they should certainly fix it or replace it.

They also completely failed to address the security problem again, so I fired off this in response (key points emphasised for this article):

From: Paul Mutton
To: O2 Broadband

I have to dispute your claim that the modem is secure to an acceptable level. As I have already mentioned several times, it is vulnerable to cross-site request forgery attacks. These would allow a remote attacker to view and change settings on the modem. I think it is entirely reasonable to expect a modem to be secure, regardless of whether it is "free" or not, so I would have to insist that you either fix it or replace it with a different modem with the same capabilities.

I phoned to tell you about this security problem more than a week ago now, and I have also pointed out that it probably affects every O2 Broadband customer who uses the O2 Wireless Box III. I get the distinct impression that you are not taking this seriously, which is of great detriment to myself and your other customers.

I'm finding it very frustrating that you are making it so difficult for me to report a serious security vulnerability that affects a huge number of your customers. I've spent quite a few hours researching the problem and trying to bring it to your attention, but to no avail. The guy I spoke to on the phone did not understand the problem, and evidently nobody at [email protected] understands it either, so I ask again: Who can I report this security problem to?

This was the second time I had explicitly asked who I can report security problems to; however, O2 once again completely ignored this question in their response:

From: O2 Broadband
To: Paul Mutton

Thanks for getting back in touch.

You have been given the answer which we deem acceptable in this circumstance. The O2 Wireless box is secured to a standard that is acceptable for home use.

If you feel that it is not suitable for your own use, then we do not force you to use this, and you are free to use any box you wish. We are under no obligation to supply you with a different router.

Unbelievable! They still maintain that the O2 Wireless Box III is secured to a standard that is acceptable for home use, even though I have pointed out that it can be controlled by a remote attacker.

This security problem needs to be fixed, but they won't even listen to me. Hoping for a case of "third time lucky", I immediately sent the following short reply:

From: Paul Mutton
To: O2 Broadband

Twice in a row you have completely ignored my question: Who can I report this security problem to?

It took O2 more than a day to respond to this simple question, and they didn't even answer it. Once again, they told me I can use a different router and that the security actually exceeds industry standards(!): "While I appreciate that you feel that the standard of the security of our router does not meet your expectations, we are confident that the level of security we offer meets and exceeds the industry standards."

Gosh – I didn't realise there were industry standards that say it's okay to let remote attackers gain unauthorised access to a home network through someone's router. On the plus side, she did say she had passed my comments on to their broadband development team. Whether anything will actually result from that is another matter – after all, they never phoned me back as promised.

Stonewalled

It's now a week and a half since I first reported the problem with the O2 Wireless Box III, and nothing has been done about it. I have made more than a reasonable effort by discovering the problem and trying to report it several times. Most people would probably have given up after being stonewalled like this, but it's very much in my interests to get this problem fixed: I am an O2 Broadband customer!

But still, I wonder how many security problems ultimately go unreported because companies make it so hard to report them? Worse still, how many vulnerabilities like these get sold to illegal groups of hackers? I'm starting to suspect that the latter option may be easier and more lucrative!

It is regrettable that I have had to announce this problem publically, but O2 has made it clear that they believe the security of the O2 Wireless Box III is acceptable. This is clearly not the case, so I hope this article will drive them into reconsidering the severity of the problem and fixing it quickly.

Widespread Impact

I'm not entirely sure how many users are affected by this problem, but it could be quite a lot. O2 has 457,000 fixed broadband customers as at 30 June 2009 [source: O2 PR], most of which will probably be using a Wireless Box II or III (these are the only routers currently offered to home users of O2 Broadband).

The O2 Wireless Box III is an O2 branded version of the Thomson TG585n router. If there are other ISPs that use this Thomson router, it is possible that the same vulnerability could affect their users as well. The user guide for the TG585n is available in English, German, Spanish, French, Italian, Dutch, Portuguese and Swedish, which suggests that it could be in use with many other European ISPs.

Finally, it appears that the slower O2 Wireless Box II (a customised Thomson TG585) is also affected by the same problem (thanks to SuperMatt for checking that). Additionally, there are several other ISPs that use the TG585.

Maybe the O2 Press Office Cares a Bit More?

I scoured the O2 website once again to try and find any kind of security contact, but none was listed. As a last ditch effort to try and report the problem, I phoned the O2 Press Office on 27th August 2009 to ask a few questions about the O2 Wireless Box III (including how security problems can be reported). I left my details in a voice message, but nobody called back.

I tried calling the O2 Press Office again the following morning, and they asked me to email my questions instead. I called again during the day to chase this up and eventually got a response to most of my questions that evening. They have offered to provide a statement about this security problem. I will update this article when I receive their statement.

Updates Since Publication

29 August 2009

O2 Press Office evidently does not understand the problem either. They have told me how to avoid any concerns by changing the default SSID and WEP encryption settings on the router. Changing the default SSID and encryption settings will not fix this problem. I've asked them once again who I can report security problems to.

I've also asked O2 Broadband customer services who I can report security problems to. Fourth time lucky? We'll see.

30 August 2009

No further progress. It is a Sunday, but security problems don't take the weekend off unfortunately.

31 August 2009

Oh dear, I just got the following email from O2 Broadband customer sevices (in response to my fourth request about how to report a security problem):

From: O2 Broadband
To: Paul Mutton

I've already passed your query on to our escalation team and the answer they responded with is what I've advised in my previous email.

If you would like to escalate this further please write to the following address.

O2 Complaint Review Service
[...etc...]

So there's a rather serious security problem that affects a huge number of their customers right now and I'm supposed to make a complaint about it via post?!

1 September 2009

Still haven't heard back from the press office about who to report security problems to. Feels like I've hit a brick wall with O2. What am I meant to do now? Give up?

Zen Internet has got in touch to see if their routers are affected. They seem quite proactive/receptive and will be taking the issue to Thomson as a result of their findings.

Possibly some more exciting news: I've successfully demonstrated the problem to BE (the smaller ADSL company that O2 bought a few years ago) and they have escalated the problem back to O2 on my behalf. Yay! BE uses similar routers to O2 Broadband, although only some of them are vulnerable. BE offers a staffed IRC support channel, which makes it incredibly quick and easy to report problems interactively.

Contact has been made! Chris Buggie (senior tech support manager at O2 Broadband) phoned me to apologise for the way this has been handled so far, and then to discuss the problem in detail. I explained the problem and talked him through some proofs of concept which were successfully demonstrated on his own O2 router. O2 is going to work with Thomson to introduce a fix. We also discussed ways to address the problem in the meantime. O2 Broadband customers can mitigate the risk of attack by enabling authentication on their router's HTTP configuration interface (by default, the device lets you browse directly to http://192.168.1.254 without requiring a password).

One other alarming thing that has become apparent during course of the day is that some other ISPs are affected by the same issue. This could means millions of broadband users in the UK are vulnerable.

2 September 2009

O2 issued a press release to The Register last night:

We have been notified of a potential security issue with the O2 Wireless box routers. We take this issue very seriously and are investigating it with the router manufacturer, Thomson. We thank Mr Mutton for bringing it to our attention.

Plusnet and Tiscali also have customers using the TG585. I've sent them further information and I understand that Plusnet is also talking to Thomson. I won't mention anything about Tiscali, as I don't think they want me to...

3 September 2009

As the problem evidently affects some other ISPs that are not owned by O2, it appears that there may be a general problem with the Thomson TG585 and TG585n routers. As Thomson are already aware of the problem, those will also be fixed hopefully.

I've told Nildram how to test for the problem and to mitigate it.

O2 has issued a new press release to The Register, which suggests the problem may soon be fixed without customers having to do anything:

Having been notified of a potential security issue with our O2 wireless box we have been working to find a solution. We have taken this issue very seriously and have been continuing to investigating it with the router's manufacturer, Thomson.

As a result we have identified a solution and we will be applying this remotely to all of our customers O2 wireless boxes. This means that customers will not have to take any action themselves.

So that's good news for O2 Broadband customers at last! It's just a shame that they did not react to my initial phone call, several emails to customer services and a few more emails to their press office. It was only when I published this article on the internet 2 weeks later that the problem was acknowledged. So while they may be taking this issue "very seriously" now, they certainly weren't before.

Upon reflection, I very much doubt that I would go to such lengths to report a security problem in the future. As well as getting emails from ISPs, I've also received quite a few from their customers. While the majority of these have been pleasant "good job!" sort of emails, a few have been extremely abusive, accusing me of lying about there being a problem and demanding that I publish a proof of concept.

Sadly, I have decided that it would have been much easier (and involve less hassle) to have left the problem unreported and cancel my O2 Broadband subscription and migrate to an ADSL provider which uses a different modem. There really is no incentive to report security problems when they make it this frustrating.

6 September 2009

At some point over the weekend, O2 remotely configured my Wireless Box III so that the configuration interface requires a username and password to be entered. This mitigates the original problem somewhat, but it introduces a different problem for me: they haven't told me what the username and password is!

I suppose it must be fairly secure now that even I can't log in to my own router, but it would be rather useful to be able to do things like change the firewall settings.

7 September 2009

More than 24 hours after being essentially locked out of my own router, I still haven't receieved any messages from O2 to say that they've done this, or any kind of clue as to what the username or password may be.

In fact, O2 hasn't contacted me at all since 1 September. They haven't updated me on anything yet; the only progress updates I've seen have been in press releases on various news websites. Conversely, nearly every other ISP that got in touch with me has kept me fully updated each step of the way as they mitigate the problem and work towards a solution.

I phoned O2 to find out what they had set my router's username and password to. I was told that a password isn't set up by default, so if logging in as Administrator with a blank password did not work, I could try poking the reset switch to restore the router to its default settings. I was obviously reluctant to do that, and although the person I spoke to was aware of the recent news about this security problem, he was not aware of O2 setting passwords on customers' routers. 21 minutes later, I eventually got the answer I was looking for (after it was referred to 2nd line): The passwords have been set to the serial number of the box. 21 minutes!! If you're going to change the settings on all of your customers' routers, is it really that difficult to take the additional step of telling them?

Just to stop hundreds more O2 customers emailing me, here is how to log in to your O2 Wireless Box: 

Username: Administrator
Password: [uppercase serial number from the bottom of your box]

So, O2 has applied a remote update to their Wireless Boxes which sets the password to the box's serial number. This does indeed mitigate the problem to some extent, but it does not remove the risk completely. The software release is still identified as 8.2.L.0 and it is still vulnerable to CSRF. The proofs of concept that I demonstrated to O2 (and several other ISPs) still work without further user interaction providing you have recently logged in to your router.

To summarise: The problem has not yet been fixed, but O2 has significantly reduced the likelihood of the vulnerability being successfully exploited.

Finally, a polite request to other broadband customers out there: Please stop emailing me to ask if your ISP's routers are affected. Practically all ISPs are aware of this problem now, so try asking them instead. I don't have time to reply to all of you personally, so I'd just like to take this opportunity to acknowledge everyone who has emailed me to say thanks for discovering the problem and persisting in reporting it without publically disclosing the vulnerability.

Random trivia: The O2 Wireless Box also respond to HTTPS connections on port 443, albeit with an invalid SSL certificate. The Organisational Unit (OU) field in the SSL certificate reveals the serial number of the router.

8 September 2009

This evening, I received a security update from O2 Broadband. It tells me what my password is! Better late than never, I suppose:

Dear Paul,

We've been told about a security problem that could affect your O2 Wireless Box.

The problem could let people change your router settings, which could change how it works.

What we'll do

We'll set up a password automatically to protect the settings on your O2 Wireless Box. You don't need to call us to do anything. You'll only need it if you want to change the settings. (In most cases, you'll probably never need it.)

The password will be the 11 digit alphanumeric serial number on the bottom of the O2 Wireless Box. The serial number has an "SN" in front of it e.g. "CP0749JTAM3" and a 2 digit code in brackets after it. Here's how to change it(If you would like to)

Nothing else changes

We won't change any other settings on your O2 Wireless Box. (The password you use to connect wirelessly to your O2 Wireless Box will stay the same.)

We apologise for the inconvenience, and we'll do everything we can to keep the disruption to a minimum.

Well, that's good - it clearly explains that there's a problem and what they have done to mitigate against it. More importantly, they aren't changing anything else, which is good. However, as I pointed out yesterday, the router is still vulnerable to CSRF attacks (setting a password does greatly reduce the risk of attack, but it does not remove it completely).

23 September 2009

This afternoon, I received another phone call from Mr Buggie at O2. I can't remember everything that he said, but I think these were the key points:

So it sounds like a fix will be on its way soon!

24 November 2009

It's been two months since my last update, and my O2 Wireless Box III still hasn't been updated with the fixed firmware. I'm not sure what's taking so long, as it is trivial to fix the attack vector (codewise, at least; testing and deployment are obviously another matter) and I can verify that the problem is indeed fixed in the beta release of the TG585 firmware.

13 February 2010

Oh dear... it turns out that my O2 Wireless Box III is still vulnerable to CSRF attacks, and does not require a password to access the HTTP configuration interface from the local network.

I asked O2 whether this was going to be fixed, and I was told that I could resolve it by performing a hard reset of my router. I was rather sceptical of this working, and indeed it did not fix the problem. (Thanks for wasting 20 minutes of my time while I reconfigure all of my WPA2 settings and so on!)

I'm truly disappointed by this. From my point of view at least, O2 does not appear to have fixed the firmware, and their mitigating step of applying a password to the router appears to have been revoked at some point (I'm not entirely sure when, but I think it's been like that for at least several weeks).

For avoidance of doubt, I have just re-run the proofs of concept that I came up with last year and these confirm that the O2 Wireless Box III is still vulnerable to CRSF attacks that allow an attacker to steal your WEP/WPA key, set up port forwarding, and so on. I don't think that's an acceptable position to leave customers in, particularly more than 5 months after claiming to have identified a solution.

Update: I checked what firmware version is installed on my O2 Wireless Box III, and it's still 8.2.L.0. I just phoned O2 and they confirmed that this is the latest version, and any future firmware updates would be automatically applied remotely, so it looks like they really haven't fixed it yet! A friend's O2 Wireless Box II (the slower version) is still vulnerable with firmware 7.4.20.4, but in his case at least, the HTTP configuration interface is still password protected (note that this does not actually fix the vulnerability; it merely reduces the window of opportunity to an attacker, as CSRF attacks can still piggyback on your authenticated session while you're logged into the box and until the session expires).

16 February 2010

Finally, it's fixed! After emailing O2 to point out that resetting the router did not fix the problem, they told me to phone technical support again. Technical support then put me on hold for a while before coming back and saying that I would have to phone some other number to get it sorted out. I didn't recognise the number, so I asked what department it was for, and they said it was the O2 Press Office. (I did question whether the press office would be able to fix this problem for me!)

Anyhow, it seems the press office were indeed able to get the wheels in motion very quickly. I phoned them last night to point out that the security problem hasn't actually been fixed, and they called me back this morning to say that my firmware had just been updated and they will shortly be applying the same update to all other O2 Wireless Boxes.

I've checked my O2 Wireless Box III, and it's now running firmware version 8.2.N.1. This version is demonstrably not vulnerable to the CSRF attacks I reported.

Although it's great news that it's been fixed at last, there are some things that were not so great:

On the plus side, O2 offered me a year's free broadband to make up for that last point. Unfortunately, around half of that period was spent using an insecure router that they refused to replace. Quite frankly, I don't think that really makes up for the time and effort I had to put into getting their attention, documenting the vulnerability details, developing proofs of concept to demonstrate the issue, and finally having to deal with the media onslaught that ensued.

It's sad that the only way to report this security problem was – ultimately – to disclose it publically, but it felt wrong not to persist when it was not only my router that was vulnerable, but also the routers belonging to hundreds of thousands of customers! Even so, my experiences over the past 6 months leave me with mixed feelings about what I would do if I were to discover a similar problem in a future: would you go through all of this again, or simply change to an ISP that provides a different router?

10 April 2010: New Vulnerability Discovered

Oh noes! As O2 have had plenty of time to fix the issue and roll it out to customers, I thought I'd take a quick look at discussing how the attack worked (as I thought it was quite interesting). However, before doing so, I thought it would be sensible to briefly poke my router again just to make sure that the new firmware (8.2.N.1) really is secure. As I previously verified, the vulnerability I originally reported has been fixed.

Nonetheless, I thought I'd have a quick look for similar problems elsewhere on the router. And...

Within one minute I found an identical vulnerability elsewhere on the router. I was amazed. I don't know if this new vulnerability was also present in the previous version of the firmware, as my router has been updated, but I was quite surprised to have found it so quickly, particularly as this new firmware had taken so long to be released and had also (allegedly) undergone security testing by two independent companies. I don't think much of their security testing if they missed something that I managed to find within a minute!

You may remember that O2 gave me the opportunity to test the beta version of this new firmware back in September last year. I declined this invitation without giving a reason, as quite frankly I don't think that security testing should be expected for free. I doubt the two independent companies did their security testing for free, even though they (presumably) did not detect the vulnerability that I was able to find in less than a minute.

Anyway, to clarify the situation, it seems that the O2 Wireless Box III is still vulnerable to the same types of attack as I described last year (i.e. theft of wireless encryption keys, port forwarding, setting passwords on otherwise passwordless routers, etc). I cannot comment on the situation with the Wireless Box II just yet, as I don't have one, and I'm not sure if this once again extends to all TG585 routers used by other ISPs in the UK. I have phoned O2 to let them know, as a search for "security" on their broadband help page does not return any results(!).

If you believe you may be affected, you can reduce the risk of attack by setting a password on your router (assuming you haven't already been attacked!); however, it is important to note that this does not fix the vulnerability, regardless of how strong your password is — it merely reduces the window of opportunity to the attacker. To coin an analogy, it's a bit like having a house with a front door that only locks itself 20 minutes after you've left the house.

This could have some serious implications after this week's introduction of the Digital Economy Bill. Even if you've taken every reasonable step to ensure the security of your own wireless network — including using the best WPA2 encryption — you're not really safe if there is some security vulnerability in your router that you couldn't reasonably be expected to know about. Although O2 takes a sensible stance on the whole debill thing, if someone were to use this vulnerability to hack into your router and download illegal material, you could be the one who ends up in trouble.

20 April 2010

I've not heard of any concrete progress from O2 yet, but they are aware of the problem and have passed it on to Thomson (which has rebranded as Technicolor). Just to answer a few common questions in the meantime:

27 April 2010

I just got an update call from Chris, who suggested that a fix may be ready in a matter of weeks as opposed to months. Technicolor (the new name for Thomson) have now identified the problem and have an early build of the firmware which addresses it.

I forgot to ask if they've tested the router for other similar problems. I don't have a lot of faith in the router at the moment :)

Vulnerability Details

I will not be revealing specific details of this problem until it has been fixed. Please do not ask me for any specific details unless you believe you are an affected party (e.g. an ISP that uses a similar router for its customers).

Full details (including example code) will be published here after the problem is fixed. Releasing a public proof of concept at this stage would be irresponsible, as it would demonstrate how to exploit the flaw before it has been fixed (so please stop asking).

The author of this page may be contacted via email at the address at the bottom.

 

Search this site

 

Copyright Paul Mutton 2001-2013
http://www.jibble.org/
Feedback welcomed
email

~
Dreamhost
Web Hosting

~
Dreamhost
Web Hosting