SatSig topic:

Tooway KA-SAT outage Ukraine Feb 2022

(Read 8099 times)

Mar 1^st, 2022 at 7:04pm

"The Stack" has published an article about an outage of Ka-Sat service, possibly due to a suspected "cyber-event", first noticed 24th Feb 2022.

Read more:
https://thestack.technology/viasat-ka-sat-outage-cyber/

If you have been affected or know someone whose Tooway terminal has been affected please email me eric@satsig.net with details, including the town and country of the affected site.

Tooway beam coverage

Best regards, Eric

Mar 5^th, 2022 at 3:57pm

5 March 2022:

I've had two reports so far, both from France

1. Bedeille, in France a bit to the north of midway between Pau and Tarbez. Site appears to be well within beam 11(GW7) and marginal on the edge of beam 15(GW4).

2. Vion, in France midway between Le Mans and Angers. Site is within overlap of Beam 16(GW8) and Beam 22 (GW7).

More reports welcome in case any pattern can be discerned. If you know of any Tooway sites affected and their locations, please tell me eric@satsig.net

15 May 2022: Additional site reported, also failed 24 Feb 2022:
Location near Kowal (Wloclawski Park) Poland. Midway in good coverage in cross over region of beam 53 GW4 and beam 62 GW6. Advised to get replacement modem from their service reseller.

Mar 6^th, 2022 at 6:58pm

Also affected on 24th Feb.

Reported site is to the east of Marrakech in Morocco. Affected site is a Tooway terminal shared by 14 families. Beam 78 (GW1).

Help please to get it working again !

Mar 7^th, 2022 at 12:57pm

Please tell me here the status of the lights on your Surfbeam modem, in particular the receive RX status.

RX receive: After power ON, the RX indicator starts as OFF, then blinking SLOW, then blinking FAST, then STEADY ON once completely locked on the data stream. This receive acquisition process can take some long time and as it tries tuning to different possible downlink carriers at different frequencies and both polarisations.

If your RX status remains permanently OFF then your modem is no longer able to tune to any downlink carrier, or the satellite has stopped transmitting downlink carriers to your location.

Since some terminals are still working and some have gone dark on receive (RX OFF), it suggests to me that something has got corrupted in start up firmware of the terminals that have lost service.

Question: Was any firmware software update sent out to the affected sites, starting 24th Feb 2022 ?

Mar 8^th, 2022 at 11:24am

1. Report today 8th March.
At a community in the south of France with 5 customers. 4 failed with all LEDs off. 1 site ok. Beam and gateway unknown.

2. Some more speculative analysis here:

SATCOM terminals under attack in Europe: a plausible analysis

My guess is still that some firmware update was sent out from the Tooway network hub (and not from any specific passive gateway site) and that this update included malware that made the terminals unable to start up correctly and tune for a good downlink carrier.

It also appears that at affected terminals it is no longer possible for the customer to connect a PC and look at the modem status - signal levels etc.

Does a Surfbeam modem have a backup copy of its boot configuration in case an over-the-air upgrade fails?

Should affected customers send their Surfbeam modems back to ViaSat to have their firmware put back to factory default ?

Having lost control of a remote VSAT you don't have much option but to visit the site or send out new hardware.

I'm not sure that telling customers how to reprogram their firmware on-site is a good idea.

Mar 8^th, 2022 at 3:54pm

I've had a report that an affected terminal now has a very dim power indicator light.

If your terminal has been affected, what does the power LED look like on your terminal?

Below is what it should look like:

Further to this above, people are reporting that all the four indicator lights are off, except that there is an internal green LED still glowing. This internal LED may partially illuminate the back of the power light and give the impression of dim activation.

Mar 9^th, 2022 at 1:31pm

Faulty KA-SAT modems wanted for investigation/repair

I've been contacted by someone who has hardware/software expertise on SurfBeam 2 / SurfBeam2+ modems and is interested in attempting a repair.

Note that the attempted repair may involve interfering with the circuit board and components and there is a definite risk that the attempt will make the modem permanently unrepairable. This is a risky proposal.

If you have a modem that failed since 24th Feb 2022 and are willing to give (i.e. donate) your modem for possible repair please email me eric@satsig.net and I will pass your email to the person who has offered to help.

Meanwhile I presume that ViaSat are also doing their best to find a solution, so the alternative is simply to wait.

Mar 9^th, 2022 at 1:44pm

I've found today more information about ViaSat modem console access, memory dump and uart identification.

It gives hope that it may be possible to fix the modems without unsoldering components.

Don't despair! Please help if you are willing to donate a so-called 'bricked' modem, due to the 24th Feb 'cyber event' !

Mar 10^th, 2022 at 8:24pm

I have access to such terminals (both affected and unaffected) and can dig into the hardware. Please share information where the console is so I can check what can be done. The ethernet port is completely dead. Electrically it is ok but no ETH packets observed.

Zmrol

Mar 12^th, 2022 at 4:28pm

A site in Beam 74 (GW8) last worked at 4:45am on 24th Feb 2022

Mar 16^th, 2022 at 1:21pm

A report from France, near Chalus 87230. None of the front lights flash any more. There is a small led inside that lights up when power is on. Beam 15 (GW4).

Update:
11 March: Confirmation that hackers got internet access into the network management system, due to some unspecified misconfiguration.

They then presumably accessed the modems and corrupted their software. The affected modems need to be reprogrammed either by a site visit or by return to a repair centre. Meanwhile the network appears stable and affected customers are gradually being returned to service. If you are affected I suggest you keep in touch as best you can with your reseller. Explain what you use the terminal for so they can appropriatey prioritise you.

Mar 21^st, 2022 at 2:40pm

Report: SkyDSL terminal near Ronshausen / Germany (Beam 43 gateway 8).

SkyDSL / Viasat access point is broken since 24th Feb 2022. All four LED of the modems are slightly on constantly. There is obviously no self recovery implemented. I believe a corrupted firmware update has caused the issue. Support from skydsl was poor as could be. No info, no help. What shall I do with the modem?

Admin edit/addition: If you are still under contract with SkyDSL it is for them to fix it. My guess is that you will be asked to return the modem for repair of the firmware. A rather unlikely scenario, which has not been demonstrated, is that it may be possible to boot the modem using an inserted USB dongle sent to you by your reseller. The ViaSat repair centre should be able to access the firmware using the internal console serial port and reprogram the flash memory.

The problem is that affected modems will no longer boot up, do the initial self test sequence and then tune to a received downlink carrier from the satellite. Only once it is receiving a downlink carrier from the satellte is it possible for the network management centre to send out firmware and configuration updates to your modem.

Mar 31^st, 2022 at 5:06pm

Update 31 March 2022.

Ruben Santamarta has managed to dump the flash memory from a corrupted Surfbeam2 modem and compared it with a clean, unaffected version.

More details and comments here:

https://www.linkedin.com/posts/rubensantamarta_i-eventually-had-access-to-one-of...

Apr 2^nd, 2022 at 1:35pm

News from Viasat: 30 March 2022. Explanation of KA-SAT attack and restoration.

Viasat have provided an explanation about what happened and how Tooway resellers are to be able to help customers restore service.

Full details: https://www.viasat.com/about/newsroom/blog/ka-sat-network-cyber-attack-overview/

Briefly, hackers got on-line access into the network management computer system and caused it to send out data in order to corrupt Tooway customer modems so that they no longer connected themselves to the network. The modems were not permantly damaged and appear recoverable by reprogramming.

Meanwhile, Viasat has already shipped tens of thousands of replacement modems to distributors and is ready to ship additional modems as needed.

More technical investigations by enthusiasts:
https://www.reversemode.com/2022/03/viasat-incident-from-speculation-to.html

My recommendation: If you were affected, get in touch with your reseller and ask for replacemrnt modem.

Apr 3^rd, 2022 at 10:42am

SentinelLabs has been investigating the KA-SAT/Tooway outage.

SentinelLabs is an open venue where investigators into cyber crime share their findings and collabarate.

Regarding KA-SAT and Tooway, their conclusion is that it is likely that the Surfbeam modems had their software corrupted by a "wiper" program previously downloaded from the network management centre.

Ref: https://www.reversemode.com/2022/03/viasat-incident-from-speculation-to.html

When this malware "wiper" program was run it erased the intended code in the modem and replaced the code with spurious data, as shown on the left above. The malware code is called "Acid Rain" and the headline "AcidRain - A Modem Wiper Rains Down on Europe" seems most appropriate, although it is no laughing matter. The result and implications of this attack are very serious and all networks involving customer modems and routers, whether IOT devices, ADSL modems, cable modems or satellite modem/routers need to be on thier guard.

The SentinalLabs report is here and I strongly recommend you to read and download:
https://www.sentinelone.com/labs/acidrain-a-modem-wiper-rains-down-on-europe/

There are a number of references, which I list below for further reading and background information.

https://www.wired.com/story/viasat-internet-hack-ukraine-russia/

https://www.cisa.gov/uscert/ncas/alerts/aa22-076a

https://media.defense.gov/2022/Jan/25/2002927101/-1/-1/0/CSA_PROTECTING_VSAT_COM...

https://www.airforcemag.com/hackers-attacked-satellite-terminals-through-managem...

https://nps.edu/documents/104517539/104522593/RELIEF12-4_QLR.pdf/9cc03d09-9af4-4...

https://www.reuters.com/business/media-telecom/exclusive-hackers-who-crippled-vi...

https://www.viasat.com/about/newsroom/blog/ka-sat-network-cyber-attack-overview/

https://blog.talosintelligence.com/2018/05/VPNFilter.html

https://blog.talosintelligence.com/2018/06/vpnfilter-update.html?m=1

https://blog.talosintelligence.com/2018/09/vpnfilter-part-3.html

https://www.ncsc.gov.uk/files/Cyclops-Blink-Malware-Analysis-Report.pdf

https://www.trendmicro.com/en_us/research/21/a/vpnfilter-two-years-later-routers...

https://www.cisa.gov/uscert/ncas/alerts/aa22-054a

Apr 9^th, 2022 at 4:07pm

9 April 2022: A SkyDSL customer in Germany affected with same symptoms on night of 23/24 March 2022. All 4 front panel LEDs OFF and internal Green LED ON.

Surfbeam modem: All front panel LEDs OFF.

Surfbeam modem: Internal Green LED ON.

Beam 32. Gateway GW2, Rambouillet, France.

Follow up: SkyDSL was informed, the affected Modem was RMA returned to SkyDSL and a replacement modem has been sent to and received by the customer.

Admin comment: I will add more details here once the replacement modem is actually working properly and service has been restored.

The new modem has Power LED on, blue, the RX LED is blinking slow in blue since 24 hours, no internet. Sky DSL is almost impossible to be reached. It is not clear if the TX and NETWORK LEDs are dim and also slightly blinking slow.

Surfbeam modem: Are the lower LEDs flashing or not ?

Admin: Once the Power is ON the modem will start trying to find the best downlink carrier. This involves tuning systematically to multiple likely frequencies and also switching polarisations in case you are in the border area of 2 or 3 adjacent beam coverages. During this time the RX will flash slow (1 sec) and the TX will be OFF.

If your replacement modem won't lock to any downlink carrier contact SkyDSL as your modem may not have been prepared with carrier frequencies for the KaSAT network.

Once locked to the best downlink carrier the RX will go double rate fast flashing (0.5 sec) and the TX will flash from time to time as it attempts to range the satellite and register with the network management centre.

Once ranging and registration are complete the RX will go solid ON and TX will flash when transmitting traffic.

14th April 2022: Finally fixed and service restored.