Its been a crazy busy and a bit tiring few days. Going to try and give a quick overview of how things unfolded here for thebaldgeek…
Thursday I had a solar panel over-voltage alarm on my solar UPS. To protect itself, it turned off. Thus dropping power to all loads connected it, ie my computers. Computers don’t like having their power dropped like that, so it was busy stressful afternoon/evening and the usual 3am Friday morning, but we got all systems up bar one Intel NUC that is just dead.
Midday Sunday Freddys girlfriend invited us over for dinner. Rare invite out, I was a bit ‘worried’ as we have training at work that week and I know how exhausting and stressful that is for this introvert and dinner can also be a bit draining, but we accepted and had a really nice evening.
Because of the dinner date, I did not see all my emails come in live as I often do, so laying in bed going back over them, around 9:20pm I see this one that came in around 5:10pm….
At approximately 22.50 UTC, Inmarsat experienced a full service outage on their Inmarsat I-4 F1 Satellite. Inmarsat are currently in the process of restoring services on other satellites within their fleet.
Inmarsat engineers are working to have service restored as soon as possible.
Ok, that’s not good… So not good I almost did not believe it. I checked a few other websites and was nothing about this outage. I really wanted to sleep, but there was no way I was going to be able if I could not confirm it, so I got out of bed and in front of my computers and sure enough, there was no aircraft data from Asia Pacific. Like, none. There was also no STDC maritime data. (The only two data streams I monitor from this satellite).
Not only no messages, but no signal. Like someone had turned off the satellite….
Not really believing it to be true still at this point, I thought I should just send a Tweet out before trying to sleep thinking some of my AVGEEK buddies would be wondering if I was asleep at the computer…. I now regret just how blasé I was on this first Tweet.
Have just been advised that Inmarsat 4F1, Asia Pacific have a major outage on their L-Band transmission. No ETA from them for restoration of services.
TL;DR it’s them, not us for once.
I mean, to be fair, any failures usually are me… Anyway, Tweet sent thinking the usual hard core 5-8 people would see it, the data would be back by morning and off I go to sleep.
Getting my sadly usual roughly 5ish hours of shuteye, I’m up at 3am and checking stuff on my phone… Uh.. the data is not back….. I mean, that cant be right… Whats going on?
Checking for updates from Inmarsat shows nothing.
Checking Google shows nothing…. I then check Twitter… Uh. Oh.
My Tweet is at around 1.3k views… That’s pretty much never happen before.
I double check things and jump on our Telegram chat group and the guys are all over it and confirming, it really is them and not me.
Not only is it them, but 4F1 has stations in Perth, Australia and Hawaii. For both to go down means some sort of major network failure.. or… and this just cant be it…. the satellite itself has failed….. Yikes.
I Tweet just a few things at this stage, adding to my thread.
About an hour later I get a Twitter DM from a farmer asking me what information I had… Odd, I know a lot of farmers have aircraft, but I do satcom ACARS, not small aircraft… Yeah, so turns out that Inmarsat also provide GNSS which is GPS augmentation (cm accuracy) and a ton of farming equipment need it for crop work.
Once again, I love to learn just how little I know.
A few more updates from our Telegram team and I am getting ready to head out the door to start the weeks training class and I get a DM from an aircraft dispatcher in New Zealand asking what I know about the outage. I’m starting to see a pattern here.
The aircraft procedure for satcom failure is to give position reports via HF and so the dispatcher workload has really gotten busy. I assure him I know nothing but if I do hear anything, I will try and let him know.
Work is busy teaching class to 12 people from around the world. I can feel my phone vibrating in my pocket a little more often than usual. I have already let the Telegram guys know I am away from all computers and they can jump on the Twitter thread any time they want and a few of the guys do as they figure stuff out and or get any updated information from any and all sources.
Looking at some of the Tweets in reply to my growing thread, I see people asking what all is covered by this satellite, so I tweet the above image to help people get their heads around just how massive this outage is.
A short time later, I Tweet the following two images to again try and help answer their questions, of ‘well ok, but just how many aircraft are we talking about?’
I guess that really got some attention on the matter as these images got around 96k impressions or views over the next 8 or so hours.
With no release of information from Inmarsat people are left to fill the vacuum, so my Telegram buddies start digging into NOTAMS (Notice To Airmen – alert bulletins) and clearly the FAA and other aviation parties are more than aware of the outage and are advising to give position reports via HF, this we know, but what the team starts really looking at is the ‘report via HF until’.. ie, how long they think its going to be out for…. and this gives us all some concern as they are not quoting hours, but days… .this is a really big deal.
About now, the satellite has been out for just over 28 hours and Inmarsat’s silence is really becoming loud. More and more people, mostly avgeeks, farmers and shipping folk are connecting with the Tweet thread. The thread is pretty solid, not your usual Twitter garbage, people are keeping the jokes and what they feel are ‘clever’ (and they never are) comments to themselves and so its pretty good quality overall.
The scary part for me is that the view (impression) numbers are just climbing to crazy, never been seen by me before, numbers. Most of my Tweets get 100’s of views, for each Tweet in this thread, we are looking at north of around 5,000 views with most over 50,000 views. Nuts.
Finally Inmarsat send an update that it is indeed a payload problem.
There seems there was a power glitch or power failure and to protect itself the satellite powered most of its transmitters and a lot of systems down and so they are diagnosing the issue and rebooting the satellite in orbit.
At this point, the Telegram team move from ‘guessing’ to ‘monitoring’. We slowly see some life with a few test carriers and data signals start to come back.
We Tweet each one and slowly we even seen aircraft show up on the map.
Finally around 2 days after the initial outage, we get some official word that all but sat-phones are back up and running.
So yeah, that is pretty much how my week went.
I hope that it was some help. Seemed like I did not do all that much beyond just try and let people know ‘its them, not us’.
Want to give a big shout out to the team on Telegram. Really solid bunch of smart folks that are cool to hang out with and at times like this make me look good.