Advertisment: Broadband via satellite
Advertisment: Planet Earth rotating animation

www.satsig.net

Satellite Internet Forum.

Welcome, Guest.        Forum rules.
      Home            Login            Register          
Pages: 1

Remote links went down

(Read 5148 times)
Ex Member
Ex Member


Sep 28th, 2009 at 11:31am  
Hi TDMAMike/Eric Johnson:

I am facing strange problem from last couple of days, all remote links went down suddenly without any reason (Forward Link keep in tact and visible on spectrum analyzer but return channels go flat), But Hub Chassis,PP,NMS remain powered up and accessed through LAN. No physical disconneciton occurs.

When I open effected remote link,iMonitor's Events/Conditon window dosen't show polling that usually update after every 20 seconds and so on. The following polling information is available for your review.

Time      Name      Type-SN      Type      Network      Event Level      Event Description
9/28/2009  3:50:45 PM      Client1      3100.75610      Remote      IS      Info      UCP!: SO( +43), FO( -6666), PO( +1.0) NID(6)

I without any reason delete one of my dummy idirect modem from iBuilder 8.0.2.1 then suddenly polling of infected remotes begin starts and after few minutes return channel (TDMA Carrier) is being seen on spectrum analyzer which were not shown during problem (Flat Line on Spectrum Analyzer).

My Question is that Why this is happening to our network. Is there any procedure/SOP for NMS/PP to check what is the problem that is being faced us. Furthermore, We are using 8.0.2.1 IDS version in 15050 Infinite Hub.

Looking forward to your response.

Thanks and Regards,

Jawad
Back to top
 
 
IP Logged
 
TDMAMike
Senior Member
★★★
Offline



Posts: 826
Reply #1 - Sep 28th, 2009 at 1:50pm  
I would need additional granualarity to even take a stab at this.  Please answer the following questions:

1.  You can confirm that your forward carrier remains up?  Visible on specan?  Any chance of energy underneath that carrier?

2.  TDMA upstreams:  Can you confirm whether or not they are visible on the specan?  I am talking strictly RF.  Can you see any TDMA carriers?

3.  In the Events and Conditions tab of iMonitor, are you saying they are maintaining UCP?  That polling information that you discuss above is obviously a capture from the events and conditions tab.  Is that what you observe at the time of the anomaly?

4.  What are the remotes reporting?  Do they still have a green RX andd TX LED?  What about NET LED?  If you have the opportunity, I would like you to have one of them open a telnet session (at the time you observe this issue) and see if they are receiving sweeping commands.  You can do this by typing acq on at the command line of a telnet session.

5.  Is what you are reporting happening everyday?  How often are you seeing this and is it happening at the same time everyday?  If it is happening everyday at the same time, what time is it happening (on the NMS).  

6.  Can you detect any bursts on the RX line cards at the time of the outage?  Please telnet to one or all of your RX line cards and run a dumpb 100 -a command on the line card(s).  It is a non-intrusive command, and will not cause a service interruption.  All you are doing with this command is dumping bursts on the line card (as they arrive).  Where I am going with this, is to check if the remotes are trying to burst in (keep in mind, this would be visible on a specan as well).  Alternative option: You can also look for rx bytes on the line cards by right clicking on them (in iMonitor) and reviewing the line cards stats for each of them.

7.  Do you have any LBand line amps on the downlink chain at the earthstation (hub).  Are you seeing an fluctuations in rx_power at the time you observe this issue.  Please check your rx_power levels on the rx line cards to ensure you are not seeing any radical power levels (this is an unlikely culprit, but I need to ask you to check).

8.  Are you using the downlink compenents to sustain any OTHER networks?  Read: are you using that same antenna/IF cabling/downconverter to terminate any other networks?  (just asking).

Lots of variables.  Answer what you can above.

Back to top
 

Regards, &&&&M
 
IP Logged
 
Ex Member
Ex Member


Reply #2 - Sep 29th, 2009 at 12:38pm  
Hi TDMAMike:

Thanks for reply and guidance.

1- Yes, Forward carrier remains UP and no other carrier underneath it.

2- No upstream carrier(return channels) are detected/visible on spectrum analyzer.

3- Remotes in iMonitor Tab (Event/Condition) there is no UCP, when all remote links went down.

4- Yes all remotes has RX LOCK (SOLID Green LED). TX and NET LED went Orange (Blinks).

5- There is not specific time/day of downtime of the network.

6-  Today the network is stable and i am checking RX Line Card status in iMonitor and tell you later when i found network down.

7- There is no L-Band line amplifier on downlink chain at earthstation.

8-  No extra component to run other network.

Best Regards,

Jawad
Back to top
 
 
IP Logged
 
TDMAMike
Senior Member
★★★
Offline



Posts: 826
Reply #3 - Sep 29th, 2009 at 1:43pm  
If your returns are flat (noise floor only) and there are not obvious signs of TDMA carriers, your remotes are losing their timeplans.  Hence the reason for their TX LEDs being amber/flashing.

The fact that they are completely flat tells me that your downlink chain (at the hub) is likely good. From here, I would like you to consider having one of your remotes open up a telnet session and check for the presence of ACQ sweeps (the next time this anomaly is observed).   Command in telnet acq on  

Need to determine if acq sweeps are visible on each remote at the time you observe this issue (trying to determine if the TDM downstream carrier and PPs are doing their jobs).

If ACQ sweeps are not visible (when that RX LED is green and the TX and NET LEDs are flashing amber)means you might have something going on with your TDM forward carrier, or something is amiss in the PPs.   Before we spin off on a tangent, I would like you to confirm those sweeps on a single remote (it has to be at the time you observe the issue you describe).  

If I were to take an educated guess at your problem, I would say that SOMETHING, or SOME kind of energy was being injected into the same spectrum as your forward.  You mentioned that you deleted a dummy remote to unhang it previously (which brought your upstreams back up)?  Is there any chance that dummy remote was radiating the same center freq or chunk of MHz as your forward?  In such a case/instance, your forward carrier would appear healthy (as the remotes are seeing..with RX LEDs that are green), but the TDM Forward would not be capable of carrying data.
Back to top
« Last Edit: Oct 1st, 2009 at 7:35pm by TDMAMike »  

Regards, &&&&M
 
IP Logged
 
Ex Member
Ex Member


Reply #4 - Oct 1st, 2009 at 3:42pm  
Dear TDMAMike,

I have observed following few thing which are happened at infected client site.

1- NO RX LOCK

2- When RX LOCKED, some observation are noted at one of the infected site.

   i- NET LED blinks (orange in color)
   ii- No TX  LED blinks

3- When telnet (while RX LOCKED) to infected site with following command to run. We got some results to watch

   acq on

(2415)/tmp/3100.72992.pkg (6093) – ignore (target-list)

4- When NET LED is about to LOCK, a few seconds before to this following result came on telnet screen of infected site as well as on iMonitor's Event/Condtion window (when we apply acq on command)

UCP!: SO(+45), FO(-6945), PO(+0.5) <022280.764>

Please guide me about the problem.

My Observations:

- Is there any PP problem???

Regards,

Jawad
Back to top
 
 
IP Logged
 
Ex Member
Ex Member


Reply #5 - Oct 3rd, 2009 at 1:58pm  
Hi TDMAMike,

I have studied IOM Level I iDS v8.0 book chapter no. 5's page no. 20. It says...

If no sweeps are seen on iMonitor it means that the PP does not know about the remote.

I have two PP in our network. PP1 looks fine when we switched to PP2 only one site came to online and others dose not, only RX is locked.

TX LED never comes on (it means remote is locked on the SCPC carrier but doesn't appear to be receiving any ACQ sweeps from the PP.

I want sincere help from you or any other person who has solution about the problem.

I think PP or NMS is damaged.

Regards,

Jawad
Back to top
 
 
IP Logged
 
TDMAMike
Senior Member
★★★
Offline



Posts: 826
Reply #6 - Oct 3rd, 2009 at 2:25pm  
Questions:

1. Two sites correct?
2. Do the remotes have a receive light?
3. When each of the sites turn on ACQ messages in telnet (acq on) do they or dont they see ACQ sweep commands?  
4.  Do these remotes reside in the same network?  Same inroute group?

I dont think your PP is damaged, but if the remotes have a green RX LED and you are not able to see ACQ sweeps, then there is obviously something amiss.  Let not get carried away with that assessment just yet.  Please answer the above questions as clear as possible so I can gainn an understanding of what you are, or arent seeing. 

M


Back to top
 

Regards, &&&&M
 
IP Logged
 
Scout
Senior Member
★★★
Offline



Posts: 86
Reply #7 - Oct 6th, 2009 at 1:52pm  
Quote:
I have two PP in our network. PP1 looks fine when we switched to PP2 only one site came to online and others dose not, only RX is locked.

TX LED never comes on (it means remote is locked on the SCPC carrier but doesn't appear to be receiving any ACQ sweeps from the PP.



I think PP or NMS is damaged.

Regards,

Jawad

OK.  when you have PP2 active and you only have one modem online, please go to iMonitor and check the remote status TAB of a modem that is offline, look for any messages, if the PP is sweeping for the remote you will see clear messages with SWEEP in the line.  If the PP is not sweeping for the remote you will get a green Rx LED on the modem and no Tx as the PP is not telling the remote to start transmitting.  This sounds likely the cause.  when you are switching between the PP machines, how long are you leaving it before you test?  it can take 7 minutes for a modem to switch from a failed, or switched off PP.
How many modems do you have on the network? 
Do all modems come online with just PP1 active and PP2 switched off?
have you run any blade re-balance commands on the PP controller in the NMS to ensure the PP SAMNC process is talking correctly to the NMS?
I doubt the PP is damaged, there seems to be an issue with the PP and NMS communicating.
Back to top
 
 
IP Logged
 
TDMAMike
Senior Member
★★★
Offline



Posts: 826
Reply #8 - Oct 11th, 2009 at 5:00pm  
Scout wrote on Oct 6th, 2009 at 1:52pm:
I doubt the PP is damaged, there seems to be an issue with the PP and NMS communicating.
I was thinking the same thing.  Process related.
Back to top
 

Regards, &&&&M
 
IP Logged
 
Pages: 1