Remote links went down

I would need additional granualarity to even take a stab at this.  Please answer the following questions:

1.  You can confirm that your forward carrier remains up?  Visible on specan?  Any chance of energy underneath that carrier?

2.  TDMA upstreams:  Can you confirm whether or not they are visible on the specan?  I am talking strictly RF.  Can you see any TDMA carriers?

3.  In the Events and Conditions tab of iMonitor, are you saying they are maintaining UCP?  That polling information that you discuss above is obviously a capture from the events and conditions tab.  Is that what you observe at the time of the anomaly?

4.  What are the remotes reporting?  Do they still have a green RX andd TX LED?  What about NET LED?  If you have the opportunity, I would like you to have one of them open a telnet session (at the time you observe this issue) and see if they are receiving sweeping commands.  You can do this by typing acq on at the command line of a telnet session.

5.  Is what you are reporting happening everyday?  How often are you seeing this and is it happening at the same time everyday?  If it is happening everyday at the same time, what time is it happening (on the NMS).  

6.  Can you detect any bursts on the RX line cards at the time of the outage?  Please telnet to one or all of your RX line cards and run a dumpb 100 -a command on the line card(s).  It is a non-intrusive command, and will not cause a service interruption.  All you are doing with this command is dumping bursts on the line card (as they arrive).  Where I am going with this, is to check if the remotes are trying to burst in (keep in mind, this would be visible on a specan as well).  Alternative option: You can also look for rx bytes on the line cards by right clicking on them (in iMonitor) and reviewing the line cards stats for each of them.

7.  Do you have any LBand line amps on the downlink chain at the earthstation (hub).  Are you seeing an fluctuations in rx_power at the time you observe this issue.  Please check your rx_power levels on the rx line cards to ensure you are not seeing any radical power levels (this is an unlikely culprit, but I need to ask you to check).

8.  Are you using the downlink compenents to sustain any OTHER networks?  Read: are you using that same antenna/IF cabling/downconverter to terminate any other networks?  (just asking).

Lots of variables.  Answer what you can above.

If your returns are flat (noise floor only) and there are not obvious signs of TDMA carriers, your remotes are losing their timeplans.  Hence the reason for their TX LEDs being amber/flashing.

The fact that they are completely flat tells me that your downlink chain (at the hub) is likely good. From here, I would like you to consider having one of your remotes open up a telnet session and check for the presence of ACQ sweeps (the next time this anomaly is observed).   Command in telnet acq on  

Need to determine if acq sweeps are visible on each remote at the time you observe this issue (trying to determine if the TDM downstream carrier and PPs are doing their jobs).

If ACQ sweeps are not visible (when that RX LED is green and the TX and NET LEDs are flashing amber)means you might have something going on with your TDM forward carrier, or something is amiss in the PPs.   Before we spin off on a tangent, I would like you to confirm those sweeps on a single remote (it has to be at the time you observe the issue you describe).  

If I were to take an educated guess at your problem, I would say that SOMETHING, or SOME kind of energy was being injected into the same spectrum as your forward.  You mentioned that you deleted a dummy remote to unhang it previously (which brought your upstreams back up)?  Is there any chance that dummy remote was radiating the same center freq or chunk of MHz as your forward?  In such a case/instance, your forward carrier would appear healthy (as the remotes are seeing..with RX LEDs that are green), but the TDM Forward would not be capable of carrying data.

Hi TDMAMike,

I have studied IOM Level I iDS v8.0 book chapter no. 5's page no. 20. It says...

If no sweeps are seen on iMonitor it means that the PP does not know about the remote.

I have two PP in our network. PP1 looks fine when we switched to PP2 only one site came to online and others dose not, only RX is locked.

TX LED never comes on (it means remote is locked on the SCPC carrier but doesn't appear to be receiving any ACQ sweeps from the PP.

I want sincere help from you or any other person who has solution about the problem.

I think PP or NMS is damaged.

Regards,

Jawad

Quote:

OK.  when you have PP2 active and you only have one modem online, please go to iMonitor and check the remote status TAB of a modem that is offline, look for any messages, if the PP is sweeping for the remote you will see clear messages with SWEEP in the line.  If the PP is not sweeping for the remote you will get a green Rx LED on the modem and no Tx as the PP is not telling the remote to start transmitting.  This sounds likely the cause.  when you are switching between the PP machines, how long are you leaving it before you test?  it can take 7 minutes for a modem to switch from a failed, or switched off PP.
How many modems do you have on the network? 
Do all modems come online with just PP1 active and PP2 switched off?
have you run any blade re-balance commands on the PP controller in the NMS to ensure the PP SAMNC process is talking correctly to the NMS?
I doubt the PP is damaged, there seems to be an issue with the PP and NMS communicating.