Guys i need your help on this.

I have notice that on my satellite link that my daily bandwidth utilisation hits 240kbps on a link which is a 256kbps symmetric SCPC link with a DVB modem that recieves and a comtech 570L modem for transmit. My service provider keeps saying that its from my network but my router activity does not support that argument.

From my network setup, a router separates the vsat network from the local area network. this internal router (LAN) gets traffic from the LAN and delivers to the firewall which then passing traffic to the router hosting the vsat. On the internal router the network traffic is almost at zero level after office hours (that is when users have closed for the days work).

On the other hand, i noticed that during peak periods (when users are online) the software used to monitor (cacti) indicates that the link performance hits an average of 100kbps whereas at off peak it hits 200kbps and above, thats almost working on full capacity.

I would please need suggestions on this issues i have stated and what i can do at my end to trouble shoot my VSAT link so that i can present more facts to my service provider shld the problem be from them.

Secondly i would like to know why my link is slow when uses are online. we have less than 70pcs on this link in a fully and welled segmented network.  All users are on Vlans (about 5vlans) and we use high end switches for inter vlan routing and 3550 cisco switches at the distribution level, there are no hubs in this network.

Any more information on this network to help in resolving this problem will be made available.

Thanks for your anticipated suggestions.

Just a start.   I hope others will add more replies.

If you have the IP module in the 570L there are statistics available about traffic both on the LAN side and the satellite side.  Header and payload compression stats may also available.  It would be interesting to see how the satellite uplink traffic corresponds to what is going into the modem.  Then step back and investigate the traffic from the first router connected.   If it is a Cisco you may be able to bridge the port and monitor all the traffic on a 'parallel' listen port.  Otherwise just put a hub in the way to break out an ethernet.  Run it into ethereal for a while and see where the packets are coming from and going to, and why.

The high uplink traffic during off peak period is strange.  it might be a virus on a PC sending out emails at top speed all of the time, some server PC distributing MP3 files, a disk back up system dumping to some off-site back up device etc.  Does the firewall itself dump its disk to an offsite server every night ?

None of the above may apply, so please will others add their comments below.....

Best regards, Eric.

just couple of fast notes:

1. From your description, it sounds like your link does reach its limits: If you see on the MRTG (or any other sampling application you are using) that it reaches 240kbps, remember that it is just a sampling and the actual usage is always bit more. How much more - it depends on couple of factors, one of them is the sampling interval.
2. I agree with Eric that you should take a deep look for trojans and worms on all the pc's of your network. I suggest you to troubleshoot it doing this:
First, connect all your machines to the network and turn them on, open all the applications (like explorere, messenger and so on) but dont do anything - dont browse or touch the pc. Check the bandwidth usage. If you locate a problem, try to disconnect half of the machines to localize which machine(s) are contaminated. Once you will pinpoint the problem you will be able to solve it as well.

Good luck,
Nimrod

Thanks for your contributions so far.

For viruses, we have a mcafee enterprise solution that is centrally managed for spam and viruses. Also there is no off-site backup or dumping of disk. All backups are manually done and during weekends (afternoon). All the switches and routers we have are Cisco products. the FW is Cisco PIX. The 570L does not have an IP module.

Is it possible for ethereal to monitor traffic of a router interface from my pc? if yes, how? I have installed it but cannot find an place where i can specify the IP or mac address of an interface i want to monitor except that of the NIC on the pc.

Also what are the major things i should check for to ascertain if my VSAT link is in proper shape?

Nimrod i have worked on your suggestion in locating a contaminated pc but i am yet to conclude on it becasue i have over 70pcs distributed in 5floors. This will take some time to complete.

Regards.

well you can check the physical parameters, like Ebno on both sides, CRCs and so on.

Another two checks you can do is:
1. try to ping -w 500 and -l of, lets say, 1000 for a long time and see how much time outs do you have and what is the pattern of the time outs.
2. check the counters on your router, erase them and check if they are increasing.

This is that result i get from Ping 66.94.234.13 -l 1000 -t

Ping statistics for 66.94.234.13:
   Packets: Sent = 62, Received = 0, Lost = 62 (100% loss),

and for ping 66.94.234.13 -w 500 -t
Ping statistics for 66.94.234.13:
   Packets: Sent = 62, Received = 0, Lost = 62 (100% loss),

I have also reset the counters on the router interfaces. My EbNo is 16db, how do i check the CRC?

Regarding ethereal, you need to connect your monitor PC in parallel with the traffic you want to monitor. Two ways:
1. Find the existing cable to be monitored and make a break out.  Put a passive hub with a side cable to your PC.
2. If one end of the cable goes to or comes from a Cisco router or switch and you know how to configure it, it may be possible to use a spare port and bridge it across in parallel with the port to be monitored.  Cisco have some good traffic monitoring facilities, but be very careful about any Cisco reconfiguration.    If you have a switch on each floor then I guess you might be able to use a spare port to do a monitor on any particular PC connected.   If you have one main switch connected to each floor then at that switch you may be able to use a spare port to monitor any whole floor.   The connection between your firewall and the satellite modem may need solution 1 above, using a passive hub.

Regarding the Eb/No of your transmit carrier (from the 570L) you need to call the hub and see what is the reading at their end at different times of day.  It is possible that your transmit carrier is low power (moisture in cable, antenna moved slightly etc).  It could get worse at night if the satellite moves out of the beam during that period.  This can happen if the dish is mispointed.   During clear sky conditions at both ends you should have a good rain margin of perhaps 4 or 5 dB.  During clear sky conditions at both ends try reducing your transmit level by 3 dB.  The network should continue operating normally.  If is fails then you have less than 3 dB margin which is not good enough.  
Contact the hub and re-peak your dish and adjust your transmit level, while talking to them, till the Eb/No is correct at the hub.

Best regards, Eric.

imepals wrote on Apr 24^th, 2007 at 11:50am:

This is that result i get from Ping 66.94.234.13 -l 1000 -t Ping statistics for 66.94.234.13:    Packets: Sent = 62, Received = 0, Lost = 62 (100% loss), and for ping 66.94.234.13 -w 500 -t Ping statistics for 66.94.234.13:    Packets: Sent = 62, Received = 0, Lost = 62 (100% loss), I have also reset the counters on the router interfaces. My EbNo is 16db, how do i check the CRC?

For the Ebno it is ok. You will need to check the ebno at your service provider though. For the ping, I can ping it very well, I guess the ping is blocked by your firewall.

The firewall is not blocking the ping. If i run a normal ping you get this

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>ping 66.94.234.13 -t

Pinging 66.94.234.13 with 32 bytes of data:

Request timed out.
Reply from 66.94.234.13: bytes=32 time=4191ms TTL=47
Request timed out.
Request timed out.
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3217ms TTL=47
Reply from 66.94.234.13: bytes=32 time=2812ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3052ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3337ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3491ms TTL=46
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3355ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3361ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3568ms TTL=46
Reply from 66.94.234.13: bytes=32 time=2907ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3331ms TTL=47
Reply from 66.94.234.13: bytes=32 time=2733ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3356ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3541ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3654ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3992ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3694ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3793ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3506ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3749ms TTL=47
Reply from 66.94.234.13: bytes=32 time=4334ms TTL=47
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3808ms TTL=46

Ping statistics for 66.94.234.13:
   Packets: Sent = 28, Received = 22, Lost = 6 (21% loss),
Approximate round trip times in milli-seconds:
   Minimum = 2733ms, Maximum = 4334ms, Average = 3490ms
Control-C
^C

Using about 20 packets, I get a wide variety of ping times the other way to an IP address ending in .178

Test 1.    646mS  3958mS average: 1090mS.   No lost packets.
Test 2. 1821mS  3498mS average  2656mS  No lost packets
Test 2.    834mS 2598mS average 1693mS  No lost packets

Try tracert also.

Best regards, Eric.

It sounds as though you have a DVB/SCPC service, not SCPC/SCPC.  Your DVB download may be oversubscribed more than you expect.

I did not see any mention of TCP/HTTP Acceleration.  If you don't have TCP Acceleration it's quite normal that you would not be able to fully utilize the circuit.  The TCP protocol is a guaranteed delivery protocol.  It sends a little data, stops, waits for an ACK from the remote receiver, then doubles up the amount of data, sends, stops, etc.  In this manner it learns the speed and congestion of the link.  Since satellite delay is at least 1/2 second, the TCP protocol interprets this as a very slow or congested circuit.  Regardless of how much bandwidth you have, you are unlikely to get much more than about 70 - 90 Kbps per TCP session.  Of course multiple sessions will allow you to fill up the circuit, but given all the TCP overhead, it's unlikely you'll ever see full use of your circuit.  With TCP Acceleration, you may get throughput in excess of the link speed because a lot of the TCP overhead is stripped off.

Comtech has an IP module that provides QoS and other features, but it does not include integrated TCP Acceleration.  For this you need an external device.  It also helps if your network operator is providing TCP/HTTP Acceleration on the DVB portion of the link.

HTTP Acceleration is also useful to speed up browsing.  Every single piece of content on a web page must go  through a 3-way handshake process.  Again, with the 1/2 second latency, this can really slow down web downloads between pieces of content.  

See someone like Packeteer/Mentat for a TCP/HTTP Acceleration solution, or perhaps UDCast.  There are others out there, that I'm sure people on the forum can point you to.  Most 2-way services such as iDirect have this built in.  

Hope this helps,

Pat

Gentlemen,

I am going to put it in a bit different perspective. 256 kbit/s Rx is way too small for 70 PCs in an office environment, even if it is all CIR. I would recommend a maximum of 5-10 simultaenously online PCs per a link providing 256 kbit/s throughput, depending on the usage patterns and environment. How many simultaneously online PCs do you have in average?

As for the pings, what was the network state when you did the ping test? Have you disconnected all the PCs or you ran the pings with the network connected? If second, then the link is overloaded. It will be slow if the usage won't be controlled.

Additionally, some providers have their DVB carriers are a bit oversubscribed to attract the customers with cheap pricing. Patrick is correct. Some providers offer 256 kbit/s CIR, however that's not a true CIR. That's probably why you can see it's operating at 200 kbit/s. If "256 kbit/s CIR" is not stated in SLA, the sharing factor could be even more.

What you may want to do in this case:

1) Squeeze the most out of your link. Implement HTTP acceleration and compression. This can be done with 3rd party accelerators as Pat mentioned. BusinessCom has it's own PEP (Performance Enhancing Proxy) which is an accelerator/compressor. We can deploy a trial version of PEP for you free of charge so you could see it in action. It doesn't require any proprietary hardware, you'll need a Linux server which will work as a transparent proxy for all the HTTP traffic.

2) Control over how the services are used in your network. You may want to implement smart traffic engineering solutions such as Packeteer or once again, BusinessCom TES-1000 unit to control traffic flows, deprioritize unwanted traffic, implement a throttler. That helps when the network is busy and budget is tight.

3) Add more bandwidth into the VSAT backbone. 256 kbit/s for 70 PCs is really too small. You may also want to consider changing platform to provide you with a broader BIR quota if your traffic is burstable. Probably for the same cash you're buying a 256/256 CIR link you could get something like 1024/256 BIR with 128/32 CIR which may perform better. This needs a deeper analysis though and it's hard to tell right away.

Please let me know if you would like me to assist you with the PEP trial. That may save you some kbit/s.

Thanks for all your response to this post. Maxim Usatov pls do send me the PEP trial. I have also considered your suggestions and but would like to know how to calculate the capacity if satellite you need for a given number of PCs. Over here like i stated before we have about 70pcs, 50 of which can be online simultaneously.

Another problem i have ji=ust noticed is the high latency on my network. a roundtrip delay returns at >3000ms sometimes 4000ms.

An investigation revealed that a ping from my fast-ethernet port to the serial port of my boarder router returns at >1000ms! These 2 interfaces are on the same router (cisco 3745 router).

Also in the past week i have observed from my Cacti software (a bandwidth utilization monitoring tool) that i am transmitting at over 307kpbs (i have not upgraded to this service rate) whereas i am subscribed for a bandwidth of 256/256kpbs. I have reported this to the service provider but nothing has been done about it and besides the latency is still high.

On the DVB, is there a way i can determine the overbooking or oversubscription rate from the DVB box or is it only the provider that has access to such information? And what’s the main difference between DVB/SCPC and SCPC/SCPC?

Pinging yahoo.com [216.109.112.135] with 32 bytes of data

Reply from 216.109.112.135: bytes=32 time=3989ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3960ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4049ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4288ms TTL=53
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4332ms TTL=54
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4274ms TTL=53
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4094ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3859ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3658ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3837ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3771ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4057ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4157ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3816ms TTL=54
Reply from 216.109.112.135: bytes=32 time=4015ms TTL=54
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4127ms TTL=53

Ping statistics for 216.109.112.135:
   Packets: Sent = 20, Received = 16, Lost = 4 (20% loss
Approximate round trip times in milli-seconds:
   Minimum = 3658ms, Maximum = 4332ms, Average = 4017ms

Expecting your professional advise as usual.

70 clients on a 256k?  My goodness, talk about trying to do more with less.  That sounds painful.  

I would venture to say that your link is beyond oversubscription.

The general idea here seems to portary that my network is overloaded. can someone pls suggest how i can calculate the bandwidth that will be appropriate for 70 online pcs including servers (file, print, mail and application servers) and also 2 VoIP phones?

Thank you in advance for your expert advise.

well, it is complicated.
I remember visiting around four years ago a cyber cafe in West Africa that had more than 50 machines, using Shiron RG384, with the generous bandwidth of.....32k up and 64k down..!
It depends on what application you allow on your network and what are the demands. Thats all.
download movies does not take the same bandwidth like writing a line in the messenger.
Its all depends what do you want to do and allow in your network thats all.

My opinion on bit rate requirements is that you should allow at least 10 kbit/s download rate per PC. Upload rate needed is about a quarter of this, assuming web browsing and emails.   An all VoIP phone shop site will need equal capacity up and down.

Avoid, if possible, web servers at your VSAT site which serve the whole internet.  Put them at main data centres instead and feed new content to them once only.

With optimium header compression implemented, such as on CDM 570L with IP module, VoIP needs 11 kbit/s both ways.   Without header compression and with inappropriate voice coding selected, the bit rate requirement could be 64kbit/s or more.  Configuring VoIP and Cisco routers requires considerable skill.

An SCPC link is simple and straightforward, a stream of bits simply goes from one end to the other and all is under your control.   You have dedicated service and full control of the technology, routing, configuration etc.  Price up to $700 per 100kbit/s per month. Using CDM570L modems you can improve the throughput using header and payload compression. QoS may be activated to prioritise voice packets ahead of email packets.   You can reduce monthly space segment costs by up to half if you use a bigger dish and 8-QAM or 16-QAM modulation.   Additional external equipment can improve matters further such as Performance Enhancing Proxy servers (ref Maxim Usatov above) and TCP accelerators (ref Pat Gannon above).

A DVB-S type link is a big download carrier, typically about 20 to 30 Mbit/s.  It is essentially a regular digital TV carrier modified to carry IP data.   It is shared with thousands of receive only sites who typically have a plug in DVB-S card in a PC and a receive only satellite TV dish.   It is possible to subdivide the high bit rate and blocks of (say) 256k might be assigned to a small service provider who can further resell this capacity to 1 end user or to many end users.   Service can be very cheap as the sharing ratio can be very high but with average bit rate per receive site very low.  You need to trust the provider regarding sharing ratios and router configurations.   Don't expect much help, or much of a service, if you are on a very low tariff.  If your monthly tariff divided by the number of PCs you have connected is less than $70, you are probably congested.   DVB-S2 is a recent enhancement to allow higher order modulation methods for larger receive dishes.  In a designed 2-way VSAT system, like LinkStar and Wildblue, the DVB-S2 modulation may be altered according to the weather to improve the bit rate in clear sky conditions.

For a customer with 50 to 70 PCs and two VoIP phones I would suggest SCPC both ways.  500kbit/s download and 200kbit/s upload.   Price up to $4900 per month, less with larger dish.

The ping results of several seconds are clearly very bad.  What results do you get doing tracert ?  Where is the delay occurring ?  Do the results improve if you disconnect your network?.   With a Cisco router you have lots of interface stats to investigate.  1 second delay on internal pings within your LAN is probably a fault you can resolve locally.

I have written a lot here.  Please don't let others be discouraged from providing alternative answers to the questions with their own ideas.  

Best regards, Eric.

traceroute output:

C:\>tracert yahoo.com

Tracing route to yahoo.com [216.109.112.135]
over a maximum of 30 hops:

 1    <1 ms    <1 ms    <1 ms  172.16.31.1
 2    <1 ms    <1 ms    <1 ms  172.16.20.161
 3    <1 ms    <1 ms    <1 ms  host-216-226-225-177.wdb.ses-americom.net [216.226.225.177]
 4  3504 ms  3918 ms  3779 ms  us-wdb-in2s4-1.ses-americom.net [216.226.226.237]
 5  4149 ms     *        *     66-178-112-81.reverse.newskies.net [66.178.112.81]
 6  3826 ms  3688 ms  3936 ms  host-66-133-55-189.wdb.ses-americom.net [66.133.55.189]
 7  3250 ms  3217 ms  3143 ms  xr01-f6-1.wdb.ses-americom.net [66.133.55.225]
 8  2962 ms  2622 ms  2401 ms  65.90.201.213
 9  3180 ms  2353 ms  2219 ms  p7-0.c0.nwyk.broadwing.net [216.140.10.13]
10  2017 ms  1993 ms  2159 ms  216.140.17.121
11  2740 ms  3205 ms  2976 ms  p0-2-0.a1.wash.broadwing.net [216.140.8.90]
12  2507 ms  2794 ms  2388 ms  216.140.8.230
13  2565 ms  2535 ms  2661 ms  67.99.66.10
14  2954 ms  3430 ms  4233 ms  ge-0-0-0-p100.msr1.dcn.yahoo.com [216.115.108.1]

15  3237 ms  3263 ms  3418 ms  ge10-2.bas1-m.dcn.yahoo.com [216.109.120.203]
16  4248 ms  4057 ms  4025 ms  w2.rc.vip.dcn.yahoo.com [216.109.112.135]
Thanks for your contributions. The following are results of my investigation based on issues you have raised.

The delay is within the router interface. From my (internal network) LAN i can ping all servers and the ethernet interface on the router at <1ms. From the router too, i can ping my next hop router at <1m. But when i log into the my router and try to ping the serial interface on the same router i get >1000ms!

Therefore the delay i am getting is partly resident in te router! There seems to be no faults on the serial interface from the statistics i have attached below. Pls have a look and confirm if there is issue on that statistic i have not taken into consideration.

From your advise, do you think a link budget of 500/200kbps with a higher compression rate be enough for about 70pcs (SCPS both ways) and can the CDM570L handle the functionality of my DVB? Note the on my present next work the CDM570L is configure for TX exclusively while the DVB is for Rx.

Trace complete.


Serial Int. details
Serial0/0/0 is up, line protocol is up
 Hardware is GT96K Serial
 Description: Connected to Internet
 Internet address is xxx.xxx.xxx.xxx/30
 MTU 1500 bytes, BW 256 Kbit, DLY 20000 usec,
    reliability 255/255, txload 65/255, rxload 65/255
 Encapsulation HDLC, loopback not set
 Keepalive not set
 Last input 00:00:10, output 00:00:10, output hang never
 Last clearing of "show interface" counters never
 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 609
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 30 second input rate 66000 bits/sec, 59 packets/sec
 30 second output rate 66000 bits/sec, 59 packets/sec
    1169296 packets input, 146868851 bytes, 0 no buffer
    Received 240 broadcasts, 0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    1169337 packets output, 146888695 bytes, 0 underruns
    0 output errors, 0 collisions, 5 interface resets
    0 output buffer failures, 0 output buffers swapped out
    0 carrier transitions
    DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

Thanks  so far.

glad to see youve already isolated some conegstion to your lan, i was going to suggest direct connecting to your modem to test your provider versus your lan.

otherwise, yea 70 users is quite a bit for 256k.

as for the bandwidth usage, other than p2p apps and viruses etc. once or twice i've come across equipment trying to auto update after hours causing some instances like this, especially if it's a pc with a full hard drive trying to download the same update over and over with no where to save it. same for a/v updates etc. try not to schedule them all at the same time.

All good suggestions and hopefully you have this worked out by now, would like to know if you have and if so what the problem was.

Anyway, off peak updates have been know to cause problems like this.

As for your high latency on the ping test, if your ping test was conducted when you were at your CIR, the ping is treated as a best available peice of data so it wil not have any priority in ensuring that it actually gets through. That being the case it will be pushed to the back of the line causing an indication of high latency.