VSAT troubleshooting/Bandwidth utilization issues?

Guys i need your help on this.

I have notice that on my satellite link that my daily bandwidth utilisation hits 240kbps on a link which is a 256kbps symmetric SCPC link with a DVB modem that recieves and a comtech 570L modem for transmit. My service provider keeps saying that its from my network but my router activity does not support that argument.

From my network setup, a router separates the vsat network from the local area network. this internal router (LAN) gets traffic from the LAN and delivers to the firewall which then passing traffic to the router hosting the vsat. On the internal router the network traffic is almost at zero level after office hours (that is when users have closed for the days work).

On the other hand, i noticed that during peak periods (when users are online) the software used to monitor (cacti) indicates that the link performance hits an average of 100kbps whereas at off peak it hits 200kbps and above, thats almost working on full capacity.

I would please need suggestions on this issues i have stated and what i can do at my end to trouble shoot my VSAT link so that i can present more facts to my service provider shld the problem be from them.

Secondly i would like to know why my link is slow when uses are online. we have less than 70pcs on this link in a fully and welled segmented network.  All users are on Vlans (about 5vlans) and we use high end switches for inter vlan routing and 3550 cisco switches at the distribution level, there are no hubs in this network.

Any more information on this network to help in resolving this problem will be made available.

Thanks for your anticipated suggestions.

just couple of fast notes:

1. From your description, it sounds like your link does reach its limits: If you see on the MRTG (or any other sampling application you are using) that it reaches 240kbps, remember that it is just a sampling and the actual usage is always bit more. How much more - it depends on couple of factors, one of them is the sampling interval.
2. I agree with Eric that you should take a deep look for trojans and worms on all the pc's of your network. I suggest you to troubleshoot it doing this:
First, connect all your machines to the network and turn them on, open all the applications (like explorere, messenger and so on) but dont do anything - dont browse or touch the pc. Check the bandwidth usage. If you locate a problem, try to disconnect half of the machines to localize which machine(s) are contaminated. Once you will pinpoint the problem you will be able to solve it as well.

Good luck,
Nimrod

well you can check the physical parameters, like Ebno on both sides, CRCs and so on.

Another two checks you can do is:
1. try to ping -w 500 and -l of, lets say, 1000 for a long time and see how much time outs do you have and what is the pattern of the time outs.
2. check the counters on your router, erase them and check if they are increasing.

Regarding ethereal, you need to connect your monitor PC in parallel with the traffic you want to monitor. Two ways:
1. Find the existing cable to be monitored and make a break out.  Put a passive hub with a side cable to your PC.
2. If one end of the cable goes to or comes from a Cisco router or switch and you know how to configure it, it may be possible to use a spare port and bridge it across in parallel with the port to be monitored.  Cisco have some good traffic monitoring facilities, but be very careful about any Cisco reconfiguration.    If you have a switch on each floor then I guess you might be able to use a spare port to do a monitor on any particular PC connected.   If you have one main switch connected to each floor then at that switch you may be able to use a spare port to monitor any whole floor.   The connection between your firewall and the satellite modem may need solution 1 above, using a passive hub.

Regarding the Eb/No of your transmit carrier (from the 570L) you need to call the hub and see what is the reading at their end at different times of day.  It is possible that your transmit carrier is low power (moisture in cable, antenna moved slightly etc).  It could get worse at night if the satellite moves out of the beam during that period.  This can happen if the dish is mispointed.   During clear sky conditions at both ends you should have a good rain margin of perhaps 4 or 5 dB.  During clear sky conditions at both ends try reducing your transmit level by 3 dB.  The network should continue operating normally.  If is fails then you have less than 3 dB margin which is not good enough.  
Contact the hub and re-peak your dish and adjust your transmit level, while talking to them, till the Eb/No is correct at the hub.

Best regards, Eric.

The firewall is not blocking the ping. If i run a normal ping you get this

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>ping 66.94.234.13 -t

Pinging 66.94.234.13 with 32 bytes of data:

Request timed out.
Reply from 66.94.234.13: bytes=32 time=4191ms TTL=47
Request timed out.
Request timed out.
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3217ms TTL=47
Reply from 66.94.234.13: bytes=32 time=2812ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3052ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3337ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3491ms TTL=46
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3355ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3361ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3568ms TTL=46
Reply from 66.94.234.13: bytes=32 time=2907ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3331ms TTL=47
Reply from 66.94.234.13: bytes=32 time=2733ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3356ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3541ms TTL=46
Reply from 66.94.234.13: bytes=32 time=3654ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3992ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3694ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3793ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3506ms TTL=47
Reply from 66.94.234.13: bytes=32 time=3749ms TTL=47
Reply from 66.94.234.13: bytes=32 time=4334ms TTL=47
Request timed out.
Reply from 66.94.234.13: bytes=32 time=3808ms TTL=46

Ping statistics for 66.94.234.13:
    Packets: Sent = 28, Received = 22, Lost = 6 (21% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2733ms, Maximum = 4334ms, Average = 3490ms
Control-C
^C

It sounds as though you have a DVB/SCPC service, not SCPC/SCPC.  Your DVB download may be oversubscribed more than you expect.

I did not see any mention of TCP/HTTP Acceleration.  If you don't have TCP Acceleration it's quite normal that you would not be able to fully utilize the circuit.  The TCP protocol is a guaranteed delivery protocol.  It sends a little data, stops, waits for an ACK from the remote receiver, then doubles up the amount of data, sends, stops, etc.  In this manner it learns the speed and congestion of the link.  Since satellite delay is at least 1/2 second, the TCP protocol interprets this as a very slow or congested circuit.  Regardless of how much bandwidth you have, you are unlikely to get much more than about 70 - 90 Kbps per TCP session.  Of course multiple sessions will allow you to fill up the circuit, but given all the TCP overhead, it's unlikely you'll ever see full use of your circuit.  With TCP Acceleration, you may get throughput in excess of the link speed because a lot of the TCP overhead is stripped off.

Comtech has an IP module that provides QoS and other features, but it does not include integrated TCP Acceleration.  For this you need an external device.  It also helps if your network operator is providing TCP/HTTP Acceleration on the DVB portion of the link.

HTTP Acceleration is also useful to speed up browsing.  Every single piece of content on a web page must go  through a 3-way handshake process.  Again, with the 1/2 second latency, this can really slow down web downloads between pieces of content. 

See someone like Packeteer/Mentat for a TCP/HTTP Acceleration solution, or perhaps UDCast.  There are others out there, that I'm sure people on the forum can point you to.  Most 2-way services such as iDirect have this built in. 

Hope this helps,

Pat

Thanks for all your response to this post. Maxim Usatov pls do send me the PEP trial. I have also considered your suggestions and but would like to know how to calculate the capacity if satellite you need for a given number of PCs. Over here like i stated before we have about 70pcs, 50 of which can be online simultaneously.

Another problem i have ji=ust noticed is the high latency on my network. a roundtrip delay returns at >3000ms sometimes 4000ms.

An investigation revealed that a ping from my fast-ethernet port to the serial port of my boarder router returns at >1000ms! These 2 interfaces are on the same router (cisco 3745 router).

Also in the past week i have observed from my Cacti software (a bandwidth utilization monitoring tool) that i am transmitting at over 307kpbs (i have not upgraded to this service rate) whereas i am subscribed for a bandwidth of 256/256kpbs. I have reported this to the service provider but nothing has been done about it and besides the latency is still high.

On the DVB, is there a way i can determine the overbooking or oversubscription rate from the DVB box or is it only the provider that has access to such information? And what’s the main difference between DVB/SCPC and SCPC/SCPC?

Pinging yahoo.com [216.109.112.135] with 32 bytes of data

Reply from 216.109.112.135: bytes=32 time=3989ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3960ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4049ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4288ms TTL=53
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4332ms TTL=54
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4274ms TTL=53
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4094ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3859ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3658ms TTL=54
Reply from 216.109.112.135: bytes=32 time=3837ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3771ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4057ms TTL=53
Reply from 216.109.112.135: bytes=32 time=4157ms TTL=53
Reply from 216.109.112.135: bytes=32 time=3816ms TTL=54
Reply from 216.109.112.135: bytes=32 time=4015ms TTL=54
Request timed out.
Reply from 216.109.112.135: bytes=32 time=4127ms TTL=53

Ping statistics for 216.109.112.135:
    Packets: Sent = 20, Received = 16, Lost = 4 (20% loss
Approximate round trip times in milli-seconds:
    Minimum = 3658ms, Maximum = 4332ms, Average = 4017ms

Expecting your professional advise as usual.

The general idea here seems to portary that my network is overloaded. can someone pls suggest how i can calculate the bandwidth that will be appropriate for 70 online pcs including servers (file, print, mail and application servers) and also 2 VoIP phones?

Thank you in advance for your expert advise.

My opinion on bit rate requirements is that you should allow at least 10 kbit/s download rate per PC. Upload rate needed is about a quarter of this, assuming web browsing and emails.   An all VoIP phone shop site will need equal capacity up and down.

Avoid, if possible, web servers at your VSAT site which serve the whole internet.  Put them at main data centres instead and feed new content to them once only.

With optimium header compression implemented, such as on CDM 570L with IP module, VoIP needs 11 kbit/s both ways.   Without header compression and with inappropriate voice coding selected, the bit rate requirement could be 64kbit/s or more.  Configuring VoIP and Cisco routers requires considerable skill.

An SCPC link is simple and straightforward, a stream of bits simply goes from one end to the other and all is under your control.   You have dedicated service and full control of the technology, routing, configuration etc.  Price up to $700 per 100kbit/s per month. Using CDM570L modems you can improve the throughput using header and payload compression. QoS may be activated to prioritise voice packets ahead of email packets.   You can reduce monthly space segment costs by up to half if you use a bigger dish and 8-QAM or 16-QAM modulation.   Additional external equipment can improve matters further such as Performance Enhancing Proxy servers (ref Maxim Usatov above) and TCP accelerators (ref Pat Gannon above).

A DVB-S type link is a big download carrier, typically about 20 to 30 Mbit/s.  It is essentially a regular digital TV carrier modified to carry IP data.   It is shared with thousands of receive only sites who typically have a plug in DVB-S card in a PC and a receive only satellite TV dish.   It is possible to subdivide the high bit rate and blocks of (say) 256k might be assigned to a small service provider who can further resell this capacity to 1 end user or to many end users.   Service can be very cheap as the sharing ratio can be very high but with average bit rate per receive site very low.  You need to trust the provider regarding sharing ratios and router configurations.   Don't expect much help, or much of a service, if you are on a very low tariff.  If your monthly tariff divided by the number of PCs you have connected is less than $70, you are probably congested.   DVB-S2 is a recent enhancement to allow higher order modulation methods for larger receive dishes.  In a designed 2-way VSAT system, like LinkStar and Wildblue, the DVB-S2 modulation may be altered according to the weather to improve the bit rate in clear sky conditions.

For a customer with 50 to 70 PCs and two VoIP phones I would suggest SCPC both ways.  500kbit/s download and 200kbit/s upload.   Price up to $4900 per month, less with larger dish.

The ping results of several seconds are clearly very bad.  What results do you get doing tracert ?  Where is the delay occurring ?  Do the results improve if you disconnect your network?.   With a Cisco router you have lots of interface stats to investigate.  1 second delay on internal pings within your LAN is probably a fault you can resolve locally.

I have written a lot here.  Please don't let others be discouraged from providing alternative answers to the questions with their own ideas.  

Best regards, Eric.

glad to see youve already isolated some conegstion to your lan, i was going to suggest direct connecting to your modem to test your provider versus your lan.

otherwise, yea 70 users is quite a bit for 256k.

as for the bandwidth usage, other than p2p apps and viruses etc. once or twice i've come across equipment trying to auto update after hours causing some instances like this, especially if it's a pc with a full hard drive trying to download the same update over and over with no where to save it. same for a/v updates etc. try not to schedule them all at the same time.