Packet Over SONET: Good For Nothin’?

Here’s a question: who uses Packet-Over-SONET interfaces and why the heck would you ever want to do that? At first glance, POS looks ridiculous because of the price. The following is a sample of the Cisco POS Shared Port Adapters available on the ASR1000 platform:

Part # Desc. MSRP Bandwidth Price
per Meg
SPA-2XOC3-POS 2-port OC3/STM1 POS SPA $ 11.5k

300 Mbps

 $ 38.33
SPA-1XOC12-POS 1-port OC12/STM4 POS SPA $ 14.0k 601 Mbps  $ 23.29
SPA-1XOC48POS/RPR 1-port OC48/STM16 POS/RPR SPA

 $ 38.0k

2405 Mbps  $ 15.80
SPA-OC192POS-XFP 1-port OC192/STM64 POS/RPR XFP Optics

 $ 125.0k

9620 Mbps  $ 12.99

By the time you get to the OC192 interface, the price-per-meg has finally dropped out of the stratosphere.  If you are looking at POS you may be building an ISP backbone – so you’re probably deploying 10 Gbps links anyway.  I work for an ISP, and we don’t build anything with less than 1 Gbps.  Of course you’re dropping a cool $125k for a single interface.

Difference #1

In contrast, a 10-Gigabit Ethernet SPA (SPA-1X10GE-L-V2) is $2 per meg!  Whoa.  What gives?  A 10GE interface runs at approximately the same speed as an OC192.  Why would you pay 13X more for the same speed?

Difference #2

There are operational differences between Ethernet and POS.

Let’s say you have a /30 network provisioned across a PTP Ethernet link.  You still need to know the MAC address of the device on the other side.  That means you will have to perform an ARP.  That might take a few hundred mili-seconds but you only have to do that once because the ARP cache should stay fresh as long as the link is seeing traffic.

POS uses PPP to encapsulate the IP traffic onto the SONET framing.  As soon as the interface and the line protocol come up, it’s ready to go – no ARP needed.  The advantage here is definitely with POS.

Difference #3

OSPF will handle an Ethernet link as a multi-access network.  Each router on either end will flood a Type-1 LSA across the entire area to announce they know about the /30 transit network.  The link-state ID will be the interface IP address of the designated router.  There will have to be an election to determine who will be the designated router.  Each router will also flood a Type-2 LSA that lists all routers on the segment (a grand total of two).  So, that’s four flooded LSAs and one election that has to be dispensed with.

OSPF on POS acts as a true PTP link.  Each router floods a Type-1 LSA advertising their knowledge of a transit network.  No Type-2 is flooded and no election occurs.  Your OSPF database will be smaller and your Dijkstra’s algorithm will run faster.  Advantage: POS.

Summary

POS has real advantages over Ethernet, but does it justify the cost?  I’m saying it doesn’t.  A link may be revenue generating with lots of customers on it, but the benefits are too minor to add up.

Has anyone out there done any POS WAN links?  What were your experiences?  Did I miss something here?  Feel free to chime in.

Dan Massameno
Dan Massameno is the president and Chief Engineer at Leaf Point, a network engineering firm in Connecticut.
Dan Massameno

Latest posts by Dan Massameno (see all)

  • ISP Geek

    Why not just change the link type to PTP for the Ethernet OSPF and then the election is no longer an issue?

    • Anonymous

      Your PTP on Ethernet idea for OSPF is awesome!  I never thought of changing the network type on an Ethernet interface.  I just tried a couple of quick things in the lab and it looks like it stops the election process and the Type-2 LSA is gone, but has some interesting side effects.  I’m going to look at this a little bit more and maybe post another blog entry just on this topic.  Thanks.

      • http://twitter.com/mellowdrifter Darren O’Connor

        I know this thread is a bit old, but had to comment. We use network type point to point on all our p2p ethernet links. Always have and never had a problem (why should there be)

        Ethernet wins a lot these days as Juniper’s MX line and Brocade Netiron line as exclusively ethernet. Only the older M/T models did POS

  • Guest

    Its the overhead….

  • http://cdplayer.livejournal.com/ Dmitri Kalintsev

    The greatest difference is that a SONET/SDH link (that POS interfaces typically use) always comes with proper OAM and really quick fault propagation mechanism.

  • Rob Heath

    Static ARP and a bit of OSPf config will sort out the last 2 differences. Tools like BFD and Y.1371 will provide a replacement for the alarm structure inherent in SDH, although it comes with some overhead. If you are using routers that can’t handle the signalling overhead to enable a timely failover, you are probably using routers that aren’t suitable to handle 10G. The STM1 and 4 interface might suit service providers that don’t need a huge amount of bandwidth and have deployed SDH networks that can provide cheap backhaul.

    • http://cdplayer.livejournal.com/ Dmitri Kalintsev

      Rob,

      You may want to check a minimum configurable interval for Y.1731/802.1ag CC messages on your particular equipment; there could be surprises. Can’t comment on BFD, but I doubt it can match that of SONET/SDH without the risk of false positives and subsequent link flaps.

      Whether you actually need it that quick is a different story, of course.

    • Anonymous

      Static ARP would be difficult to maintain but certainly solves the ARP problem (if it is a problem).  I’m going to do some lab work on possible OSPF config changes (configuring it as a PTP link).

  • http://networkdongle.wordpress.com Garry Baker

    i think the key is sonet/sdh is different then ethernet at layer one and because ethernet “won” a long time ago that is what most people now a days defer to…

    technologies that were built a long time ago still have a purpose at times today, as always “it depends”

    i used to work on fddi networks that have dual channels and no one even know we had a problem until the one of the 2 would go down, then everyone was like what happen fddi has dual channels, oh look at that the backup channel has been down for 6 months…

    but we had to tear it all out to put in ATM and LANE cause it was the next big thing, about a zillion dollars and 2 years later putting in gigabit ethernet switches…

    does sonet have an ethernet over sonet protocol? anyone use that vs packet over sonet?

    but the price comparison is interesting…

  • http://networkdongle.wordpress.com Garry Baker

    i think the key is sonet/sdh is different then ethernet at layer one and because ethernet “won” a long time ago that is what most people now a days defer to…

    technologies that were built a long time ago still have a purpose at times today, as always “it depends”

    i used to work on fddi networks that have dual channels and no one even know we had a problem until the one of the 2 would go down, then everyone was like what happen fddi has dual channels, oh look at that the backup channel has been down for 6 months…

    but we had to tear it all out to put in ATM and LANE cause it was the next big thing, about a zillion dollars and 2 years later putting in gigabit ethernet switches…

    does sonet have an ethernet over sonet protocol? anyone use that vs packet over sonet?

    but the price comparison is interesting…

  • mkashin

    From my point of view Sonet/SDH has two major advantages. 
    1) The distance. Remember that these technologies were originally invented to be used on SP long-distant networks. Where you’ll never be able to use Ethernet.
    2) Error detection/correction/recovery. Being a transport technology it is a MUST have. 
     
    And BTW SDH/Sonet DO have a GFP (generic framing protocol) to transfer ethernet over

    • Dan Massameno

      In terms of distance, you can probably find some exotic SONET transceivers that beat 10GE transceivers.  But, the current Cisco transceivers for both SONET and 10GE both rate at 80km.  The SONET part would be the XFP-10GZR-OC192LR.  The 10GE part would be the SFP-10G-ZR.  The MSRP on both are $16,000.

  • Ameer Abbas

    how we can check the transmission errors on 10GE router interface which is connected with transmission system SDH.

  • BobDobb

    I think that the _real_ big difference is APS or Automatic Protection Switching and SONET alarms. With SONET/SDH you get _very_ detailed alarm indications and it’s very easy to see what’s going on at each level in the hierarchy, so you can see which end of a link is going bad, or if you have multiplexers at which level your mux is going bad. With APS, you accidently pull a cable out and 30ms later all of your traffic is on the backup fiber with no upper layer intervention required at all. Ethernet just cannot do that. There are new specs for “Ethernet OAM” which is trying to replicate a lot of this functionality with Ethernet, but I don’t know how widely deployed any of it is.