Trusting in Non-Linearity as the FNG

I’m officially two months in at Salesforce.com and it’s really kind of neat to be in a non-telecom environment for the first time. I’ve had to really get settled with the idea that I am not going to be “fully productive” or feel the confidence of “understanding the unique value that I own” in my place of work for some time. (I’ve known this all along but getting the gut to catch up with the head is hard work sometimes.)

This is the plight of the new guy. There’s just too much you don’t know and the only way to get to know it is to buckle down and start doing stuff… probably badly. There are three things I am trusting here. I have trust in myself because I have managed to become top-tier effective wherever I have applied myself. I have trust that my team is going to give me the feedback I need so that I can work with them in a way that is congruent with the goal and with their work in progress. And I trust in the non-linear nature of this kind of learning.

What do I mean by non-linear? Don’t make too much fun of my “line”. I was too lazy to use a straight-edge but not too lazy to use a scanner.

Anyway, I trust that as I apply myself to my work, the things I learn will gather, have sex in my brain, and multiply such that I can understand much more than two times what I can understand today when we reach the four months mark. And, hopefully, that translates into smarter results, better output which factors in the many requirements from our many constituents and clients.

Vagrant and Juniper's Firefly Perimeter

Looks like Juniper is working hard to get their Firefly perimeter usable on Vagrant. See also, the firefly-packer github for info and caveats.

fluong@fluong-ltm:vagrant$ mkdir firefly010
fluong@fluong-ltm:vagrant$ cd firefly010/
fluong@fluong-ltm:firefly010$ ls
fluong@fluong-ltm:firefly010$ vagrant init juniper/ffp-12.1X46-D20.5
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
fluong@fluong-ltm:firefly010$ ls
Vagrantfile
fluong@fluong-ltm:firefly010$ vim Vagrantfile
fluong@fluong-ltm:firefly010$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'juniper/ffp-12.1X46-D20.5' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'juniper/ffp-12.1X46-D20.5'
    default: URL: https://vagrantcloud.com/juniper/ffp-12.1X46-D20.5
==> default: Adding box 'juniper/ffp-12.1X46-D20.5' (v0.1.2) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/juniper/boxes/ffp-12.1X46-D20.5/versions/3/providers/virtualbox.box
    default: Progress: ##########################################################==> default: Successfully added box 'juniper/ffp-12.1X46-D20.5' (v0.1.2) for 'virtualbox'!
==> default: Importing base box 'juniper/ffp-12.1X46-D20.5'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'juniper/ffp-12.1X46-D20.5' is up to date...
==> default: Setting the name of the VM: firefly010_default_1411482071141_14275
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: No guest additions were detected on the base box for this VM! Guest
    default: additions are required for forwarded ports, shared folders, host only
    default: networking, and more. If SSH fails on this machine, please install
    default: the guest additions and repackage the box to continue.
    default:
    default: This is not an error message; everything may continue to work properly,
    default: in which case you may ignore this message.
fluong@fluong-ltm:firefly010$ vagrant ssh
--- JUNOS 12.1X46-D20.5 built 2014-05-14 20:38:10 UTC
vagrant>

vagrant> show interfaces terse
Interface               Admin Link Proto    Local                 Remote
ge-0/0/0                up    up
ge-0/0/0.0              up    up   inet     10.0.2.15/24
gr-0/0/0                up    up
ip-0/0/0                up    up
lsq-0/0/0               up    up
lt-0/0/0                up    up
mt-0/0/0                up    up
sp-0/0/0                up    up
sp-0/0/0.0              up    up   inet
                                   inet6
sp-0/0/0.16383          up    up   inet     10.0.0.1            --> 10.0.0.16
                                            10.0.0.6            --> 0/0
                                            128.0.0.1           --> 128.0.1.16
                                            128.0.0.6           --> 0/0
dsc                     up    up
gre                     up    up
ipip                    up    up
lo0                     up    up
lo0.16384               up    up   inet     127.0.0.1           --> 0/0
lo0.16385               up    up   inet     10.0.0.1            --> 0/0
                                            10.0.0.16           --> 0/0
                                            128.0.0.1           --> 0/0
                                            128.0.0.4           --> 0/0
                                            128.0.1.16          --> 0/0
lo0.32768               up    up
lsi                     up    up
mtun                    up    up
pimd                    up    up
pime                    up    up
pp0                     up    up
ppd0                    up    up
ppe0                    up    up
st0                     up    up
tap                     up    up
vlan                    up    down

Kicking off My Mini-Sabbatical... Why I Left and What I Want to Build

I left my position at Juniper on Friday. It was a happy-but-bittersweet parting because I really liked my team there. I received good cheer and wishes for great success and votes of confidence from nearly everyone. It’s really heartwarming and I get a bit misty thinking about it.

People who know me in my life as a computer geek know me as the network guy who will use any excuse to write another script. This is true. I’ve always been fascinated with the power of computers to allow a person to do more than he would be capable of alone. They say time is money. I say that well-directed code is time.

At early points in my career I remembering adding VBA logic to my Word and Excel documents and developing templates in Visio to make my workflow easier. This is the same thing as what I do as a network engineer: Remove the boredom of repetition.

Repetition does have a purpose. I used a lot of it when I was preparing for my JNCIE-SP exam. With a little study, I already knew the concepts for that test. The rest was a matter of practice and getting faster at doing things.

But once you have achieved the goal of learning or habituating something unfamiliar, the repetition no longer serves a purpose to you and it starts to make sense to code it away. This has always been a part of my work. When I am being self-deprecating and humorous, I say it is because I am “lazy”. When I am glorifying myself, I say I’m working “smarter”. The truth is somewhere in the middle.

I am moving on to join Salesforce.com to steep myself in the datacenter environment and to help them to build systems that will deploy and support their networks. For all of the scripts that I have written so far, they have managed to be useful tools with longevity but have not yet added up to a sustainable and supportable system that would stand long after I leave. This is what I hope to achieve.

This is an exciting time for me and other code-friendly network operators/engineers/architects. And it is a tough challenge to tackle. A lot of systems people consider the network as a black box, as a given, as a fundamental requirement that you do not mess with. Indeed, even some level of network state is required to configure the network devices effectively.

People often wonder in their blogs why the network has lagged servers in becoming agile, virtualized, and orchestrated. But servers couldn’t have become virtualized and orchestrated without the network already there connecting everything.

So deploying servers depends on the network being there. And complex network deployment depends on the of the network being there. This makes deploying the final network state a meta-problem. How do you deploy the network and guarantee you’re not breaking it and your ability to fix it? How do you change state as frequently as needed without killing your network devices by a thousand tiny cuts?

I don’t feel defeated by these thoughts. These are the challenges. These are the interesting sorts of problems to solve. And I intend to spend part of this week thinking about it, and part of it resting, and part of it just reading, and part of it writing posts like this one.

See you soon.

-Franco

Fetching Configuration Subtrees Using Netconf

It took me a bit of struggling to figure out how to pull configuration subtrees via NetConf. Part of this was because I was using tags instead of . But also, I was hitting 256 character line-limits.

I’ll work that out later… the main reason for this post is to document what I found on how to get configuration subtrees. This particular example fetches the subtree for a specific neighbor under [protocols l2circuit].

To structure the rpc request, we use the command “get-config” and specify a source of candidate. Then we add a “filter” tag with an attribute of @type=“subtree” and under that we can specify the configuration subtree as xml tags.

<rpc message-id="1002 Tue Jun 17 23:25:48 -0400 2014">
    <get-config>
        <source><candidate/></source>
        <filter type="subtree">
            <configuration>
                <protocols>
                    <l2circuit>
                        <neighbor>
                            <name>192.168.104.69</name>
                        </neighbor>
                    </l2circuit>
                </protocols>
            </configuration>
        </filter>
    </get-config>
</rpc>
]]>]]>

This is the reply I got.

rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/12.3R4/junos" message-id="1002 Tue Jun 17 23:25:48 -0400 2014">
    <get-config>
        <source><candidate/></source>
        <filter type="subtree">
            <configuration>
                <protocols>
                    <l2circuit>
                        <neighbor>
                            <name>192.168.104.69</name>
                        </neighbor>
                    </l2circuit>
                </protocols>
            </configuration>
        </filter>
    </get-config>
</rpc>
<data>
<configuration xmlns="http://xml.juniper.net/xnm/1.1/xnm" junos:changed-seconds="1402600329" junos:changed-localtime="2014-06-12 19:12:09 UTC">
    <protocols>
        <l2circuit>
            <neighbor>
                <name>192.168.104.69</name>
                <interface>
                    <name>ae0.3909</name>
                    <virtual-circuit-id>55509</virtual-circuit-id>
                </interface>
                <interface>
                    <name>ae0.3933</name>
                    <virtual-circuit-id>55533</virtual-circuit-id>
                </interface>
                <interface>
                    <name>ae0.3959</name>
                    <virtual-circuit-id>55559</virtual-circuit-id>
                </interface>
                <interface>
                    <name>ae0.3983</name>
                    <virtual-circuit-id>55583</virtual-circuit-id>
                </interface>
                <interface>
                    <name>ae0.3991</name>
                    <virtual-circuit-id>55591</virtual-circuit-id>
                </interface>
            </neighbor>
        </l2circuit>
    </protocols>
</configuration>
</data>
</rpc-reply>
]]>]]>

My particular application in this case was that I wanted to get the interface associated with a certain virtual-circuit-id and neighbor. This made it easy to collect what I wanted and to also present the xml for logging.

Data Center Networking @ Facebook by David Swafford

I’ve been looking into data center fabrics and how you handle the scale of large networks lately so I decided I should take some time today to fully view the presentation(video and PDF) by David Swafford which he did at NANOG 59 late last year.

I met David Swafford when Facebook came to town for MPLS 2013. He was a really cool guy. I was inspired even at the time by hearing the way that they are going about support their networks. Very smart!

I took away a lot of nuggets from watching it. Here are a few:

  • Assume we can’t trust any rack
  • We can’t trust networking boxes either
  • Backbone devices are powerful in the wrong ways for a data center. They can handle many routes but don’t have the desired port density.
  • Going from 2 large leaf switches to many smaller leaf switches allows you to move from 1+1 to N+1.
  • Beware of silent failures by complex networking devices. They are hard to detect, BTW.
  • Automating ToR switch upgrades and handing a “push-button” interface to the service owners helped to remove the roadblocks for full upgrades of ToR switches. (I found it analogous to app upgrades on my phone)
  • They even scripted many parts of the process, such as determining who the on-call is for a given group at a given time. Fascinating.

Monitor all the things:

  • interface statistics and state
  • bgp statistics and state
  • FIBs
  • TCP retransmits

Respond to your Alerts with Automation:

  • FBAR stands for Facebook Automation Remediation
  • Receive Alert, login to device, verify still down, either ignore or remedy.

He also covers a lot of thoughts on engineers that automate:

  • Spend less time doing repetitive tasks
  • Spend more time solving interesting problems or learning

His final challenge: What would you do if you weren’t afraid?