Verizon Networks

Principal Network Security Engineer, Network Intelligence

• Develop and deliver software and systems for high-performance network management and monitoring of Verizon's Global IP network spanning 150 countries. Develop next generation network element data collection and management software (Hyrda) in Go delivering order-of-magnitude speed gains; from tens of minutes to seconds for large networks like AS701, AS702, AS703 and others.
• Develop systems and software for integrating global logging and monitoring into the Network Intelligence (Network Security) data center.
Build integrations and automation such as Splunk alarms and triggers that drive further automation to make real-time network changes such as automatic creation of MPLS LSPs on bandwidth alarms. Development and management of intermediate systems to support this automation such as Redis and custom HTTP/JSON interfaces and custom loggers such as this Python code that can generate RFC5424 compliant messages with no change to the standard Python logging method call signatures: MiniRFC5424SysLogHandler
• Rapid development and deployment of software and automation to fix emergent network problems such as this outage: BGP super-blunder: How Verizon today sparked a 'cascading catastrophic failure' that knackered Cloudflare, Amazon, etc
• Continue with overlapping responsibility for prior projects as a Principal DevOps Engineer for Verizon's Tier III Network Operations Center, Global Data Network Management (GDNM) group. Extensive automation, network and systems management including automation, deployment and management of their next generation Software Defined Network implementation based on Juniper Network's Northstar Controller infrastructure via automated Linux deployment and management on Dell, Cisco Unified Computing Environment (UCS) hardware and VMWare/ESXi.

Verizon Networks

Principal DevOps Engineer, Network Operations and Engineering

• Desgin, develop and deploy DevOps systems for next generation software defined network (SDN)/Juniper Northstar Controller infrastucture, providing the control plane intelligence for Verizon Business' Converged Core network. Implement automated deployments of Redhat/CentOS Linux Operating Systems supporting this infrastructure. Automate interaction with Dell iDRAC racadm command line interface (CLI) via Secure Shell (SSH) and Python pexpect, and via the Redfish HTTP/REST API for server low-level (RAID, firmware, CPU, PSU etc) configuration deployment and compliance checks.
• Design, develop and deploy DevOps Linux Infrastructure to support the NOC/NMC's management of Verizon Business' Global IP Networks. Built virtualized multi-machine Linux clusters for hosting network management, monitoring and reporting infrastructure, as well as critical NOC tools, including Web integration. Built automated deployment of VMs on Proxmox hypervisors, running on Cisco UCS hardware, with integrated life-cycle systems management through ISCONF (http://www.infrastructures.org). Built standardized images and OS deployment automation using Kickstart, DHCP/PXEBoot, with versioning via Git.
• Design and deploy web infrastructure with development, test and production web servers fronted with NGINX to allow flexible reverse proxy to multiple backends including Apache, Ruby Puma (Oxidized), Zope, and Python WSGI instances.
• Extensive development of critical network management tools in Python and Perl to support automation for key network management efforts such as MPLS/LSP generation and deployment (labeled BGP, meshing new sites), cBGP meshing and demeshing, router maintenance pre/post maint verifications.
• Extensive migration of legacy Perl tools for router automation to migrate from the clear text Telnet protocol, to Secure Shell connections, and migration of web tools to support encrypted web (HTTPS/TLS) connections. Write new versions of these tools in Python, factoring out components in to reusable libraries to support better modularity and reuse of software components and higher quality. Deployment of these tools on new Linux systems, as well as migration to new IP NOC Solaris servers. New libraries include:
  • Automated SSH and Out-of-band (OOB) console logins to multiple platforms including JUNOS, IOS, IOSXR, TiMOS and Dell iDRAC, as well as smart console connections that can detect if they have connected to the wrong instance (eg backup RP on IOSXR), and break the connection rather than getting hung. Code is factored to be used either from CLI, or as a reusable Python library
  • Automated Traffic Removal from Juniper routers through automation of the procedures in the Global Data Network Management(GDNM) Method of Procedure (MOP) 1125 "Traffic Removal for Juniper Routers", and built modular library components in Python supporting this, including:
    • Automated test of OOB console prior to maintenance, and automated OOB router login, configuration parsing and representation of key information into Python objects.
    • Intelligent analysis of the Interior Gateway Protocol IGP (ISIS) state, and the router's upstream and it's upstream, with cross plane analysis to ensure operational redundancy prior traffic removal.
    • BPG neighbor connections, and state.
    • Determine VoIP peers and analyze APS (Automatic Protection Switching) with automatic switch of Working and Protect circuits.
    • VRRP state with master/backup switch-over and priority normalization.
    • Detect and shutdown BFD (Bidirectional Forwarding Detection) to packet voice gateways (PVGs).
    • Ensure Private IP (PIP)/SIG Route Reflector connectivity will not be lost.
    • Ensure regions with only two egress circuits are not isolated, and all other steps required by the MOP.
    • Integration with MASTARS maintenance system with incorporation of pre-maintenance and post-maintenance router configuration and state comparisons (maintdiff).
    • Tracking of the state of the maintenance via storage of Python objects into a Zope Object Database (ZODB) which persists arbitrarily complex Python objects directly, facilitating more rapid software development without the need to translate datastructures to and from a relational model and to re-translate for iterative changes in the software data model.
• Troubleshooting and support of several XO Communications' tools such as OSPF costing and audit (Perl), as well as assistance with systems administration. Migrated XO's Rancid router config archive tool from failing hardware, to Oxidized, and incorporated VZ network elements with a Web archive and diff interface.
• Deployment of mechanisms for software developed on new Linux systems to access critical legacy database systems such as UUNET_LIVE/Sybase and MASTARS/Oracle via multiple languages (Perl, Python and Java JTDS).
• Extensive one-off tools development for special projects including LATAM MPLS mesh, border router (BR) flap damping, detection of various configuration anomalies (HW specific config compliance e.g. FPC/MPC/DPC sampling-instance, identification of customers potentially impacted by maintenance as existing tool not accurate, and many others)
• Instituted best practices in software development and systems management using Git version control, coding style (PEP8), and literate programming practices via Python documentation features (Docstrings, reStructuredText) and generation and publishing of HTML documentation using Sphinx. Used these mechanisms and wrote extensive documentation of new software including generation of web documentation, eg: http://networkops.vzbi.com/doc/tactools, https://networkops.vzbi.com/doc/starman.
• Instituted a "Request for Automation" requirements process modelled on RFCs and PEPs in an environment where there were no documented requirements, in order to get the right solution delivered as fast as possible, raise awareness and eliminate duplicated code and effort and move common functions into a smaller reusable codebase for better manageability and higher quality. (https://networkops.vzbi.com/doc/RFA/RFA0.rst, https://networkops.vzbi.com/doc/RFA/RFA1.rst)
• Provide consultation to other VZB departments on automated Operating System (Linux) deployment, and management, along with extensive support, training and mentoring of network engineers in the IP NOC, Maintenance Engineering, and vendors, in areas related to UNIX, programming, scripting and automation.
Adaptation of Python objects to Zope so that tools can work interchangeably via the UNIX CLI or the Web so network engineers can use Zope's through-the-web (TTW) interface, including creating Web content without knowledge of web technologies like HTML, CSS and Javascript.

Self Employed

Residential Construction and Remodelling

• Renovation and sale of a single family residence. Demolish, design and install new kitchen (cabinets, gas appliances, countertops, sink, tile backsplash, hardwood flooring), three bathrooms (tile, electric, vanities, plumbing). Combine two small bathrooms into one larger one. Install french drain under basement slab. Install floor over old stairwell hole in living room including integrate new oak flooring and refinishing first level hardwood floors. Reshingle roof and porch. Remove large asphalt circular driveway and pour replacement cement driveway. Relocate basement stairwell, including building new stairs. Remove and replace floor joists under addition and side of house due to termite damage. Remove addition carpet and tile floor below it, and install wood finish laminate flooring. Install new siding on entire house and addition. Re-grade 10,000 sq ft lot to improve water runoff. Replace windows throughout. Extensive interior remodel and drywall installation and finishing.

Radio Free Asia

Sr. Network & Software Engineer, Technical Operations

• Involvement in a wide range of Linux systems and IP network deployment and management efforts as well as software development of internal and external web applications to support RFA's mission of delivering uncensored news and information in nine native Asian languages: Chinese Mandarin and Cantonese, Korean, Tibetan, Uyhgur, Vietnamese, Lao, Khmer, and Burmese.
• Involvement in Internet circumvention efforts to allow target country entities such as dissedents and reporters, to gain access to inbound and outbound communications channels and resources such as the Internet and other telecommunications systems that are blocked or censored by foreign governments.
• Developed a method and software for hiding English messages in Cantonese vocabulary lists: CantoGraph
• Involvement in network, system, and information security efforts to protect RFA assets and information from attacks from entities hostile to RFA's mission.
• Significant involvement in wide ranging network and systems design, deployment and migration efforts with extensive automation (Python/Perl) of information gathering and reporting for planning phases as well as deployment and monitoring along with development of automated intrusion prevention mechanisms, and custom automated access controls.
• Converted RFA's proprietary NEC digital phone system to VoIP; consisting of 300 desktop phones in Washignton DC, and deployment and integration with remote PBXs in locations such as Bangkok, Phnom Phenh, Hong Kong and other locations, eliminating long-distance calling charges and providing local outdials including DISA (direct inward system access). Migration included integration of remaining T1/ISDN/POTS lines. Integration with Telos VX studio phone system (20+ studios), and various ISDN/POTS gateways (MediaTrix/Grandstream). Network and systems design and deployment with high availability/fail-over routing and PBX on CentOS/HP. Implemented with Asterisk opensource PBX. Developed Python/Perl tools for automated/bulk provisioning, configuration and QA of IP phones. Built custom interview recording solution along with web interface to recordings with authentication integrated with voicemail PIN. Full replacement of prior PBX features, including shared line appearance, voicemail, conferencing, FAX, special analog setups, hunt groups, paging, and phone driven automation of computer systems.
• Developed server imaging system for automated server install and configuration of the systems involved in the VoIP deployment. Any of these machines can be automatically rebuilt from scratch to their prior running configuration with no human intervention in about 15 minutes via a modified/simplified ISCONF infrastructure management suite. Essentially a combination of PXE/DHCP/DNS/make/rsync and CVS. Built associated PXE/TFTP/DHCP/DNS servers and incorporated centralized syslogging. Integrated with enterprise firewall (Stonegate) dhcp-relay/VLAN design.
• Developed web based, multi-language news gathering platform to aggregate feeds from news agencies: Xinhua News Agency, Reuters, AP, AFP and others to provide a central resource for RFA's editorial staff. Includes ingest of disparate feed sources via FTP, IMAP, RSS, NewsML, along with advanced multilanguage search and features such highlighting of search terms in paged results. Python language, with Sphinx documentation, and MySQL backend.
• Developed an integrated system for segmenting and transmitting RFA's live broadcast channels via HTTP Live Streaming (HLS) using opensource tools. The platform uses Axia IP Audio live broadcast sources on a Windows (Axia Linux driver no longer available) virtual machine running on a CentOS KVM host, which is then served with Icecast. A Linux Ubuntu guest VM running ffmpeg takes the Icecast source inputs and feeds them through Carson McDonald's opensource live_segmenter and tools to generate MPEG-2 Transport Stream segements and related extended M3U playlists.
• Developed a virtual-machine based, broadcast audio archiving system using above mentioned technologies, as the organization's Telos Audioactive hardware archivers are failing after years of use. Simultaneous recording at multiple sample/bit rate combinations allows one virtual machine to replace multiple hardware devices. Developed web interface to audio archives.

Verizon Business (MCI Communications)

Sr. Network & Software Engineer, Global Data Network Management

• Serve as a member of the IP Tier 3 Network Operations Center (NOC), responsible for global IP backbone operations for Verizon Business' (formerly MCI/UUNET), public IP backbone network; one of nine IP Tier 1 networks.
• Serve as the group's network software engineer, developing software for network management, monitoring, reporting, analysis and automation including:
  • Analysis and automated update of existing MPLS LSP bandwidth contracts (30,000 paths) for large networks (AS701 - North America and Canada, AS702 - Europe, AS703 - Asia Pacific) for transit traffic optimization.
  • Provide automation for all large scale, including global, network updates for: protocol changes (BGP, ISIS, MPLS), access list updates, authentication, routing policy, standards compliance monitoring, router OS upgrade deployment, including rollback capability.
  • Automate analysis of routing problems that are difficult to diagnose manually, such as those which require multiple simultaneous connections to routers with precise timing information correlated to events and debugging information, an example being 'receive side buffering' conditions on the Cisco platforms.
  • Developed software to quickly identify, and summarize or condense network blocks, prefix and access lists supporting rapid resolution to extreme traffic imbalances; similarly optimized prefix lists resolved issues with configurations previously too large to fit into router memory.
  • Developed software for automated pre-maintenance router traffic removal following complex Method of Proceedure rules.
  • Automate configuration and testing of high density aggregation hardware such as the Cisco Blizzard and Frostbite line-cards. Automated configuration of controllers, and BERT and loopback testing.
  • Provide software for global network monitoring and analysis, along with current and historical reporting (Web, MySQL, RRDTool) on network statistics including router and line-card CPU and memory utilization, network latency/SLAs, and automated detection of known anomalies to support quick response, or proactive intervention.
  • Develop tools for automated router-config generation and modification to support network upgrades, maintenance, and grooming, including large scale customer moves onto higher density hardware, full router chassis migrations (Juniper M160 to T640 moves), quality assurance of configurations, BGP and MPLS/LSP meshing/de-meshing of newly deployed and decommissioned hardware respectively, and related efforts.
  • Serve as systems administrator for the group's dedicated computing resources (Solaris).
  • Provide consultation to in-house IT groups on internal applications closely related to network operations, such as those systems for network configuration management.
• Perform various roles, on a rotational basis, with other members of the IP Tier3 organization, and engage in more dedicated roles for some longer term projects of the group, which has the ultimate responsibility for operations of the company's global IP network, and handing escalations that cannot be resolved by the Customer Support Technical Assistance Center (CTAC), and the Tier 2 NOC staff. Examples include:
  • Work hand-in-hand with Global Data Network Engineering for design, planning, analysis and field verification (FVO) of routing hardware, router OS software, and router configuration implementations as well as problem resolution of complex routing issues at the nexus of operations and engineering.
  • Perform all customer impacting (L1) maintenance on public IP network devices primarily consisting of Juniper and Cisco routers, FORE ATM switches, and Lucent Frame-Relay switches.
  • Define maintenance procedures and develop, document and update Maintenance Operating Procedure (MOPS) documentation for operating procedures that are utilized for all network maintenance performed by Tier2/3 NOC personnel.
  • Perform quality assurance assessment of hardware and software configuration of all new network devices prior to activation.
  • Review all network maintenance requests (approximately 70 daily), to ensure planned physical and logical configuration changes are sound, proper designation of network and/or customer impact and notifications are indicated, proper procedures are indicated, and that redundancy or other network elements will not be compromised or adversely effected when unnecessary.
  • Involvement in planning, and design related to major initiatives, as well as deployment and management of the design for projects such as the Voice over IP (VOIP) backbone (currently converting 1 million minutes per month from traditional phone infrastructure to IP), and Communications Assistance for Law Enforcement Act (CALEA) related network upgrades to support lawful intercept of network traffic by enforcement agencies.

Zope Corporation

Senior UNIX Administrator

• Deployment and management of a high-availability, high-volume (20-50+ million Web requests daily) managed hosting environment for customers including Viacom (cbsnewyork.com, cbs2chicago.com, many other media properties), Knight-Ridder, and the Chronicle of Higher Education. Design, deployment, and management of load-balancing (LVS), redundancy/failover, reverse proxy, caching, security, storage, and data backup/recovery components of this environment as well as related performance management and monitoring systems.
• Administration of several hundred enterprise and managed hosting systems; primarily Red Hat Enterprise Linux, with some Windows (2000/XP), FreeBSD UNIX, and Windows virtual machines on Linux (VMware).
• Design and deployment of systems supporting automated network operating systems installs (PXE boot/DHCP/systemimager), and rapid deployment and/or recreation of machines based on class, functional purpose, or specific purpose, along with class based (cache, app, storage server), or machine specific automated software installation and configuration including deterministic ordering of post OS imaging changes to a desired, or previously known-to-be-good state.
• Configuration and Management of systems and networks including mail (SMTP/POP/IMAP - Sendmail, Postfix, UCE controls(SPAM)), DHCP, DNS (Bind 9), Cisco 2950 LAN switches, Cisco 2611 series routers, Networking/NAT, Wireless Access, Secure Shell/public/private key access, VLAN, VPN, Firewall/netfilter/iptables, CVS version control, Web Servers (Zope/Apache), Nagios/LogWatch/syslog systems monitoring, database servers (ZODB, MySQL, PostGreSQL), LDAP (OpenLDAP), and Voice over IP/Asterisk software PBX.
• Conversion of the company's Alcatel digital phone system to the free, OpenSource Asterisk software based PBX, and VoIP phones, providing a dramatic reduction in capital equipment costs, long distance and conferencing charges, and elimination of a $5,000 annual service contract.
• Designed creative solutions for systems administration such as modifying the OpenSSH server to retrieve public keys via the HTTPS protocol allow for easy administration and control of access to network systems via the presence of public key files at a given URL (See: sshd PKI, as well as developing an IP billing platform based on the Linux Virtual Server (LVS)).

The Washington Post

Web Infrastructure and Deployment Engineer

• Responsibility for deployment and management of Web infrastructure components for all Web based Applications at the Washington Post newspaper (3000 users/systems). Management of Novell iChain reverse proxy, caching appliance including integration with the clustered Novell LDAP based eDirectory, and TCP traffic load balancing via Cisco Local Directors and later Cisco Content Switches, along with WebLogic application clustering integration via Apache/IIS I/ASAPI plug-ins primarily for J2EE based applications.
• Responsibility for AAA security implementation via the iChain/eDirectory components for access and performance management for all internal and external Web applications including the Washington Post Online Classifieds (50,000 users with 1,000 new users per week), Microsoft Active Directory/Windows integration, PeopleSoft HR and Corporate Meta-Directory integration (Affiliated companies i.e. Kaplan/Newsweek Interactive), CCI Web access (electronic publishing), Lotus Notes (e-mail/groupware), the Advertising Portal, eTearsheets, Kintana (Time tracking), and others.
• Extensive in-depth responsibility for deployment, configuration and management of Web infrastructure systems including hardware installation (Dell/Compaq), OS installation and configuration (Windows 2000 Advanced Server, Windows 2000 Workstation, Netware, Linux, AIX), iChain in a cluster configuration on Netware, eDirectory in a cluster configuration on Win2K, Apache 1.3/2.0 on UNIX and Windows, networking, routing and load balancing via iChain/Cisco configuration and integration, extensive LDAP directory deployment and management via ConsoleOne/iManager, OpenLDAP tools, and custom programming with Perl/Java/PHP, database deployment and integration (MySQL/Oracle), and extensive problem solving and troubleshooting of Web application technologies in this environment including TCP/IP and related protocols (SMTP/FTP/POP), HTTP 1.0/1.1 (Headers/Cookies/Sessions w/Failover/SSL/Auth), JavaScript, Java/WebLogic integration, LDAP, HTML, XML, CSS and related.

UUNET Technologies / WorldCom

Sr. Internet Systems Software Engineer

• Broad responsibility for wide ranging, multi-platform (UNIX/Windows) Web and enterprise applications and systems supporting operational deployment and management of a global Internet backbone.
• As a member of the Network Planning and Implementation department, and supporting the Capacity and Strategic Planning group, developed automated Weekly Executive Network Summary Report providing analysis and summary of UUNET's global IP network traffic (25 million records daily) for distribution to the President, heads of Engineering, Operations, Network Planning and Capacity and Strategic planning.
• Provided technical leadership and consulted with management and staff on current and emerging technologies and innovative and successful approaches for delivering large value through information technology.
• Lead, train, support and mentor a team of developers (4 - 12) and analysts in development, systems, analytic methods and effective business processes.
• Responsibility for design, development and implementation of numerous large scale, production Web applications, portals, and related systems, including extensive integration with large operational database systems and enterprise applications covering functional areas including customer portals, network monitoring, configuration, automation, performance, and management; customer billing, traffic engineering, capacity and strategic planning, SLAs, executive reporting on network capacity and utilization for all network traffic, circuit ordering and management, provisioning, and configuration, systems and asset management.
• Hands-on lead design, development, and implementation of many key applications, database systems, and high value automation
• Sophisticated Intranet Websites for Network Planning and Implementation, Capacity and Strategic Planning, Network Design, Network Provisioning, Network Coordination, Backbone Test and Acceptance, Quality Systems Management, Network CAD, Global Program Management. (Solaris, Windows, Apache/IIS, Perl, PHP, CGI, HTML/CSS, JavaScript, Java, Sybase ASE, Oracle, SQL, MS-Access)
• Marketing Web prototypes for new products and systems.
• Optical Circuit (SONET) test and acceptance tool suite for testing all OC/DS3 backbone circuits. FORE ATM switches.
• Backbone/Peering Circuit Management application
• Voice over IP (VoIP) Quality Engineering System for route suitability, SLA and SLA baseline, monitoring, troubleshooting of the company’s VoIP products. SIP, RTP, Cisco VoIP MIB, RTR responder.
• Automated DSL provisioning tool suite
• Automated Customer Network Design application
• Automated Customer Edge migration suite for migrating 22,000 T1 customers to a newer Cisco 12008/Lucent CBX500 platform

National Aeronautics and Space Administration (NASA)/ Boeing, Washington, DC.

Systems Management Analyst

• Lead systems management responsibility for a large heterogeneous enterprise network and related systems. 2,500 Windows, Macintosh (700), Novell, and UNIX systems.
• Advise management and staff on methods and systems for effective systems and configuration management.
• Implement and manage Microsoft Systems Management Server (SMS), Norton Administrator for Networks for hardware and software inventory, automated software distribution, and configuration management, analysis and reporting.
• Provide systems management supporting the migration of all 2,000+ desktop and server systems. WFW3.11 to Win98 and MacOS upgrade on the desktop, and Novell to WinNT server farm in the back office.
• Designed and developed NHQIS, the NASA headquarters multi-platform, unified systems management analysis and reporting system for integrating systems management information from disparate systems management applications. (Visual FoxPro for Windows/Mac)
• Designed and developed custom TCP/IP clients and servers for systems inventory and configuration management.
• Designed and developed dynamic Web and MS-Access front ends for analysis and reporting of systems management information from Microsoft SMS / MS-SQL Server

Residential / Commercial Construction and Development

• Residential remodeling and electric construction, as well as some commercial construction
• Computer systems management and support for several small firms
• Active in personal computing with Windows, Linux, and Internet including learning new languages such as Perl and HTML, as well as various Internet Protocols including NNTP, SMTP, HTTP.

Environmental Protection Agency (EPA), Martin Marietta/UNISYS, Washington, DC

Systems Analyst / Network Administrator

• Plan, deploy, manage and administer Local Area Networks (LANs) and related enterprise desktop systems as part of a unified computing environment consisting of 110 Novell networks, and mainframe and minicomputer resources.
• Implement and manage several Novell LANs (300-500 end users), along with support of related desktop systems and software as part of a large, multi-tiered support organization with a centralized help desk.
• Manage servers and database systems including the National Comprehensive Environmental Response, Compensation and Liability Act (CERCLA) database management system.
• Develop custom applications, and innovative software for systems inventory, mainframe connectivity, data transfer and printing, and cc:Mail post office synchronization before commercial solution was available.
• Conducted formal and informal training, including course preparation.

L.E. Peabody and Associates

Economic Analyst

• Develop computer models and systems for economic modeling of transportation systems supporting successful litigation ($100M+) before the Interstate Commerce Commission (ICC), now the Surface Transportation Board (STB).
• Develop financial (discounted cash flow), engineering, and operational models of transportation systems and integrate and optimize these models into unified economic modeling suite supporting rapid analysis of various economic approaches to litigation.
• Conduct studies that utilize various formulas employed by the STB in the development of costs for common carriers, including the Uniform Railroad Costing System (URCS) formula.
• Extensive analysis in the area of stand-alone costing (Staggers Rail Act), including route layout, design, and construction costs as well as detailed operating plans for various stand-alone railroads