Building Debian 13 anycast nodes from cloud images with cloud-init support: configuring multiple loopback addresses using netplan instead of /etc/network/interfaces, exabgp service configuration for anycast BGP route announcements, and deploying redundant anycasted DNS infrastructure.
Troubleshooting RouterOS User Manager RADIUS server in routed networks: discovering that RADIUS replies sourced from loopback interfaces are rejected by clients, requiring RADIUS server to use interface IPs or non-loopback addresses for proper operation.
Testing RouterOS 7 container functionality: building static Go binary, creating Docker image, exporting and importing container on RB5009UPr+S+, configuring veth interfaces and bridge networking, with sub-5ms latency for containerized web services.
MikroTik CRS3xx series switch evaluation: comparing CRS309-1G-8S+IN and CRS328-24P-4S+ models, tracking feature progression through RouterOS versions (spanning-tree, port isolation, DHCP snooping, MLAG support, VXLAN), and multi-vendor interoperability with HPE/Cisco switches.
Troubleshooting duplicate MAC addresses in redundant firewall HA deployments: discovering non-unique VRRP group IDs across sites caused site-to-site traffic loss, with MAC addresses appearing on unexpected switch ports during failover events.
Pull-based RouterOS address-list synchronization using dynamic webserver scripts: generating conditional add/update commands from database backend, fetching and executing scripts on multiple routers, tracking updates with timestamps, and avoiding API version compatibility issues.
Implementing exabgp 4.x built-in healthchecks for anycasted DNS: configuring health check intervals, rise/fall thresholds, withdraw-on-down behavior for automatic route removal on service failure, and multi-address-family BGP sessions for IPv4/IPv6 anycast announcements.
Implementing geographic fencing for nginx applications by ISP networks: using nginx geo module to whitelist IP subnets, aggregating prefixes by source AS from BGP looking glass, and generating dynamic ACL lists for regional access restrictions.
Building BGP looking glass on OpenBSD 6.6 using httpd and bgpd: configuring slowcgi for CGI support, setting up bgplg restricted socket, enabling ping/traceroute in chroot environment, and configuring dual IPv4/IPv6 BGP peering sessions for route lookups.
Understanding MPLS label swapping vs stacking: tracing label forwarding tables across router hops showing how labels are replaced at each hop using LDP distribution, with example traceroute output demonstrating label swapping through P/PE routers to destination.
Troubleshooting OSPF duplicate router IDs in VPLS networks: discovering 10-second route flapping caused by cloned PE router configuration, identifying router ID duplication, and understanding network-wide effects of duplicate router identifiers in MPLS service provider deployments.
Deploying enterprise WiFi with MikroTik CAPsMAN: configuring centralized management for multiple access points, setting up channels/datapaths/security configs, choosing between local and centralized forwarding, and enabling adaptive noise immunity on Atheros chipsets.
Analyzing BGP route leak causing AS7122 outage: AS53443 leaked 535,292 DFZ routes from AS6327 during second provider turnup due to missing outbound prefix filters, combined with AS7122 accepting routes without customer filters, congesting 70% of internet-bound traffic for 13 minutes.
Installing FastNetMon community edition from Ubuntu 18.04 universe repository: configuring exabgp 4.0.2 with new config format using socat and pipe communication, testing BGP BLACKHOLE route injection via /var/run/exabgp.cmd for automated DDoS mitigation.
Logging BGP routes with GoBGP MRT dumps: configuring TOML-based RIB dumps every 8 hours and BGP update files every 5 minutes, processing MRT files with BGP Scanner and mrt2mysql.py for Canadian AS-PATH tracking and routing analysis.
Building MPLS Service Provider networks from scratch: implementing VPLS on MikroTik and full L3VPN on HPE Comware, configuring LDP label distribution, understanding MPLS benefits for latency reduction and QoS, and managing customer VRFs with vpnv4 BGP peering.
Implementing AS112 reverse DNS sinkhole using BIRD BGP daemon: installing kernel routes from BGP, configuring prefix filters for RFC1918/RFC6890 anycast addresses, and using arouteserver to automate IXP AS112 deployments with BIRD.
Deploying RFC1918/RFC6890 reverse DNS sinkhole with AS112 using exabgp: configuring anycasted 192.175.48.0/24 and 2620:4f:8000::/48 prefixes, implementing exabgp-healthcheck for DNS service monitoring, and advertising BLACKHOLE routes only when service is operational.
Researching Canadian BGP routing over four years: evolving from exabgp JSON piped to CouchDB, to MySQL database ingestion, to automated province-by-province ASN status page generation with git-tracked history and BGP growth analysis.
Building anycasted highly-available recursive DNS with dnsdist load balancers and exabgp: using BGP MED-based failover across 4 dnsdist nodes with PowerDNS recursor backends, custom orderedwrandom load balancing policy, and automated healthchecks for route withdrawal.
Logging continuous ping latency and packet loss for troubleshooting using fping -D -l flags: capturing unix timestamps with latency measurements, converting timestamps with Perl, and logging output to file for later analysis.
Configuring IPv6 first-hop redundancy with MikroTik VRRPv3: troubleshooting Router Advertisement issues where IPv6 sessions died during failover, solving by disabling RA on VLAN interfaces and configuring IPv6 ND settings for proper virtual gateway operation.
Implementing RFC 7999 BGP BLACKHOLE community (65535:666) on MikroTik RouterOS: configuring routing filters to set blackhole route type, integrating with FastNetMon for automated DDoS mitigation, and handling provider-specific communities for upstream blackholing.
Large BGP Community draft RFC addressing 32-bit ASN limitations in standard BGP communities: tracking implementation status across ExaBGP, Cisco IOS XR, Nokia SR-OS, and Bird, with development on Github.
I ran into an obscure limitation in PowerDNS 4.0 authoritative server recently. I have one nameserver which also acts as a slave to two other sets of zones with are transfered using AXFR. Some of those zones are DNSSEC enabled, and PowerDNS is only able to handle DNSSEC on the first backend loaded. This was causing several forward and reverse zones to fail to serve the DNSSEC records along with the queried records, and DNSSEC validation to partially fail.
PowerDNS makes a mighty fine authoritative, and also recursive DNS server. They also recently added a DNS-aware DNS load balancer. This article deals with load balancing multiple backend caches to keep all of them hot and working the most efficiently.
Building a Slack /bgp slash command using custom API at api.hextet.net: integrating BGP lookups, whois, and looking glass functionality directly into Slack channels via web API POST integration.
I first came across dnsdist in a NANOG post in the discussion of exploitation of a BIND DOS bug last summer. Jared Mauch had recommended dnsdist to easily implement DNS backend diversity.
Launching AS395089 for Hextet Systems from ARIN assignment to full BGP operations in one day: configuring IPv6 /44 and IPv4 /24 announcements, setting up PeeringDB/BGPmon accounts, creating IRR objects (aut-num, route/route6, AS-HEXTET as-set).
MBIX Peering LAN IP space hijacked via BGP and used for spam: investigating unauthorized BGP announcements of 206.72.208.0/24 through AS-Path 9002 44050 131788, demonstrating why IX peering LAN space should not be globally routable.
Riot Games engineering blog on optimizing BGP routing for League of Legends: why buying transit from Tier 1 providers doesn't solve all latency problems and how peering at Internet Exchanges is critical for game performance.
Implementing BGP community-based filtering to control route advertisements and prevent route leaks: designing community number ranges for route sources, localpref modifiers, prepending, and selective upstream/peer/customer announcements with example Cisco route-maps.
Building Nagios 4 from source on Debian to monitor 75 hosts and 130 services: implementing custom checks for TCP services, DNS zones, XMPP, Tor, HTTP/HTTPS with cert expiry, plus SMS alerts via nagios-twilio and enhanced email with mtr output.
Configuring MikroTik hairpin NAT to access external IPs from internal network: implementing dst-nat and masquerade rules to route traffic destined for public IP back to internal reverse proxy when DNS returns external address.
Rewriting ipquail.com in Go using pilu/traffic router: replacing SSI/Apache hacks with proper API endpoints for IP/PTR lookups, nginx reverse proxy handling IPv4/IPv6 termination, and JSON response formatting.
Monitoring Tor hidden services with Nagios using torsocks: creating check_commands for HTTP/SSH/TCP over Tor, tracking onion availability/performance, and publishing monion repository with sample configs for .onion monitoring.
Implementing IPv6 monitoring in Nagios: creating dual-stack service checks with -4/-6 flags, defining custom _ADDRESS4/_ADDRESS6 variables for DNS independence, and navigating IPv4-only plugin limitations like check_icmp.
AS7122 enables atomic aggregate on /16 announcements, accidentally suppressing four Manitoba customer ASNs (21876, 23001, 32433, 54937) from global routing table—analyzing route withdrawals via RIPE collectors and multihoming implications.
Fixing Smokeping after Debian Wheezy to Jessie upgrade: Apache 2.2 to 2.4 migration breaks conf.d structure, requiring manual symlinks in conf-enabled directory for smokeping and OTRS configurations.
Creating RPSL objects in ARIN's Internet Routing Registry: setting up MNTNER with MD5 authentication, generating route/inetnum objects via email templates, and validating IRR entries for BGP prefix filtering automation.
Configuring RSA public key authentication on Cisco IOS 15 switches/routers: using ip ssh pubkey-chain for passwordless SSH, handling 254-character line length limits on 2960/3560 platforms, and enforcing SSHv2.
Understanding Bandwidth Delay Product and Long Fat Networks: how RTT and TCP receive window size limit per-thread throughput, why download managers use multiple connections, and how TCP window auto-scaling optimizes performance.
Implementing IPv6 Unique Local Addresses (RFC4193) with OpenBSD NAT66: generating fd00::/8 ULA prefixes, translating to public IPs, and workarounds for single /64 provider limitations or private VPS networks.
Tracking dramatic Manitoba ASN growth in 2013-2014: IXP launches (MBIX/WpgIX) drive half of 60+ ASNs to peer locally, exchanging ~1 Gbps traffic and improving rural ISP access to universities over major providers.
Troubleshooting MikroTik CCR1036 SNMP failure on loopback interface: packet captures reveal implementation quirk refusing to return traffic via different interface than request arrived on, requiring monitoring via closest interface IP.
Building distributed monitoring with SaltStack returners to CouchDB: storing nagios check results globally from multiple VPS locations, visualizing traceroutes/latency, and designing contextual alerts beyond simple up/down states.
Multi-phase migration to BGP-announced IP space: AS62758 turnup, moving LMS and external services from provider IPs, pre-change testing with reduced DNS TTLs, 1:1 NAT configs, and phased rollout strategy.
Observium 0.14.11 released on schedule: new alerting system, Cisco ASA IPv4 session graphing from FIREWALL-MIB, Cambium Canopy support, and enabling built-in VMware ESXi SNMP monitoring.
Automating Canadian ASN discovery using Blockfinder: daily diff emails tracking ARIN assignments, patching TTY progress bar issues for cron jobs, and proposing ARIN's arin-issued list for AS numbers (later implemented).
Responding to POODLE vulnerability by disabling SSLv3 across all services: enforcing TLSv1.0+, updating cipher suites from Mozilla wiki, logging protocol/cipher combinations to verify modern client support.
Tracking down rogue DHCP server on corporate network: using arpwatch to identify Mac sharing causing 192.168.137.x leases, correlating MAC to CDP/switch port, and why DHCP snooping prevents these issues.
Analyzing OpenBSD PF firewall logs with enhanced Pantz PFlog Stats: added GeoIP support using Maxmind DB, updated whois links for global regions, and automated blocked packet analysis with Perl scripting.
Building a JSON API for ipquail.com using Python/Flask and uWSGI/nginx: RESTful endpoints returning IPv4/IPv6 addresses, inspired by ARIN's Whois-RWS, with CORS support and production SSI implementation.
Attending ARIN on the Road in Winnipeg: IPv6-focused presentations, hands-on with RESTful web services API, discussions on DNSSEC/RPKI implementation, and networking with ARIN leadership.
Personal milestone: all public-facing services now dual-stack IPv6—anycast web, mail (SMTP/IMAP/POP3), DNS (authoritative/recursive), SSH/FTP, SSL VPN, SaltStack, Nagios, plus native home connectivity via HE tunnel.
Using SaltStack jinja templating to deploy custom fail2ban jail.local configs: mail server-specific protection for dovecot/postfix auth failures with iRedMail hardening policies.
Canadian ISP hijacks 51 prefixes to redirect cryptocurrency miners via unfiltered AS6939 peer: bogon routes, more-specifics targeting Digital Ocean/AWS, and lessons in prefix filtering failures.
Automating fail2ban deployment with SaltStack: using state files and custom jail.local configs to protect SSH on public-facing Debian systems, with centralized ban time management across DigitalOcean droplets.
Migrating from Cisco 3750G to Brocade ICX6610 core: six months of planning, lab testing spanning-tree interop, port-for-port migration strategy, VRRP implementation, and minimal post-change issues.
Rapid deployment of slave nameservers on DigitalOcean VMs using SaltStack automation: scripted setup from initial VM creation through salt-minion configuration and state.highstate execution across multiple regions.
Brocade ICX6430/6610 fundamentals: VLAN-centric port configuration (versus Cisco's port-centric approach), default-vlan-id behavior, dual-mode for voice/data, and key differences between access/trunk/dual-mode port types.
Setting up apt-cacher-ng for efficient Debian package caching: dedicated VM serving 98% cache hits, integrated with SaltStack automation for pre-testing updates and distributing proxy configs.
Adding CLI-friendly IP detection to ipquail.com: returns plain IPv4/IPv6 addresses when accessed via curl, with separate subdomains for protocol-specific queries and script integration.
Automating git pulls across multiple anycast nodes using SaltStack: replacing manual SSH logins with a single salt master command that updates web directories simultaneously on all anycast instances.
Discovering ASCIIFlow: browser-based tool for drawing RFC-style ASCII network diagrams with boxes, lines, and arrows, plus Google Drive integration for saving text-based topology charts.
Getting started with SaltStack master/minion setup: key management, running commands across Linux/FreeBSD hosts, apt integration for package updates, and Python API for automation scripting.
Shell scripts for ping troubleshooting: one detects IPv4/IPv6 connectivity failures with timeout alerts, another timestamps all ping output for detailed logging and analysis.
Moving services out of the basement into a half-cabinet: OpenBSD edge router advertising IPv4/IPv6 space, ESXi host for VMs, Cisco switching, APC UPS, and initial site/monitoring DNS migrations.
Comparing GeoDNS approaches: BIND with GeoIP patches versus the Go-based abh/geodns, tradeoffs between package maintenance and flexibility, and plans to pair GeoDNS with traditional authoritative DNS.
Building a home lab DNS/DHCP appliance on Raspberry Pi: installing ISC DHCP and BIND, wiring dynamic DNS updates, and serving multiple VLANs with relay helpers.
Updated guide to run a BGP looking glass on OpenBSD 5.5+ after Apache removal: nginx/slowcgi setup, permissions for bgplg tools, and sample bgpd peers for v4/v6 visibility.
CLI subnet calculator (subcalc) with ifconfig-style syntax. Supports IPv4/IPv6, reverse DNS generation, and 6to4 conversions; maintained cross-platform on GitHub.
Investigating a suspected BGP route leak involving TeraGo impacting AS paths between my networks. Traceroute analysis, AS-PATH inspection, and comparison of expected vs. leaked routes.
Building an exabgp lab to explore route injection, anycast/HA services, route servers, and DDoS mitigation scenarios using Python-based BGP automation.
Business case for IPv6 adoption: criticality of Internet services, IPv4 exhaustion risks, deployment timelines, and presenting benefits to management with real-world considerations.
Local IPv6 advocacy group at SkullSpace hackerspace working to raise awareness through IPv6-only demos, whitepapers, and real-world infrastructure setup between multiple locations.
Overview of native IPv6-enabled services including web (nginx/Apache), email (Postfix/Dovecot), XMPP (Prosody), and DNS. Covers configuration and operational challenges.
BGP prefix hijacking incident involving Indonesian ISP Indosat (AS4761) advertising multiple ISP prefixes. Analysis of Thai BGP upstream filtering failure and impact on global routing.
Switching from subnetsmngr to NIPAP for improved IP address management. NIPAP offers flexible subnet allocation, IPv4/IPv6 parity, VRF support, and both CLI and web interfaces.
Using iperf for UDP performance testing to isolate VoIP-related network issues. Demonstrates measuring jitter, packet loss, and bandwidth with IPv6 over long distances.
Building IP Quail, an IPv4/IPv6 service to display client IP addresses. Deployed on HenchCDN with anycast distribution using SSI and Bootstrap for high performance and availability.
SSL/TLS cipher and protocol configuration for nginx, dovecot, and Apache. Securing connections by selecting strong ciphers and disabling weak algorithms. Note: See Mozilla SSL Configuration Generator for current best practices.
First impressions with Brocade ICX 6430-24P switch. CLI syntax comparison between Brocade and Cisco for VLAN/access layer configuration, POE support, and Observium integration.
Using RFC3021 /31 netmasks on point-to-point links to conserve IPv4 addresses. Comparison with traditional /30 usage and device compatibility notes for Cisco and MikroTik equipment.
Setting up anycast services using Bird OSPF to inject /32 and /128 routes across multiple locations. Using FreeBSD VMs for redundant IPv4/IPv6 anycast DNS and web services.
Comparing methods for redundant internet failover: manual, dual-wan firewalls, and BGP routing. BGP provides automatic failover, simplified troubleshooting, and better control over multi-provider connectivity.
BGP redundancy techniques: prepending, localpref coordination, and address splitting. Comparing methods for inbound failover with pros and cons of each approach.
Troubleshooting UDP stream connectivity issues using packet captures and Wireshark. Identified TTL expiration as root cause of packets not reaching destination across multiple hops.
Experiences launching Winnipeg Internet Exchange (WpgIX). Discusses benefits of local peering, network configuration challenges, and impact of content delivery infrastructure.
Personal experiences with IPv6 adoption from 2001 tunnel access through ISP-level deployment. Covers client behavior (Happy Eyes), NAT implications, security considerations, and network implementation perspectives.
Implementing BCP38 ingress filtering on Cisco routers to prevent spoofed traffic from leaving your network. Includes ACL examples to filter bogus source addresses and private ranges.
Setting up bgplg, OpenBSD's built-in shell and CGI BGP looking glass. Includes handling ping/traceroute in chrooted environments and considerations for nginx migration.
First impressions of Observium network monitoring platform after 10 years with Cacti. Auto-discovery, 64-bit counters, total device traffic graphs, and MAC/ARP tables make it a compelling alternative.
Using openssl s_client to test SSL/TLS connections and verify certificate chains for encrypted email services like POP3, IMAP, and SMTP from the command line.
Overview of major websites with IPv6 support including Google, Facebook, Yahoo, and Cloudflare. Browser extensions like IPvFox and ipvfoo help identify IPv6-enabled sites.
Step-by-step guide to enabling full IPv6 support on Cisco Catalyst 3560/3750 switches, including SDM mode configuration, unicast routing, and interface addressing setup.