Systems
RouterOS DHCP Scripts and RADIUS Start/Stop Surprises
RouterOS DHCP server scripting with RADIUS accounting only firing on start/stop, not renewals.RouterOS User Manager in Routed Networks
Troubleshooting RouterOS User Manager RADIUS server in routed networks: discovering that RADIUS replies sourced from loopback interfaces are rejected by clients, requiring RADIUS server to use interface IPs or non-loopback addresses for proper operation.RouterOS 7 Container Test
Testing RouterOS 7 container functionality: building static Go binary, creating Docker image, exporting and importing container on RB5009UPr+S+, configuring veth interfaces and bridge networking, with sub-5ms latency for containerized web services.Mikrotik CRS3xx Switching
MikroTik CRS3xx series switch evaluation: comparing CRS309-1G-8S+IN and CRS328-24P-4S+ models, tracking feature progression through RouterOS versions (spanning-tree, port isolation, DHCP snooping, MLAG support, VXLAN), and multi-vendor interoperability with HPE/Cisco switches.Duplicate MAC Address Madness
Troubleshooting duplicate MAC addresses in redundant firewall HA deployments: discovering non-unique VRRP group IDs across sites caused site-to-site traffic loss, with MAC addresses appearing on unexpected switch ports during failover events.RouterOS 7.x Filtering
Implementing comprehensive BGP filtering policies on RouterOS 7.x: defining community lists for roles (customer/transit/peering), configuring route protection and RPKI validation, applying per-link inbound/outbound filters with prefix size validation and community-based localpref manipulation.Syncing RouterOS address-lists
Pull-based RouterOS address-list synchronization using dynamic webserver scripts: generating conditional add/update commands from database backend, fetching and executing scripts on multiple routers, tracking updates with timestamps, and avoiding API version compatibility issues.RPKI w/ RouterOS v7
Implementing RPKI route validation on RouterOS v7 with routinator RTR server: configuring RTR connections with preference/failover, validating routes with routing filter rules, rejecting invalid routes while preferencing valid routes, and managing ROA growth tracking.exabgp 4.x, anycast, and healthchecks
Implementing exabgp 4.x built-in healthchecks for anycasted DNS: configuring health check intervals, rise/fall thresholds, withdraw-on-down behavior for automatic route removal on service failure, and multi-address-family BGP sessions for IPv4/IPv6 anycast announcements.nginx ISP fence
Implementing geographic fencing for nginx applications by ISP networks: using nginx geo module to whitelist IP subnets, aggregating prefixes by source AS from BGP looking glass, and generating dynamic ACL lists for regional access restrictions.OpenBSD 6.6 BGP Looking Glass
Building BGP looking glass on OpenBSD 6.6 using httpd and bgpd: configuring slowcgi for CGI support, setting up bgplg restricted socket, enabling ping/traceroute in chroot environment, and configuring dual IPv4/IPv6 BGP peering sessions for route lookups.MPLS Labels Explained
Understanding MPLS label swapping vs stacking: tracing label forwarding tables across router hops showing how labels are replaced at each hop using LDP distribution, with example traceroute output demonstrating label swapping through P/PE routers to destination.Troubleshooting a Duplicate OSPF Router ID
Troubleshooting OSPF duplicate router IDs in VPLS networks: discovering 10-second route flapping caused by cloned PE router configuration, identifying router ID duplication, and understanding network-wide effects of duplicate router identifiers in MPLS service provider deployments.Centrally Managed WiFi with MikroTik CAPsMAN
Deploying enterprise WiFi with MikroTik CAPsMAN: configuring centralized management for multiple access points, setting up channels/datapaths/security configs, choosing between local and centralized forwarding, and enabling adaptive noise immunity on Atheros chipsets.Fastnetmon on Ubuntu 18.04
Installing FastNetMon community edition from Ubuntu 18.04 universe repository: configuring exabgp 4.0.2 with new config format using socat and pipe communication, testing BGP BLACKHOLE route injection via /var/run/exabgp.cmd for automated DDoS mitigation.The Road to MPLS
Building MPLS Service Provider networks from scratch: implementing VPLS on MikroTik and full L3VPN on HPE Comware, configuring LDP label distribution, understanding MPLS benefits for latency reduction and QoS, and managing customer VRFs with vpnv4 BGP peering.AS112 and BIRD
Implementing AS112 reverse DNS sinkhole using BIRD BGP daemon: installing kernel routes from BGP, configuring prefix filters for RFC1918/RFC6890 anycast addresses, and using arouteserver to automate IXP AS112 deployments with BIRD.AS112 and Exabgp
Deploying RFC1918/RFC6890 reverse DNS sinkhole with AS112 using exabgp: configuring anycasted 192.175.48.0/24 and 2620:4f:8000::/48 prefixes, implementing exabgp-healthcheck for DNS service monitoring, and advertising BLACKHOLE routes only when service is operational.OSX Built-in TFTP Server
Starting macOS built-in TFTP server using launchctl: loading tftp.plist from System/Library/LaunchDaemons, starting com.apple.tftpd service, and using default /private/tftpboot directory for file transfers.HA Recursive DNS with dnsdist and exabgp
Building anycasted highly-available recursive DNS with dnsdist load balancers and exabgp: using BGP MED-based failover across 4 dnsdist nodes with PowerDNS recursor backends, custom orderedwrandom load balancing policy, and automated healthchecks for route withdrawal.MikroTik, VRRP, and IPv6
Configuring IPv6 first-hop redundancy with MikroTik VRRPv3: troubleshooting Router Advertisement issues where IPv6 sessions died during failover, solving by disabling RA on VLAN interfaces and configuring IPv6 ND settings for proper virtual gateway operation.BGP BLACKHOLE Community
Implementing RFC 7999 BGP BLACKHOLE community (65535:666) on MikroTik RouterOS: configuring routing filters to set blackhole route type, integrating with FastNetMon for automated DDoS mitigation, and handling provider-specific communities for upstream blackholing.HA Authoritative DNS w/ dnsdist
I ran into an obscure limitation in PowerDNS 4.0 authoritative server recently. I have one nameserver which also acts as a slave to two other sets of zones with are transfered using AXFR. Some of those zones are DNSSEC enabled, and PowerDNS is only able to handle DNSSEC on the first backend loaded. This was causing several forward and reverse zones to fail to serve the DNSSEC records along with the queried records, and DNSSEC validation to partially fail.dnsdist with pdns recursors
PowerDNS makes a mighty fine authoritative, and also recursive DNS server. They also recently added a DNS-aware DNS load balancer. This article deals with load balancing multiple backend caches to keep all of them hot and working the most efficiently.ESXi 6up2
VMware ESXi 6.0 Update 2 improvements: using HTML5 VMware Host Client for full VM provisioning without Windows vSphere client, functional browser-based console, fixed macOS Remote Console launching, and no Flash required for administration.Load Balancing DNS with dnsdist
I first came across dnsdist in a NANOG post in the discussion of exploitation of a BIND DOS bug last summer. Jared Mauch had recommended dnsdist to easily implement DNS backend diversity.HTML Email with Nagios
Implementing HTML-formatted Nagios alerts using nagios-html-email: installing via npm, configuring custom notification commands, and replacing mailx with sendmail to properly render HTML email notifications.How I Use Nagios
Building Nagios 4 from source on Debian to monitor 75 hosts and 130 services: implementing custom checks for TCP services, DNS zones, XMPP, Tor, HTTP/HTTPS with cert expiry, plus SMS alerts via nagios-twilio and enhanced email with mtr output.MikroTik and Hairpin NAT
Configuring MikroTik hairpin NAT to access external IPs from internal network: implementing dst-nat and masquerade rules to route traffic destined for public IP back to internal reverse proxy when DNS returns external address.ipquail-go
Rewriting ipquail.com in Go using pilu/traffic router: replacing SSI/Apache hacks with proper API endpoints for IP/PTR lookups, nginx reverse proxy handling IPv4/IPv6 termination, and JSON response formatting.Monitoring Onions
Monitoring Tor hidden services with Nagios using torsocks: creating check_commands for HTTP/SSH/TCP over Tor, tracking onion availability/performance, and publishing monion repository with sample configs for .onion monitoring.Nagios and IPv6
Implementing IPv6 monitoring in Nagios: creating dual-stack service checks with -4/-6 flags, defining custom _ADDRESS4/_ADDRESS6 variables for DNS independence, and navigating IPv4-only plugin limitations like check_icmp.Running Smokeping on Jessie
Fixing Smokeping after Debian Wheezy to Jessie upgrade: Apache 2.2 to 2.4 migration breaks conf.d structure, requiring manual symlinks in conf-enabled directory for smokeping and OTRS configurations.Nagios Logs, Handy One-Liner
Converting Nagios log timestamps to human-readable format using Perl one-liner: piping epoch timestamps through localtime() substitution for on-the-fly date conversion in monitoring logs and BGP data.Working with ARINs Route Registry
Creating RPSL objects in ARIN's Internet Routing Registry: setting up MNTNER with MD5 authentication, generating route/inetnum objects via email templates, and validating IRR entries for BGP prefix filtering automation.Removing Stubborn Trash Items
Fixing macOS Yosemite trash items that won't delete: using CLI to remove immutable flags with chflags -R nouchg and force deletion when GUI methods fail.SSH Pubkeys with Cisco IOS 15
Configuring RSA public key authentication on Cisco IOS 15 switches/routers: using ip ssh pubkey-chain for passwordless SSH, handling 254-character line length limits on 2960/3560 platforms, and enforcing SSHv2.Latency and Effects on TCP Performance
Understanding Bandwidth Delay Product and Long Fat Networks: how RTT and TCP receive window size limit per-thread throughput, why download managers use multiple connections, and how TCP window auto-scaling optimizes performance.virtio NIC on OpenBSD under KVM
Fixing intermittent OpenBSD vio(4) network dropouts under KVM: disabling RingEventIdx via config -ef /bsd kernel flag adjustment resolves interface connectivity issues requiring periodic ifconfig cycling.Using ULA and NAT66
Implementing IPv6 Unique Local Addresses (RFC4193) with OpenBSD NAT66: generating fd00::/8 ULA prefixes, translating to public IPs, and workarounds for single /64 provider limitations or private VPS networks.ASN Growth in Manitoba
Tracking dramatic Manitoba ASN growth in 2013-2014: IXP launches (MBIX/WpgIX) drive half of 60+ ASNs to peer locally, exchanging ~1 Gbps traffic and improving rural ISP access to universities over major providers.Sleep Talking
Troubleshooting Intel I217-LM NIC bug causing 3300 pps IPv6 multicast storms during S1 sleep: using tcpdump/Wireshark to identify MLDv2 listener reports flooding network, resolved with driver v19.5 update.Does Everything Still Work?
Building test-driven infrastructure with Bash Automated Testing System (BATS): creating 27 pre-change validation tests for websites, redirects, SSL endpoints, and dynamic content to answer "does everything still work?"MikroTik SNMP Oddity
Troubleshooting MikroTik CCR1036 SNMP failure on loopback interface: packet captures reveal implementation quirk refusing to return traffic via different interface than request arrived on, requiring monitoring via closest interface IP.SaltStack, CouchDB, and Nagios
Building distributed monitoring with SaltStack returners to CouchDB: storing nagios check results globally from multiple VPS locations, visualizing traceroutes/latency, and designing contextual alerts beyond simple up/down states.Phased Service Moves
Multi-phase migration to BGP-announced IP space: AS62758 turnup, moving LMS and external services from provider IPs, pre-change testing with reduced DNS TTLs, 1:1 NAT configs, and phased rollout strategy.Observium November Release
Observium 0.14.11 released on schedule: new alerting system, Cisco ASA IPv4 session graphing from FIREWALL-MIB, Cambium Canopy support, and enabling built-in VMware ESXi SNMP monitoring.Service Status via SaltStack 2014.7 with Nagios
Exploring SaltStack 2014.7's nagios module: remote execution of nagios-plugins on minions, pillar-defined check lists with Jinja templating, distributed monitoring ideas, and contributing documentation fixes to SaltStack.New and Removed ASNs
Automating Canadian ASN discovery using Blockfinder: daily diff emails tracking ARIN assignments, patching TTY progress bar issues for cron jobs, and proposing ARIN's arin-issued list for AS numbers (later implemented).Troubleshooting ICMPv6 with tcpdump
Fixing OpenBSD PF firewall ICMPv6 rules by using tcpdump to identify required neighbor discovery types (133-137): isolating NDP traffic with packet filters and properly allowing router/neighbor solicitation/advertisement messages.SSLv3 Disabled
Responding to POODLE vulnerability by disabling SSLv3 across all services: enforcing TLSv1.0+, updating cipher suites from Mozilla wiki, logging protocol/cipher combinations to verify modern client support.Internet Connection Sharing
Tracking down rogue DHCP server on corporate network: using arpwatch to identify Mac sharing causing 192.168.137.x leases, correlating MAC to CDP/switch port, and why DHCP snooping prevents these issues.Firewall Log Stats
Analyzing OpenBSD PF firewall logs with enhanced Pantz PFlog Stats: added GeoIP support using Maxmind DB, updated whois links for global regions, and automated blocked packet analysis with Perl scripting.CVE-2014-6271 - ShellShock
ShellShock bash vulnerability (CVE-2014-6271) allowing remote code execution via crafted environment variables: patching Debian systems with SaltStack, tracking 6 related CVEs, and emergency response coordination.Strange SNMP SetRequest Traffic
Investigating spoofed SNMP SetRequest attacks using Google DNS IPs (8.8.8.8): 45-minute packet capture reveals sequential scanning patterns targeting ISP /16 blocks, confirming Internet Storm Center findings.IP Quail API
Building a JSON API for ipquail.com using Python/Flask and uWSGI/nginx: RESTful endpoints returning IPv4/IPv6 addresses, inspired by ARIN's Whois-RWS, with CORS support and production SSI implementation.ARIN on the Road
Attending ARIN on the Road in Winnipeg: IPv6-focused presentations, hands-on with RESTful web services API, discussions on DNSSEC/RPKI implementation, and networking with ARIN leadership.Fully IPv6 Enabled
Personal milestone: all public-facing services now dual-stack IPv6—anycast web, mail (SMTP/IMAP/POP3), DNS (authoritative/recursive), SSH/FTP, SSL VPN, SaltStack, Nagios, plus native home connectivity via HE tunnel.Deploying a Host-Specific Fail2Ban Config with SaltStack
Using SaltStack jinja templating to deploy custom fail2ban jail.local configs: mail server-specific protection for dovecot/postfix auth failures with iRedMail hardening policies.Unshortening with cURL
Using curl to reveal the final destination of shortened URLs before clicking: one-liner command with redirect following and a simple shell script wrapper for safe link verification.Deploying Fail2Ban using SaltStack
Automating fail2ban deployment with SaltStack: using state files and custom jail.local configs to protect SSH on public-facing Debian systems, with centralized ban time management across DigitalOcean droplets.Well Executed Network Upgrades
Migrating from Cisco 3750G to Brocade ICX6610 core: six months of planning, lab testing spanning-tree interop, port-for-port migration strategy, VRRP implementation, and minimal post-change issues.Deploying a Nameserver at DigitalOcean in 2 minutes
Rapid deployment of slave nameservers on DigitalOcean VMs using SaltStack automation: scripted setup from initial VM creation through salt-minion configuration and state.highstate execution across multiple regions.Brocade ICX Basics
Brocade ICX6430/6610 fundamentals: VLAN-centric port configuration (versus Cisco's port-centric approach), default-vlan-id behavior, dual-mode for voice/data, and key differences between access/trunk/dual-mode port types.Debian Package Caching
Setting up apt-cacher-ng for efficient Debian package caching: dedicated VM serving 98% cache hits, integrated with SaltStack automation for pre-testing updates and distributing proxy configs.ipquail now curl friendly
Adding CLI-friendly IP detection to ipquail.com: returns plain IPv4/IPv6 addresses when accessed via curl, with separate subdomains for protocol-specific queries and script integration.Anycast Administration with SaltStack
Automating git pulls across multiple anycast nodes using SaltStack: replacing manual SSH logins with a single salt master command that updates web directories simultaneously on all anycast instances.ASCII Network Diagrams
Discovering ASCIIFlow: browser-based tool for drawing RFC-style ASCII network diagrams with boxes, lines, and arrows, plus Google Drive integration for saving text-based topology charts.SaltStack Automation
Getting started with SaltStack master/minion setup: key management, running commands across Linux/FreeBSD hosts, apt integration for package updates, and Python API for automation scripting.Manually Patching ESXi (5.5)
Simplified ESXi 5.5 patching workflow using online depot: enable httpClient firewall rule, run esxcli profile update from VMware's depot URL, reboot—no manual bundle transfers needed.pinglog
Shell scripts for ping troubleshooting: one detects IPv4/IPv6 connectivity failures with timeout alerts, another timestamps all ping output for detailed logging and analysis.check-soa
check-soa is a CLI tool written in go which lets a DNS name server administrator easily verify that all servers listed as authoritative are in sync and serving the same zone by verifying the SOA record. I use this tool nearly every day at work to verify that changes have propagated to all slave nameservers.VM Moves
Moving services out of the basement into a half-cabinet: OpenBSD edge router advertising IPv4/IPv6 space, ESXi host for VMs, Cisco switching, APC UPS, and initial site/monitoring DNS migrations.GeoDNS
Comparing GeoDNS approaches: BIND with GeoIP patches versus the Go-based abh/geodns, tradeoffs between package maintenance and flexibility, and plans to pair GeoDNS with traditional authoritative DNS.Raspberry Pi DNS/DHCP Server
Building a home lab DNS/DHCP appliance on Raspberry Pi: installing ISC DHCP and BIND, wiring dynamic DNS updates, and serving multiple VLANs with relay helpers.OpenBSD 5.5 BGP Looking Glass
Updated guide to run a BGP looking glass on OpenBSD 5.5+ after Apache removal: nginx/slowcgi setup, permissions for bgplg tools, and sample bgpd peers for v4/v6 visibility.subcalc
CLI subnet calculator (subcalc) with ifconfig-style syntax. Supports IPv4/IPv6, reverse DNS generation, and 6to4 conversions; maintained cross-platform on GitHub.Leaking Routes
Investigating a suspected BGP route leak involving TeraGo impacting AS paths between my networks. Traceroute analysis, AS-PATH inspection, and comparison of expected vs. leaked routes.exabgp: first lab
Building an exabgp lab to explore route injection, anycast/HA services, route servers, and DDoS mitigation scenarios using Python-based BGP automation.Selling IPv6 to Management
Business case for IPv6 adoption: criticality of Internet services, IPv4 exhaustion risks, deployment timelines, and presenting benefits to management with real-world considerations.IPv6 Task Force
Local IPv6 advocacy group at SkullSpace hackerspace working to raise awareness through IPv6-only demos, whitepapers, and real-world infrastructure setup between multiple locations.My IPv6 Services
Overview of native IPv6-enabled services including web (nginx/Apache), email (Postfix/Dovecot), XMPP (Prosody), and DNS. Covers configuration and operational challenges.1f j00 R R34d1N 7h12 m3m0ry j00 R pWN3D
Heartbleed OpenSSL vulnerability analysis. Memory disclosure vulnerability affecting OpenSSL 1.0.1-1.0.1f, widespread device impact, and scanning/remediation strategies.Observium 0.14.4
Observium 0.14.4 Community Edition release includes security fixes and interface description parsing for port labeling and aggregate interface graphs.AS4761 April 2 2014 Prefix Origination Event
BGP prefix hijacking incident involving Indonesian ISP Indosat (AS4761) advertising multiple ISP prefixes. Analysis of Thai BGP upstream filtering failure and impact on global routing.IPAM: NIPAP
Switching from subnetsmngr to NIPAP for improved IP address management. NIPAP offers flexible subnet allocation, IPv4/IPv6 parity, VRF support, and both CLI and web interfaces.XKCD Passwords
Examining XKCD comic strip #936 on password security and memorable passphrases. Review of xkcd-password implementations on GitHub for generating user-friendly passwords with entropy calculation.iPerf UDP
Using iperf for UDP performance testing to isolate VoIP-related network issues. Demonstrates measuring jitter, packet loss, and bandwidth with IPv6 over long distances.goto fail
Analysis of critical goto-related bugs in SSL/TLS signature verification in Apple and GnuTLS. Examines the dangers of goto statements and implications of delayed security patching.UPS SNMP Management
UPS SNMP ManagementSSL Ciphers
SSL/TLS cipher and protocol configuration for nginx, dovecot, and Apache. Securing connections by selecting strong ciphers and disabling weak algorithms. Note: See Mozilla SSL Configuration Generator for current best practices.First Brocade CLI Experiences
First impressions with Brocade ICX 6430-24P switch. CLI syntax comparison between Brocade and Cisco for VLAN/access layer configuration, POE support, and Observium integration.RFC2142: Role Email Accounts
RFC2142: Role Email AccountsRFC3021: 31-Bit Prefixes on PtP Links
Using RFC3021 /31 netmasks on point-to-point links to conserve IPv4 addresses. Comparison with traditional /30 usage and device compatibility notes for Cisco and MikroTik equipment.I'm doing AnyCast!
Setting up anycast services using Bird OSPF to inject /32 and /128 routes across multiple locations. Using FreeBSD VMs for redundant IPv4/IPv6 anycast DNS and web services.Highly-Available, Redundant Internet Through BGP
Comparing methods for redundant internet failover: manual, dual-wan firewalls, and BGP routing. BGP provides automatic failover, simplified troubleshooting, and better control over multi-provider connectivity.BGP 101: Failover by Splitting Advertisements
BGP redundancy techniques: prepending, localpref coordination, and address splitting. Comparing methods for inbound failover with pros and cons of each approach.Where Old Code Goes....
Where Old Code Goes...."My Traffic Is Not Reaching The Destination"
Troubleshooting UDP stream connectivity issues using packet captures and Wireshark. Identified TTL expiration as root cause of packets not reaching destination across multiple hops.Starting an IXP
Experiences launching Winnipeg Internet Exchange (WpgIX). Discusses benefits of local peering, network configuration challenges, and impact of content delivery infrastructure.IPv6 Implementation Experiences
Personal experiences with IPv6 adoption from 2001 tunnel access through ISP-level deployment. Covers client behavior (Happy Eyes), NAT implications, security considerations, and network implementation perspectives.Implementing BCP38
Implementing BCP38 ingress filtering on Cisco routers to prevent spoofed traffic from leaving your network. Includes ACL examples to filter bogus source addresses and private ranges.OSPF and IPv6
Configuring OSPFv3 for redundant IPv6 routing on Cisco routers. Covers unicast routing setup, OSPF processes, VLAN configuration, and IPv6 ACLs for secure SSH access.Building an OpenBSD BGP Looking Glass
Setting up bgplg, OpenBSD's built-in shell and CGI BGP looking glass. Includes handling ping/traceroute in chrooted environments and considerations for nginx migration.Observium, A Worthy Adversary!
First impressions of Observium network monitoring platform after 10 years with Cacti. Auto-discovery, 64-bit counters, total device traffic graphs, and MAC/ARP tables make it a compelling alternative.SSL CLI Troubleshooting
Using openssl s_client to test SSL/TLS connections and verify certificate chains for encrypted email services like POP3, IMAP, and SMTP from the command line.An MX Record, Finally!
Setting up self-hosted email with iRedMail on FreeBSD. A polished mail server solution with ClamAV, policyd, and DKIM support that integrates well with system packaging.IPAM: IP Address Management
Setting up subnetsmngr, an open source IPAM solution for small ISPs.Enabling IPv6 on Cisco 3560/3750
Step-by-step guide to enabling full IPv6 support on Cisco Catalyst 3560/3750 switches, including SDM mode configuration, unicast routing, and interface addressing setup.NetInstalling OpenBSD on sparc/sparc64
Step-by-step guide to network boot installation of OpenBSD on SPARC and SPARC64 architecture. Covers RARP, TFTP, BOOTP, and NFS configuration for netbooting Sun workstations.