interview questions for devops

This is the last day of May 2020, hard to imagine that we are here. For someone my age, its a blessing to have lived this long. I watched Blade Runner the movie as a teen, and the future was 2019, a long way away from 1982. The writer figured in thirty years, we would have floating cars, clones, and common slaves (made from cloned humans). It didn’t all happen in 2019 for sure. To be sure of what the writer imagined, watch the movie! We have amazing technology for sure. So in 2020, you are going to get a new hot job for yourself, and get ready for the interview test. This month a friend needed some help with a interview question.

So my friend, mostly doing networking, gets into a cloud / devops job. Mostly a glorified Sysadmin position with average to below pay, but the benefits lure him in. He is in a panic about the XML interview question. Its a large text file with repeating xml/json like strings, each section for a user/computer pair that is in use by the organization. So the interview question is this:

I have a large JSON file, using any programming tool or language, I want you to change every password string in this file to a random string, instead.

Simple right? Well I do these things on a regular basis, but this one was a bit different. I didn’t give him the entire interview answer, but I gave him some hints. I was going to do the data processing with unix SED/AWK, but changed my mind and used powershell instead.

Here is the json file, what it looked like.

PS C:\Users\superDaveRDeluhery> type json.txt
db password : it43ss3489888~
username : superdave
text comment : a wonderful life
data home path : \data\user\superdave
comment : 10

Here is powershell line with REGEX, looks for semicolon : then two digits (d{2}). And replaces with ‘ken’. Notice the “comment : 10” line is now changed with new value.

PS C:\Users\superDaveRDeluhery> (Get-Content -path json.txt -Raw) -replace “\: \d{2}”, “: ken”
db password : it43ss3489888~
username : superdave
text comment : a wonderful life
data home path : \data\user\superdave
comment : ken

—— here same command, but create a new file called newjson.txt (instead of just writing to the screen)

PS C:\Users\superDaveRDeluhery> (Get-Content -path json.txt -Raw) -replace “\: \d{2}”, “: ken” >> newjson.txt

display the changed file

PS C:\Users\superDaveRDeluhery> type .\newjson.txt
db password : it43ss3489888~
username : superdave
text comment : a wonderful life
data home path : \data\user\superdave
comment : ken
PS C:\Users\superDaveRDeluhery>

Have not found the powershell command to create a random string, assign it to a variable, then use that to replace the text. Appears pretty easy to get random numbers, those might need to be converted to char so that you can get random strings (which the interview might want you to create random strings of letters and numbers, not just numbers).

##———————- another test question

June 22/ update – In life we have tests, you do them your way and go from there. Most of us take the easiest route, the path of least resistance. We all do it. But cut and paste is not really possible when you have 50,000 lines to replace. So here is a another test question. You have a list of objects in a text file, like three hundred objects in a text file. You have a single line, an ACL, and you need to create a new acl so that we replace a token in a line entry, and create three hundred line entries having a different object. Here is the example of the data:

myfile.txt contents:

object extended super_acl_55

object extended super_acl_56

object extended super_acl_533

object extended super_acl_5523

How do you take that data and create a new ACL using that data? My answer is use windows shell, use the “for” command to grab the third token (tokens=3). Each token gets echo into a line. I see how to do this in powershell. . .but have not done it, yet. When I grow up and become awesome at powershell, I will perhaps be too old to type. Haha! Back to the for loop. The for example is below:

C:\Users\users_uhery>for /F “eol=; tokens=3,4* delims=, ” %i in (myfile.txt) do @echo access-list CR.ITS line 50 extended permit tcp object %i object eq ssh

This uses the “for” shell command. The command, the for command, will process each line and will take each lines third token, example “super_acl_533) and put that data into the echo statement. So in the echo you see

echo access-list CR.ITS line 50 extended permit tcp object %i object eq ssh

See the %i, between the object and object? See that? Its the variable that for uses to hold the token. That gets replaced with the “super_acl_533”. So the final output would be

access-list CR.ITS line 50 extended permit tcp object super_acl_55 object eq ssh

access-list CR.ITS line 50 extended permit tcp object super_acl_56 object eq ssh

access-list CR.ITS line 50 extended permit tcp object super_acl_533 object eq ssh

##———————- end of test question

Since this is a cisco blog, a few cisco things. Studying CCIE wireless? Understand SSDP and mDNS, you must. Check out SSDP using UDP and multicast. Seen packets with M-SEARCH? Check out details.

network security – cover your web traffic

worked a bit with the folks at Zscaler today. Good guys, they were helpful. People talk about network security and how to make your network secure. There are thousands of things that can be done, some matter and some don’t. People discuss what should and should not be done. One thing that should be done? Cover your web traffic. Meaning create a cover of where your web traffic is coming from. Have a networks that builds a path so that someone can’t associate your users traffic to where your users data is ! Keep it separate. Create an easy way to find traces of where your datacenters are located, who would do that?

For most medium sized organizations, they have people inside the buildings/factories that browse the web, and that creates information. If you use a zscaler cloud service, the browsing leaves a footprint that simply points back to Zscaler. That is one of the things I like about it. Your organization doesn’t go around the internet leaving footprints that point back to the datacenter(s). Footprints and information that can be useful for people trying to figure out how your network operates.

I was at a common US government installation a month ago, and they have free WIFI. I thought, I wonder if this free wifi is associated with the governments internet link? That might be a bad idea. Well, I couldn’t tell, which is good. I went to a website “”. IPchicken and many others show the source of you web traffic. The government building I was in, they covered their web traffic. Stay safe. Cover your web traffic.

Copyright 2020 Rod Deluhery

network monitor

neat, a network monitor running on a cisco ISR router. this runs on the cisco device.


Cisco ISR 4451-X router with 8 GB RAM, 16GB compact flash memory, and 200 GB hard disk.

here is what Cisco says about running apps in the isr routers:

Use the Cisco 4000 Series Integrated Services Router (ISR) and ASR 1000 Series Aggregation Services Router to host not only Cisco apps, but also third-party and homegrown Linux-based apps. These routers’ Cisco IOS® XE operating system supports the Kernel-based Virtual Machine (KVM), a virtualization infrastructure for the Linux kernel that turns it into a hypervisor.

Both routers use a customized high-performance data plane for forwarding and manipulating packets. The control plane is entirely Linux running on an x86 Intel CPU. We designed the routers with extra CPU capacity for hosting VMs.

You can also turn to the Cisco 4000 Series ISR and ASR 1000 Series to host other network functions that can be deployed as VMs, including:

●     Windows domain controller

●     Print servers

●     Network analytics

●     Network functions such as WAN optimization (Cisco WAAS), intrusion detection and prevention (Snort® IPS), and visibility and security intelligence (Cisco Stealthwatch Learning Network License)

Hosting those VMs directly on your existing physical router makes a whole lot of sense.

wireless lookup fccID

No manual for a Alcatel Lucent 9962 cell? Whatever wireless device, if you have the FCC ID number, you might be able to find a manual. Take the FCC ID, and put it in the page below.

On a related note, if you need to find a wireless license, you use this page:

Once you see the FCC exhibits that have been uploaded, you might find a technical manual for the radio you are working on. For example, I wanted to know more about a Alcatel 9962 device with a FCC ID of P279962MSEC. Once I put that FCC id into the search page, I found many documents, including a detailed technical manual for the device. The manual cover and a random page are shown below.

Copyright 2020 Rod Deluhery

wireless detail in 802.11n

I ran across Praveens blog as was trying to recall the terminology for aggregate frames. I almost have the name and details of each layer, MSDU and MPDU. Praveen has some good details on Fortinet it appears. It has been years since I have looked at a Fortinet product, they are into all sorts of products now. Ahh Fortinet, I never hung out with you much. . . maybe I should now? Years ago they only sold firewalls.

The whole concept of MSDU, a service data unit, is not clear to me. Reading about it, I pulled up the IEEE specification document, IEEE Std 802.11-2016. If you want to understand how these things work, read this document. You have to register with an email address on IEEE site, and then you can download it for free (you could a few months ago). Not all the documents from IEEE are free to download, make note of that. If you can get that document, it’s a great study aid for learning 802.11 wireless. The document covers directional multi-gigabit wireless concepts and terminology. It also is one of the only places that goes into detail on how DL-MU-MIMO) actually works.

Looking up PTK and the details, a few things to note. More new things, that I found. For MESH access points, or other equipment, you may find a thing called and AP PeerKey. It’s a protocol for access points to communicate to each other. From the document: “The AP PeerKey protocol provides session identification and creation of an AP PeerKey association to
provide for security of OBSS management communication between two APs. The result of a successful run of the AP PeerKey protocol is an AP PeerKey association. An AP PeerKey association is composed of a mesh PMKSA and a mesh TKSA.

I am still learning here about wireless frames, a few things I found out. In some of the IEEE docs, they call A-MSDU a “payload protected (PP) aggregate medium access control (MAC) service data unit. Wow, that is alot to remember, I forget the terms A-MSDU. What might help is remembering what they mean.

I do have a few questions. What causes an AP or station to implement aggregate frames? Is it basically a queue, and if the queue begins to fill, and there are no transmit opportunities, then “start” and begin building a MSDU frame? I am unsure.

Question, how does a PPDU relate to a MPDU? And how does that relate to service period (sp)? Answer- I think the PPDU can be anything transmitted, like a beacon, or a packet. In the ieee doc, we see on page 2325, “PPDU is formed during data transmission by appending the PSDU to the Exteneded Rate PHY preamble and header. At the receiver, the PHY preamble and header are processed to aid in the demodulation and delivery of the PSDU.” And the service period? a time you are told to transmit or assume you can transmit. My definition, anyway.

How does a TIM (traffic indication map) differ from a beacon?

What is a TSID?

Things change with MIMO. A few things. “Very high throughput (VHT) multi-user (MU) physical layer (PHY) protocol data unit (PPDU): A
VHT PPDU with a format that is capable of carrying up to four PHY service data units (PSDUs) for up to four users and is transmitted using the downlink multi-user multiple input, multiple output (DL-MU-MIMO) technique.” thats from the document. So in MIMO with multiple users on a access point, it reads to me like the AP may have a PPDU to send that has data for up to four (4) different clients. Might make for an interesting conversation with speed uploads. If you have a busy wireless network, using MIMO, your upload speed maybe slower than download speeds. Rationale? With MIMO, downloads are more efficient to clients, from the access point. More effective use of air-time, and uploads have no such MIMO capabilities in some STA. Its the downlink that is faster (effectively faster, actually bits per second, it is the same speed. . .it is the same bits per second!)

Another question, is an MPDU with many MSDU always a good thing? I like to think most designs have good and bad. So when the concept of packing frames together into longer frames, whats the catch? Yes there is a downside to most designs. For frame aggregation, mostly all good things. But can it cause latency? I would say yes, even if it is a 1 millisecond latency, it still maybe the case. Latency of one millisecond is not a big deal, right? Well it is something to keep in mind. Latency here and there can add up. Your server latency, WAN latency, then wireless latency. . . speed is king. Yes speed is king, but you can be fast AND have latency. They call it a long, fat, network. I can see how this make wireless more effective, in general. But at times, could this cause latency issues? Or does this only happen when the signal level is very good (high speed networking)?? If signal level (RSSI) is poor, speed is low, does protocols still try to pack the data using these process? If so, latency will be impacted.

Did you read the link on wikipedia? if not, click the link, its in the previous paragraph. What did you see? From that link, it says this:
A network with a large bandwidth-delay product is commonly known as a long fat network (shortened to LFN). As defined in RFC 1072, a network is considered an LFN if its bandwidth-delay product is significantly larger than 105 bits (12,500 bytes).

So, say you have 300 mpbs wifi link, that is megabits, and that is fast. . . . . . which is really maybe around 20,000,000 megabytes per second.
What does a 1 millisecond delay cause? In one millisecond, you divide 20,000,000 divide by 1000, as 1000 milliseconds per second,
and you could have transmitted 20,000 bytes! Even though fast, your 300 mbps link is not a true speed demon, its a LFN. So you now have a LFN (long fat network) simply by having a 1 millisecond delay.

Back to PPDU on the airwaves. So within the PPDU, is MSDU, at least one. So the way I understand, these MSDU and MPDU only apply in wireless realm. Both sta have to have the HT (high throuputoptionimplemented) so that each sta knows what to do with the msdu packets. As I understand the packets are similiar to like a multiplex circuit (like LACP). After the data is recieved it is put back on the network in the original state, so the packet size goes back to 1500 or so. So there is no fragmentation.

Also the packets all need the same qos value or they won’t get packaged in a MPDU. So different qos packets should never be in a MPDU. That how I read the 802.11-2016 IEEE specifications.

Lastly, remember that all the MSDU and MPDU need encrypted before transmission. How does that happen? It appears each MSDU should be encrypted separately, then packed together. Really? What I found interesting, there is a timer in each MSDU. The timer is the lifetime of the encryption protocol. . . I believe.
“The expiration of the A-MSDU lifetime timer occurs only when the lifetime timer of all of the constituent
MSDUs of the A-MSDU have expired.” page 1365. This is the PTK lifetime timer i believe (only thing that matches that in the ieee document. So the MSDU, each one maybe encrypted differently with a different timer. . .and the MPDU will transmit MSDU with expired PTK lifetimes. . . potentially sending packets that can not be unencrypted? It seem it maybe the case.

Copyright 2020 Rod Deluhery


Found this, thought I would share. Take your network skills up a notch, if you dare. It is a packet analysis and packet creation tool using Python. It is called Scapy. The documentation is good. . . a bit difficult to find some quick how to demonstrations. Here is what I did, using my computer.

First I read up on it:

Then I used PIP (PIP – package installer for python. ) to make sure my PIP was updated.

C:\Users\rod>python -m pip install -U pip
Collecting pip
Downloading (1.4MB)
100% |################################| 1.4MB 573kB/s
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-19.3.1


Then I installed scapy using pip:

C:\Users\rod>pip install –pre scapy[basic]
Collecting scapy[basic]
Downloading (905kB)

After that I was able to run some scapy scripts against network packet captures. Here is one. In this example python/scapy script that reads a file called “capture” and counts packets and outputs to standard output.

*** START of python scapy file *****
from scapy.all import rdpcap

Read capture with Scapy

filename = ‘capture.pcap’
packets = rdpcap(filename)

Create sets to store source and ip addresses

This automatically allows us to count the number of unique addresses!

source_ips = set()
destination_ips = set()
IP = ‘IP’

Loop through all packets in capture

for packet in packets:
# If the packet has IP layer information…
if IP in packet:
source_ip = packet[IP].src
destination_ip = packet[IP].dst


print(‘There are ‘ + str(len(source_ips)) + ‘ unique source IP addresses.’)
print(‘There are ‘ + str(len(destination_ips)) + ‘ unique destination IP addresses.’)

*** END of python scapy file *****

ccie wireless

The CCIE tests are going to ask you many questions about specifics of routers and switches. Even if you take the CCIE wireless, they will ask you switch and router questions, because come on, you are supposed to be an internet expert! So do you really know the basics of switches? How about troubleshooting a connection? Cable test commands? How about “test capwap ap super” commands? You better!

You start a switch based cabled test with this command

“test cable-diagnostics tdr interface gigabitEthernet 1/0/33”

After the test is run, about 45 seconds, type this command:

superswitch1007-RX001#show cable-diagnostics tdr interface gigabitEthernet 1/0/33
TDR test last run on: December 09 10:13:25

Interface Speed Local pair Pair length Remote pair Pair status

Gi1/0/33 1000M Pair A 27 +/- 10 meters Pair A Normal
Pair B 27 +/- 10 meters Pair B Normal
Pair C 27 +/- 10 meters Pair C Normal
Pair D 27 +/- 10 meters Pair D Normal

Interface Speed Local pair Pair length Remote pair Pair status

What about abnormal conditions? Look out the output below. This is a 100megabit only network device, it actually did function at 100mbps with the below two pairs shorted.

Gi1/0/32 100M Pair A 0 +/- 1 meters Pair A Normal
Pair B 0 +/- 1 meters Pair B Normal
Pair C 33 +/- 1 meters N/A Short
Pair D 34 +/- 1 meters N/A Short

Need to do minor capwap WLC or CAPWAP ap testing, from a switch? You might find it useful to know these commands:

superswitch1007-RX001#test capwap ?
ap Cisco AP
cavium-scale Cavium Scalability Test Hack
data CAPWAP data tunnel
ha-ut ha unit test helper
memory Memory allocation and dellocation reference-counter
multicast CAPWAP multicast tunnel

superswitch1007-RX001#test capwap data ?
attrib Modify CAPWAP data tunnel attribute
create Create CAPWAP data tunnel
delete Delete CAPWAP data tunnel

superswitch1007-RX001#test capwap mult
superswitch1007-RX001#test capwap multicast ?
create Create CAPWAP multicast tunnel
delete Delete CAPWAP multicast tunnel

superswitch1007-RX001#test capwap ap ?
name AP Name

superswitch1007-RX001#test capwap ap


Copied from a good cisco article, about MPTCP (multipath TCP). The use case that I see is something more like the picture below, which shows a bit more detail. So for MPTCP, most people use a linux server wit MPTCP support. See diagram:

Here we make a user out on the internet able to connect to Rods House using a faster connection. Using two different carriers, the mobile phones provide redudancy. How to do it? More details on AP bridge, MPTCP server configuration to come.